chore: update star history

.agents/skills/adversarial-validation/SKILL.md

@@ -1,45 +0,0 @@
----
-name: adversarial-validation
-description: Review a final RustFS diff adversarially when the user requests adversarial review, the root AGENTS.md classifies the change as high risk, or a substantial PR is being reviewed. Do not use for ordinary questions, diagnosis, planning, status, documentation-only work, or routine low-risk implementation.
----
-
-# RustFS Adversarial Validation
-
-Use the risk tier and review shape defined in the root `AGENTS.md`. This skill
-routes a review to RustFS-specific probes without loading unrelated domains.
-
-## Select Lenses
-
-Read only the references required by the diff:
-
-| Lens | When to read |
-|---|---|
-| [Correctness](references/correctness.md) | Every non-exempt adversarial review |
-| [Simplicity](references/simplicity.md) | Mechanical/standard changes and production growth |
-| [Test coverage](references/test-coverage.md) | Behavior or test changes |
-| [Security](references/security.md) | Authn/authz, IAM, RPC trust, paths, secrets, parsing, browser, encryption |
-| [Concurrency/durability](references/concurrency-durability.md) | Async shared state, locks, storage commit, cancellation, persisted queues |
-| [Compatibility](references/compatibility.md) | S3 surface, MinIO interop, metadata, wire/disk formats, mixed versions |
-| [Performance](references/performance.md) | Request/object hot paths, allocation, blocking work, fsync, fan-out |
-
-Do not read all references as a precaution. A path name alone is insufficient;
-the changed behavior must touch the lens's domain.
-
-For a dedicated security audit or advisory analysis, use
-`security-advisory-lessons` instead of loading it automatically during every
-adversarial review.
-
-## Review Protocol
-
-1. Freeze the exact final diff/head and list the selected lenses.
-2. Run the review shape required by root `AGENTS.md`.
-3. For each selected lens, either report a concrete finding or a null verdict
-   naming the attacks performed.
-4. A finding needs `file:line`, a triggering input/state/interleaving, the wrong
-   outcome, and a focused fix or missing regression check.
-5. Fix or rebut every finding with code-path, test, or invariant evidence.
-6. After a non-trivial edit, rerun only lenses affected by that edit against the
-   new exact diff.
-
-Do not turn a null verdict into a long checklist. Record concise evidence that
-the relevant failure classes were attacked.

.agents/skills/adversarial-validation/references/compatibility.md

@@ -1,24 +0,0 @@
-# Compatibility Lens
-
-- Internal metadata uses `metadata_compat` helpers for dual RustFS/MinIO keys,
-  including mixed casing and removal of both twins.
-- Binary UUID metadata treats absent, empty, and nil as no value. Unversioned
-  remote tiers receive no `versionId`; versioned purge requests retain the real
-  version ID.
-- `xl.meta` changes preserve supported header/meta versions, recompute
-  signatures, decode legacy fixtures, and remain readable by old RustFS/MinIO.
-- Foreign/corrupt metadata validates parallel array lengths and missing fields;
-  it returns a decode error rather than indexing, panicking, or fabricating data.
-- Do not “correct” byte-for-byte MinIO ports without legacy fixture evidence.
-  Bitrot framing, shard math, distribution, and inline prefixes are contracts.
-- Client-visible metadata/events strip both internal prefixes
-  case-insensitively.
-- Proto fields are appended, never reused/renumbered; FlatBuffers tables extend
-  compatibly and absent new fields fail closed where authorization/quorum is
-  involved.
-- Replay real client request shapes and exact pagination boundaries for S3
-  handler changes.
-- Bucket metadata/IAM/config parsing remains compatible with pinned real MinIO
-  fixtures and encrypted migration data.
-- Compatibility shims use `RUSTFS_COMPAT_TODO(<task-id>)`, have a removal
-  condition, and default toward reading old data safely.

.agents/skills/adversarial-validation/references/concurrency-durability.md

@@ -1,23 +0,0 @@
-# Concurrency and Durability Lens
-
-- For every changed lock, enumerate overlapping lock sets and construct the
-  ABBA interleaving. Multiple-lock order must be documented and consistent.
-- Mark guard lifetimes and every `.await`, disk, and RPC call inside them.
-  Estimate contention and timeout behavior under concurrent requests.
-- Object commits remain fenced if the distributed lock is lost after shard
-  writes and before metadata rename.
-- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent
-  -> sync required ancestors`; simulate a crash after each step and honor the
-  configured durability gate.
-- Multi-disk fan-out counts every result. Quorum-minus-one cannot become success;
-  heal remains best-effort per target where that is the established contract.
-- At every new cancellable await between mutation and cleanup/commit, drop the
-  future and inspect leftover files, counters, permits, and replay state.
-- Multipart operations on the same upload ID are serialized where required;
-  abort/complete/list races cannot delete parts before durable commit.
-- Post-commit cleanup is best-effort, retry-safe, and cannot fail an already
-  committed write or delete the last surviving copy.
-- Persisted read-modify-write uses serialization/CAS. Queue replay is crash-safe
-  and duplicate delivery has an idempotency contract.
-- Streaming reconstruction failures after partial output surface as errors, not
-  successful EOF.

.agents/skills/adversarial-validation/references/correctness.md

@@ -1,29 +0,0 @@
-# Correctness Lens
-
-Attack the changed behavior, not every subsystem in the repository.
-
-- Trace new error paths to the caller. Inject the ignored/wildcard variants and
-  verify they cannot become success, not-found, or a plausible default.
-- Exercise zero/empty/missing, maximum, and exact-boundary inputs for every
-  changed count, size, index, page limit, or optional value.
-- For aggregation/quorum changes, test exactly quorum and quorum-minus-one with
-  mixed disk errors and nil/placeholder entries.
-- For listing/pagination, test `n == max`, `n == max + 1`, delimiter folding,
-  continuation markers, and object/prefix name collisions.
-- For EC/read/streaming changes, inject failure after partial output and verify
-  the client receives an error rather than a clean truncated body. Assert exact
-  bytes and length.
-- For multipart/object commits, fail before/after rename and cleanup; committed
-  data must remain readable and pre-commit cleanup must not destroy parts.
-- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
-  versions, and deterministic tie-breaking.
-- For directory-object behavior, trace `__XLDIR__` at the store layer; branches
-  below the layer that sees trailing slashes are dead.
-- For binary UUID metadata, absent, empty, and nil all mean no value. Never send
-  nil/empty `versionId` to an unversioned tier.
-- For agent rules/skill routers, test a trigger matrix covering ordinary
-  inquiry, low-risk implementation, explicit review, high-risk code, PR
-  creation, release, and post-PR monitoring. Each case must select only the
-  intended workflow and retain required safety/authorization boundaries.
-
-Null verdicts name only the probes relevant to the diff.

.agents/skills/adversarial-validation/references/performance.md

@@ -1,20 +0,0 @@
-# Performance Lens
-
-- For added clones/allocations on request/object/block paths, quantify copied
-  data and frequency. Recommend borrowing, move, `Bytes`/`Arc`, `Cow`, or
-  capacity reservation only for a concrete repeated cost.
-- Route every new sync/flush through the durability-mode and bucket override
-  gates; mode `none` must not pay the new fsync.
-- Keep blocking filesystem/CPU work off async runtime threads, but do not split
-  one small operation into many `spawn_blocking` round trips.
-- Measure lock hold time across I/O and compare acquisition order for ABBA.
-- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
-  section when they need not be there.
-- Detect per-item serial I/O/RPC in batch APIs and accidental quadratic scans;
-  use a gate or bounded concurrency when the concrete fan-out warrants it.
-- Count buffer growth and byte copies in EC/bitrot paths; preserve pool gauge
-  balance and avoid repeated metadata decode/fetch per object.
-- Repetitive success logs stay at `trace`; metrics/instrumentation on hot paths
-  require an existing gate.
-- Claims of no impact on PUT/GET/commit/erasure paths need relevant benchmark or
-  A/B evidence, especially for 4 KiB objects.

.agents/skills/adversarial-validation/references/security.md

@@ -1,31 +0,0 @@
-# Security Lens
-
-Use `security-advisory-lessons` only for a dedicated advisory/security audit.
-For an ordinary matched diff, attack these boundaries:
-
-- Admin routes: route registration, whitelist, handler authn, and the exact
-  `AdminAction` must agree. Read-only diagnostics still require admin authz.
-- IAM/service accounts: treat parent, claims, keys, groups, status, and policy
-  names as attacker-controlled; prove ownership/root authority before writes.
-- Protocol frontends: every changed/sibling command authorizes the matching S3
-  action before reaching storage.
-- Secrets/signatures: use constant-time comparison, normalize public failures,
-  keep RPC/root/STS keys independent, and fail closed when secrets are absent.
-- RPC: bind signatures to the exact method/path and timestamp; reject replay,
-  stale, malformed, truncated, and invalid-enum payloads without panic.
-- Paths/object/archive entries: reject traversal, absolute/platform escapes,
-  and normalization differences between authz and storage.
-- Copy/multipart/presigned POST: enforce source, destination, version-aware
-  actions, copy-source conditions, and every signed policy condition.
-- Logging/errors: never expose credentials, tokens, expected signatures, raw
-  secret-bearing input, or merged configs—including via `Debug` and parse errors.
-- Untrusted serde: reject unknown fields where compatible and validate
-  security-critical defaults/ranges before numeric conversion.
-- SSE/browser/CORS/trusted proxy: inspect stored ciphertext and wrapper order;
-  isolate user content; never reflect credentialed arbitrary origins or trust
-  forwarded identity from direct clients.
-- Object Lock: unreadable/fabricated/unparsable metadata fails closed across
-  foreground, lifecycle, scanner, and force-delete paths.
-
-Security findings distinguish unauthenticated compromise from a
-low-privileged authenticated bypass.

.agents/skills/adversarial-validation/references/simplicity.md

@@ -1,22 +0,0 @@
-# Simplicity Lens
-
-- Compare the production diff with the smallest equivalent local edit. Fewer
-  lines alone are not evidence; the replacement must preserve correctness,
-  compatibility, readability, and real boundaries.
-- Search the touched crate, domain owner, `crates/utils`, `crates/common`, and
-  relevant dependencies for each new helper, constant, wrapper, or fixture.
-- Reject forced reuse when normalization, error, deadline, or durability
-  semantics differ.
-- Require a concrete trigger for every new defensive branch. Keep boundary
-  checks for disk/RPC/version data and checks immediately before destructive
-  actions.
-- Flag one-caller helpers only when they merely forward or split a short linear
-  flow without adding domain naming, invariant isolation, or useful context.
-- Ensure a replacement removes the superseded in-scope path or keeps one
-  canonical core behind a documented compatibility adapter.
-- Remove narration/change-history comments; preserve concise safety, lock,
-  durability, and compatibility invariants.
-- Treat tests, fixtures, generated code, and documentation separately from
-  production growth. Do not optimize away meaningful regression coverage.
-
-A finding must include a concrete smaller design, not a style preference.

.agents/skills/adversarial-validation/references/test-coverage.md

@@ -1,24 +0,0 @@
-# Test-Coverage Lens
-
-- For every behavior claim, name the focused test/check that fails if the
-  changed hunk is reverted. If none is practical, require the reason and
-  residual risk.
-- Confirm tests exercise the real production path and assert returned values,
-  exact bytes, stored state, or the specific error variant—not only success,
-  `is_err()`, or no panic.
-- For new flags/modes, verify each branch and ask which test fails if the branch
-  is inverted.
-- For new error propagation, inject the failure and assert the caller observes
-  it; mentally replacing `?`/`return Err` with success must break a test.
-- Streaming GET tests assert the complete body and length under degraded reads.
-- Disk/wire-format tests use pinned foreign/legacy fixtures; same-code
-  round-trips are insufficient for compatibility.
-- Concurrency tests use readiness polling, isolate global state, and avoid fixed
-  sleeps or unrealistically short timeouts. Use nextest groups when process-level
-  serialization is required.
-- Internal metadata tests assert both RustFS and MinIO keys, not only read-back
-  through a helper that prefers one key.
-- Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
-  absent vs empty vs nil UUID.
-- A focused test proves only the targets/features it builds. Add compilation or
-  Clippy only for uncovered changed targets.

.agents/skills/arch-checks/SKILL.md

@@ -1,64 +0,0 @@
----
-name: arch-checks
-description: Resolve failures from the repository's architecture guard scripts — check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_unsafe_code_allowances.sh, check_logging_guardrails.sh, check_doc_paths.sh. Use when make pre-commit / pre-pr or CI fails on one of these checks.
----
-
-# Architecture Guard Checks
-
-All five run in `make pre-commit` / `make pre-pr` and in CI. Fix the cause;
-never weaken a check to get green.
-
-## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
-
-Enforces `composition (server, startup/init) → interface (admin,
-storage/ecfs, storage/s3_api) → app → infra`; no upward imports. Server source
-files are composition roots, while imports of their exported HTTP contracts
-are classified as interface dependencies. Known legacy violations live in
-`scripts/layer-dependency-baseline.txt`.
-
-Dedicated `*_test.rs` and `tests/` modules are outside this production guard.
-Inline `#[cfg(test)]` imports remain checked under their source file's layer;
-move architecture-crossing test scaffolding into a dedicated test module.
-
-- **New violation**: restructure your change so the dependency points
-  downward (move the shared type/function to the lower layer).
-- **You legitimately removed a baseline entry**: run
-  `./scripts/check_layer_dependencies.sh --update-baseline` and commit the
-  shrunken baseline. Never add new entries to the baseline to make a new
-  violation pass.
-
-## `check_architecture_migration_rules.sh` — required doc sections
-
-Asserts that the core docs under `docs/architecture/` (overview,
-crate-boundaries, runtime-lifecycle, readiness-matrix,
-storage-control-data-plane, global-state-crate-split-plan,
-ecstore-module-split-plan, …) still contain specific headings and exact
-source lines. If it fails after a doc edit, you reworded or removed a
-guarded line — restore the wording or update the script deliberately in the
-same PR, with rationale.
-
-## `check_unsafe_code_allowances.sh`
-
-Every `#[allow(unsafe_code)]` needs a `SAFETY:` comment within a few lines.
-Write the actual safety argument; don't add a placeholder.
-
-## `check_logging_guardrails.sh`
-
-A fixed list of security-sensitive files (auth, IAM, KMS, admin handlers…)
-is scanned for logging violations. If you created a new sensitive file,
-consider adding it to the script's `checked_files` list.
-
-## `check_doc_paths.sh`
-
-Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
-`docs/architecture/*.md`) must not reference repo file paths that no longer
-exist. If your refactor moved code, update the docs that point at it — the
-error message lists `doc -> stale-path` pairs.
-
-## `check_no_planning_docs.sh`
-
-Planning-type documents must not be committed (see AGENTS.md "Sources of
-Truth"). The guard fails if anything is tracked under `docs/superpowers/` —
-`.gitignore` already ignores it, but `git add -f` bypasses that, so this closes
-the hole. Fix by removing the listed file(s) with `git rm`; keep the plan or
-spec in the issue tracker or a local worktree instead.

.agents/skills/code-change-verification/SKILL.md

@@ -1,82 +0,0 @@
----
-name: code-change-verification
-description: Review a commit, PR, or merged patch when the user requests ordinary code-change verification. Do not combine with adversarial-validation; use that skill instead for explicitly adversarial, substantial, or high-risk RustFS reviews.
----
-
-# Code Change Verification
-
-Use this skill for an ordinary requested review. If the root policy or user calls
-for adversarial validation, use `adversarial-validation` instead of running both.
-
-## Quick Start
-
-1. Read the scope: commit, PR, patch, or file list.
-2. Map each changed area by risk and user impact.
-3. Inspect each risky change in context.
-4. Report findings first, ordered by severity.
-5. Close with residual risks and verification recommendations.
-
-## Core Workflow
-
-### 1) Scope and assumptions
-- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
-- If context is missing, state assumptions before deeper analysis.
-- Focus only on requested scope; avoid reviewing unrelated files.
-
-### 2) Risk map
-- Prioritize in this order:
-  - Data correctness and user-visible behavior
-  - API/contract compatibility
-  - Security and authz/authn boundaries
-  - Concurrency and lifecycle correctness
-  - Performance and resource usage
-- Give higher priority to stateful paths, migration logic, defaults, and error handling.
-
-### 3) Evidence-based inspection
-- Read each modified hunk with neighboring context.
-- Trace call paths and call-site expectations.
-- Check for:
-  - invariant breaks and missing guards
-  - unchecked assumptions and null/empty/error-path handling
-  - stale tests, fixtures, and configs
-  - hidden coupling to shared helpers/constants/features
-- If a point is uncertain, mark it as an open question instead of guessing.
-
-#### Rust-specific checks (apply to all Rust changes)
-
-Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — the canonical Rust review checklist for the unwrap/casting/cloning/locking/recursion/error-type/serde/test rules and the reuse-and-necessity checks (duplicated helpers, defensive branches without a nameable trigger, redundant error wrapping). Do not restate those rules here; carry its P0–P3 ratings over unchanged and use this skill's output format.
-
-### 4) Findings-first output
-- Order findings by severity:
-  - P0: critical failure, security breach, or data loss risk
-  - P1: high-impact regression
-  - P2: medium risk correctness gap
-  - P3: low risk/quality debt
-- For each finding include:
-  - Severity
-  - `path:line` reference
-  - concise issue statement
-  - impact and likely failure mode
-  - specific fix or mitigation
-  - validation step to confirm
-- If no issues exist, explicitly state `No findings` and why.
-
-### 5) Close
-- Report assumptions and unknowns.
-- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
-
-## Output Template
-
-1. Findings
-2. No findings (if applicable)
-3. Assumptions / Unknowns
-4. Recommended verification steps
-
-## Finding Template
-
-- `[P1] Missing timeout for downstream call`
-  - Location: `path/to/file.rs:123`
-  - Issue: ...
-  - Impact: ...
-  - Fix suggestion: ...
-  - Validation: ...

.agents/skills/code-change-verification/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Code Change Verification"
-  short_description: "Prioritize risks and verify code changes before merge."
-  default_prompt: "Use $code-change-verification for an ordinary requested diff review with prioritized findings."

.agents/skills/plugin-contract-guard/SKILL.md

@@ -1,83 +0,0 @@
----
-name: plugin-contract-guard
-description: Invariants and change procedure for the target-plugin / extension system — plugin manifests, admin plugin/extension catalog and instance APIs, secret redaction, external-plugin install policy. Use when editing crates/targets (manifest, plugin, control_plane, catalog, runtime), crates/extension-schema, or rustfs/src/admin plugin_contract.rs / plugins_*.rs / extensions.rs / target_descriptor.rs.
----
-
-# Plugin & Extension Contract Guard
-
-The "plugin system" spans four surfaces that must stay consistent:
-
-| Surface | Location |
-|---|---|
-| Manifests & registry | `crates/targets/src/{manifest,plugin}.rs` |
-| Install/enable planning (control plane) | `crates/targets/src/control_plane.rs` |
-| Extension schemas | `crates/extension-schema/src/lib.rs`, `crates/targets/src/catalog/extension.rs` |
-| Admin API contract | `rustfs/src/admin/plugin_contract.rs`, `handlers/{plugins_catalog,plugins_instances,extensions,target_descriptor}.rs` |
-
-## Hard invariants (verify before merging)
-
-1. **Secrets have one source of truth.** Secret config keys are declared only
-   in the plugin manifest (`TargetPluginManifest.secret_fields`,
-   `crates/targets/src/manifest.rs`) and flow to admin via
-   `AdminTargetSpec.secret_fields`. Never add a hand-maintained per-service
-   secret table in a handler; if redaction misses a field, fix the manifest.
-
-2. **Redaction must round-trip.** Instance GET responses replace secret values
-   with `***redacted***` (`REDACTED_SECRET_VALUE` in `plugins_instances.rs`).
-   Instance PUT restores the stored secret when it receives that placeholder
-   back (`restore_redacted_secret_values`). Any new read or write path for
-   target config must keep both halves: redact on the way out, restore the
-   placeholder on the way in. The placeholder literal must never be persisted.
-
-3. **Fixtures never reach production responses.**
-   `example_external_webhook_plugin()` (`crates/targets/src/catalog/mod.rs`)
-   is a test/demo fixture for control-plane planning tests. Production
-   catalog/extension handlers must not include it; regression tests
-   (`plugin_catalog_never_exposes_example_or_external_fixtures`,
-   `extension_catalog_never_exposes_example_or_external_fixtures`) enforce it.
-
-4. **External plugin flow is planning-only and deny-by-default.**
-   `plan_external_target_plugin_action` returns decisions, it executes
-   nothing. `TargetPluginExternalFlowGate::default()` is fully closed and
-   `TargetPluginInstallPolicy::default().allowed_download_hosts` is empty —
-   keep it that way; tests opt in via explicit policies. Install validation
-   requires https, an allowlisted host, a full 64-hex-char sha256 digest,
-   signature and provenance URIs, and an artifact matching the host
-   `target_triple`.
-
-5. **Custom target types must not collide.** Unknown target types get an
-   interned unique `custom:<type>` plugin id (`custom_plugin_id` in
-   `manifest.rs`). Custom plugins with secrets must register via
-   `TargetPluginDescriptor::with_manifest` and declare `secret_fields`;
-   `::new` derives a manifest with no secrets.
-
-## Changing the admin JSON contract
-
-- Shapes are locked twice in `plugin_contract.rs` tests: insta snapshots
-  (`rustfs/src/admin/snapshots/`) plus literal `json!` assertions. Update
-  both deliberately; a shape change is a console-facing API change.
-- Field naming is `snake_case`, except discovery blocks
-  (`runtimeCapabilities`, `clusterSnapshot`, `extensionsCatalog`) which are
-  camelCase **by cross-endpoint convention** (same shape in `system.rs`,
-  `console.rs`, `pools.rs`). Do not "fix" that inconsistency locally.
-- Contract types deliberately duplicate `rustfs_targets` types
-  (anti-corruption layer). Add a `From` impl; do not serialize internal
-  types directly.
-
-## Handler conventions
-
-- Every new admin plugin/extension route needs authorization at the top of
-  `call` and an `include_str!` guard test asserting it (repo-wide pattern —
-  see `plugin_instance_handlers_require_admin_authorization_contract`).
-- Reads use `GetBucketTargetAction` (instances) or `ServerInfoAdminAction`
-  (catalogs); writes use `SetBucketTargetAction`.
-- Refresh persisted module switches once per request
-  (`refresh_persisted_module_switches`), then evaluate the sync
-  `module_disabled_block_reason` per domain — do not re-read the store per
-  domain or per instance.
-
-## Generic bounds
-
-Event-payload generics use the `PluginEvent` blanket trait
-(`crates/targets/src/plugin.rs`). Do not respell
-`Send + Sync + 'static + Clone + Serialize + DeserializeOwned`.

.agents/skills/pr-creation-checker/SKILL.md

@@ -1,46 +0,0 @@
----
-name: pr-creation-checker
-description: Perform the final RustFS PR preflight and draft compliant English title/body metadata immediately before creating or updating a PR. Do not use during implementation or as a second general code review.
----
-
-# PR Creation Checker
-
-Use this skill only at the PR boundary. Reuse completed diff review and
-verification evidence; do not reread the repository or rerun equivalent checks.
-
-## Preflight
-
-1. Confirm the branch is based on current `origin/main` and contains only the
-   intended task diff.
-2. Inspect `git diff --stat`, `git diff --check`, and changed file names for
-   secrets, logs, generated artifacts, or unrelated edits.
-3. Confirm the checks selected by root `AGENTS.md` passed on the final diff.
-   Do not replace focused behavioral tests with a generic gate or rerun checks
-   already covered by an unchanged umbrella run.
-4. Read `.github/pull_request_template.md`. Consult `Makefile`, `.config/make/`,
-   or CI only when the required command/current gate is uncertain.
-5. Return `BLOCKED` for an unclean scope, missing required evidence, failed
-   required checks, or non-compliant metadata.
-
-## Metadata
-
-- Title: English Conventional Commit, at most 72 characters, with no tool
-  prefix.
-- Body: English, exact template headings, `N/A` where needed, concise rationale,
-  actual verification commands, and material risks/rollback notes.
-- Use repository-relative paths; never include local absolute paths.
-- Keep prose paragraphs on one logical line and never include the literal
-  sequence `\n`.
-- Use a temporary body file with `gh pr create --body-file` or
-  `gh pr edit --body-file`; never pass multiline Markdown inline.
-
-## Output
-
-- Status: `READY` or `BLOCKED`.
-- Title.
-- Complete PR body.
-- Verification commands and results.
-- Risks or `N/A`.
-
-Immediately before the GitHub write, repeat only the five preflight checks above
-against the final head.

.agents/skills/pr-creation-checker/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "PR Creation Checker"
-  short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
-  default_prompt: "Use $pr-creation-checker for final PR preflight and compliant English title/body metadata."

.agents/skills/rust-code-quality/SKILL.md

@@ -1,123 +0,0 @@
----
-name: rust-code-quality
-description: Run a focused Rust quality review when the user requests one, when reviewing a Rust PR/commit, or when another selected review workflow delegates Rust-specific checks. Do not auto-load for every implementation edit.
----
-
-# Rust Code Quality Gate
-
-Use this skill for a dedicated Rust review to cover rules that `cargo clippy`
-does not catch.
-
-## Quick Start
-
-1. Identify changed `.rs` files.
-2. Run automated checks on changed files.
-3. Run manual review checklist on the diff.
-4. Resolve or rebut every finding with evidence; P0/P1 findings cannot be deferred.
-
-## Automated Checks
-
-Use these searches to find candidates in changed `.rs` files. Inspect syntax,
-`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
-filters do not reliably distinguish production code from tests.
-
-```bash
-# 1. unwrap/expect candidates
-rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
-
-# 2. Silent type truncation via `as` cast
-rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
-
-# 3. String as error type
-rg -n 'Result<.*String>' <changed-files>
-
-# 4. Box<dyn Error> in public APIs
-rg -n 'Box<dyn.*Error' <changed-files>
-
-# 5. println/eprintln in production
-rg -n 'println!\|eprintln!' <changed-files>
-
-# 6. Ordering::Relaxed usage (verify each is intentional)
-rg -n 'Ordering::Relaxed' <changed-files>
-
-# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
-rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
-```
-
-## Manual Review Checklist
-
-For the Rust diff under review, verify:
-
-### Error Handling
-- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
-- [ ] No `Result<_, String>` in public API signatures
-- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
-- [ ] `Error::source()` is overridden when inner error is stored
-- [ ] Error messages are actionable without exposing secret input
-
-### Type Safety
-- [ ] No silent `as` truncation (negative→unsigned, large→small)
-- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
-- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
-
-### Concurrency
-- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
-- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
-- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
-- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
-
-### Memory and Performance
-- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
-- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
-
-### Recursion Safety
-- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
-- [ ] Tree/cache traversals handle corrupted/cyclic input safely
-
-### Testing
-- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
-- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
-- [ ] Test volume and line count are never treated as production-code growth
-
-### Serde
-- [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
-- [ ] `#[serde(default)]` not used on security-critical fields without validation
-
-### Code Hygiene
-- [ ] No `#![allow(dead_code)]` at crate root
-- [ ] No camelCase statics or Hungarian notation
-- [ ] New string literals don't duplicate existing constants
-
-### Reuse and Necessity
-- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
-- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
-- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
-- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
-- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
-
-## Severity Classification
-
-- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
-- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
-- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
-- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
-
-## Output Template
-
-```
-## Rust Code Quality Report
-
-### Automated Scan
-- unwrap/expect candidates inspected: N
-- numeric-cast candidates inspected: N
-- error-type candidates inspected: N
-- output-macro candidates inspected: N
-
-### Findings
-- [P1] `path:line` — description
-  - Fix: ...
-  - Validation: ...
-
-### Verdict
-PASS / BLOCKED (list blocking findings)
-```

.agents/skills/rustfs-logging-governance/SKILL.md

@@ -1,34 +0,0 @@
----
-name: rustfs-logging-governance
-description: Add or review RustFS `tracing` events with the repository field shape, level policy, privacy boundaries, and guardrails. Use when a change adds or edits a tracing macro/instrumentation site or the logging guardrail script.
----
-
-# RustFS Logging Governance
-
-Apply this skill only to changed logging sites; do not turn a local log edit into
-a broad logging cleanup.
-
-## Workflow
-
-1. Read the changed function/module context and classify the site as lifecycle,
-   request/hot path, fallback, external fetch, or summary.
-2. Match neighboring structured events and reuse existing `EVENT_*`,
-   `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` constants.
-3. Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
-   then context) and a short label last.
-4. Select the level by operational meaning:
-   - `error`: behavior/security-affecting failure;
-   - `warn`: degraded/fallback/operator-actionable state;
-   - `info`: low-frequency lifecycle/mode change;
-   - `debug`: targeted diagnostics;
-   - `trace`: repetitive request/object/shard success paths.
-5. Never log secrets, tokens, auth headers, credential payloads, raw
-   attacker-controlled bodies, or merged config dumps. Error strings and
-   `Debug` output are log surfaces too.
-6. Prefer one aggregate summary over inventories or startup banners.
-7. Run `./scripts/check_logging_guardrails.sh` and the checks selected by root
-   `AGENTS.md`.
-
-Read [logging-governance.md](references/logging-governance.md) only for a broad
-logging audit, event-model migration, or guardrail expansion. Ordinary single-
-site edits do not require the full workspace scope map.

.agents/skills/rustfs-logging-governance/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "RustFS Logging Governance"
-  short_description: "Standardize RustFS logs with structured events and guardrails."
-  default_prompt: "Use $rustfs-logging-governance to standardize or review RustFS logging, reduce noise, and update guardrails."

.agents/skills/rustfs-logging-governance/references/logging-governance.md

@@ -1,62 +0,0 @@
-# Logging Audit and Migration Reference
-
-Read this reference only for a broad logging audit, an event-model migration,
-or a change to `scripts/check_logging_guardrails.sh`. Use `Cargo.toml` for the
-current workspace/crate list instead of maintaining one here.
-
-## Audit by Operational Role
-
-- Server/protocol/admin: lifecycle, authorization failures, request boundaries,
-  and degraded subsystems; avoid normal request success at `info`.
-- Storage/heal/scanner/capacity: integrity failures and aggregate lifecycle;
-  avoid per-object, per-shard, and folder iteration noise.
-- IAM/policy/credentials/KMS/crypto: safe identifiers and enforcement results;
-  never emit secrets, claims, payloads, or expected authenticators.
-- Notify/audit/targets: target lifecycle and batch/backpressure summaries; avoid
-  per-event success logs.
-- Locking/concurrency/I/O foundations: contention anomalies and state changes;
-  prefer metrics for high-frequency worker/permit signals.
-- Shared type/schema crates: log at the operational caller boundary unless the
-  crate itself owns the failure context.
-
-## Event Shape
-
-Prefer stable fields in this order when available:
-
-1. `event`
-2. `component`
-3. `subsystem`
-4. `state` or `result`
-5. stable context such as mode, duration, reason, counts, safe identifiers, or
-   capacity/permit values
-6. short message label
-
-Reuse the module's constants and neighboring field names. Do not create aliases
-for the same concept.
-
-## Patterns to Retire
-
-- sentence-style lifecycle announcements;
-- startup banners and checklist lines;
-- repetitive success logs at `info`/`debug`;
-- raw inventories when an aggregate count is sufficient;
-- fallback prose with values embedded in the message;
-- `?value`/`Debug` output for credential-bearing or attacker-controlled data;
-- logging a parse input when the malformed input may itself be a secret.
-
-## Guardrail Changes
-
-When expanding `scripts/check_logging_guardrails.sh`:
-
-1. Add only files/patterns intentionally migrated in the same change.
-2. Keep patterns concrete and grep-friendly.
-3. Do not encode a style that remains valid elsewhere as a global ban.
-4. Run the guardrail script and the root validation tier.
-5. Treat the script as a floor; manually verify level, field shape, and privacy.
-
-Useful search seeds for the changed surface:
-
-```bash
-rg -n 'error!|warn!|info!|debug!|trace!|#\[instrument' <changed-paths>
-rg -n '\?[^,)]|secret|token|credential|authorization|merged_config' <changed-paths>
-```

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,233 +0,0 @@
----
-name: rustfs-release-publish
-description: "Run the end-to-end RustFS console gate, version bump, preview validation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
----
-# RustFS Release Publish (preview-validated pipeline)
-
-This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
-
-Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
-
-Pipeline shape:
-
-```
-check console main against its latest Release
-  -> if ahead: publish console -> wait for Release asset + latest API
-  -> bump RustFS version files to <target> (final version, ONE commit) -> merge
-  -> tag <preview-tag> at that commit -> CI green
-  -> verify preview Release assets -> run binary locally + console checks
-  -> validate with latest rc client
-  -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-```
-
-On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
-
-## Required inputs
-
-- Final target version, for example `1.0.0-beta.10`.
-- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
-
-If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
-
-## Semver gate — confirm the target version before touching anything
-
-Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
-
-```
-1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
-```
-
-Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
-
-Rules:
-
-- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
-- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
-- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
-
-## Preview tag naming
-
-- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
-- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
-- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
-
-## Hard rules
-
-- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
-- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
-- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
-- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
-- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
-- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
-- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
-- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
-- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
-
-## Phase 0 — Preflight
-
-- `git status --short` clean; `git fetch origin main --tags`.
-- `gh auth status` works; confirm you can view `gh release list -L 3`.
-- Confirm the exact final target version with the user if not explicit.
-
-### Console release gate
-
-Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
-
-1. Read the latest published Console tag and compare it with Console `main`:
-
-```bash
-CONSOLE_REPO="rustfs/console"
-CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
-gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
-  --jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
-```
-
-- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
-- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
-- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
-
-2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
-
-```bash
-CONSOLE_SCRATCH=$(mktemp -d)
-gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
-git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
-CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
-git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
-gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
-```
-
-If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
-
-```bash
-git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
-git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
-```
-
-Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
-
-3. Find the exact tag run and wait for completion:
-
-```bash
-gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
-gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
-```
-
-4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
-
-```bash
-gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
-test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
-test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
-  --jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
-```
-
-Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
-
-## Phase 1 — Version bump to the final target (once)
-
-- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
-- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
-- Get the PR merged into main. Record the resulting main commit:
-
-```bash
-git fetch origin main
-PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
-```
-
-`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
-
-## Phase 2 — Publish the preview tag
-
-```bash
-git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
-git push origin "<preview-tag>"
-```
-
-Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
-
-The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
-
-On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
-
-## Phase 3 — CI and preview Release verification
-
-- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
-- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
-- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
-- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
-- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
-
-## Phase 4 — Run the artifact locally, verify the console
-
-Work inside the session scratchpad directory; never leave stray data dirs.
-
-```bash
-gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
-cd "$SCRATCH" && unzip -o rustfs-*.zip
-./rustfs --version        # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
-mkdir -p data
-RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
-```
-
-Defaults: S3 endpoint `:9000`, embedded console `:9001`.
-
-Checks (all must pass):
-
-- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
-- `curl -fsS http://localhost:9000/health/ready` returns ready.
-- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
-- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
-
-## Phase 5 — Validate with the latest rc client
-
-`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
-
-- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
-- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
-
-```bash
-rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
-rc ls preview/
-rc mb preview/rel-check
-rc cp <local-file> preview/rel-check/
-rc stat preview/rel-check/<file>
-rc cat preview/rel-check/<file>          # matches source
-rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
-rc cp -r <local-dir>/ preview/rel-check/dir/
-rc find preview/rel-check --name "*"
-rc share download preview/rel-check/<file> --expire 1h   # presigned URL fetchable via curl
-rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
-rc rb preview/rel-check
-rc admin user list preview/
-rc admin user add preview/ relcheckuser relchecksecret12
-rc admin user remove preview/ relcheckuser
-rc alias remove preview
-```
-
-- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
-
-## Phase 6 — Publish the final tag on the validated commit
-
-No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
-
-```bash
-git fetch origin --tags
-git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
-git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
-git push origin "<target>"
-```
-
-- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
-- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
-- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
-- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
-
-## Output contract
-
-Always report:
-
-- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
-- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
-- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
-- Any deviation from this pipeline and why the user approved it.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,120 +0,0 @@
----
-name: rustfs-release-version-bump
-description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
----
-# RustFS Release Version Bump
-
-Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).
-
-Validated baseline: release pattern used in PR `#2957`.
-
-## Required inputs
-
-- Exact target version, for example `1.0.0-beta.4`.
-- Delivery scope:
-- Local only (`edit/verify`).
-- Local + git (`commit/push`).
-- Full GitHub flow (`commit/push/PR`).
-
-If target version is missing or ambiguous, stop and ask before editing.
-
-Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
-
-## Read before editing
-
-- `AGENTS.md` (root and nearest path-specific files).
-- `.github/pull_request_template.md`.
-- Current branch status and diff against `origin/main`.
-
-## Default release file scope
-
-Treat the following file list as the default checklist for each release bump:
-
-- `Cargo.toml`
-- `Cargo.lock`
-- `README.md`
-- `README_ZH.md`
-- `flake.nix`
-- `helm/rustfs/Chart.yaml`
-- `rustfs.spec`
-
-Only drop a file when the current repository release process clearly no longer requires it.
-
-## Hard release policy
-
-- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
-- Helm chart version mapping follows `beta.N -> 0.N.0`.
-- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
-- Do not change these rules without explicit confirmation.
-
-## Step-by-step workflow
-
-1. Confirm intent and isolate scope
-- Confirm target version string exactly.
-- Confirm whether user requested local-only or full GitHub flow.
-- Inspect current branch and ensure only release-related files are touched for this task.
-
-2. Update workspace versions
-- Bump `[workspace.package].version` in `Cargo.toml`.
-- Bump internal workspace crate dependency versions in `Cargo.toml`.
-- Update `Cargo.lock` so workspace package versions match target version.
-- Re-scan for partial leftovers.
-
-3. Update release assets
-- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
-- `flake.nix`: update package version to target version.
-- `helm/rustfs/Chart.yaml`:
-- `appVersion` = target version.
-- `version` follows chart mapping rule, for example:
-- `1.0.0-beta.3` -> `0.3.0`
-- `1.0.0-beta.4` -> `0.4.0`
-- `rustfs.spec`:
-- Set `Release` to prerelease suffix (example `beta.4`).
-- Add/update top changelog entry with exact format:
-- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
-- `- Update RPM package to RustFS 1.0.0-beta.4`
-- Changelog identity and time must come from current environment:
-- `git config --get user.name`
-- `git config --get user.email`
-- `date '+%a %b %d %Y'`
-- Changelog version text must match target release version exactly.
-
-4. Verify before shipping
-- Run:
-- `make pre-commit`
-- If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
-
-5. Commit strategy
-- Preferred split when both parts changed:
-- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
-- `chore(release): align release assets for <version>` for docs and packaging files.
-- If user asks for one commit, use one commit.
-- Stage only intended release files; do not include unrelated working tree changes.
-
-6. Push and PR
-- Push branch:
-- `git push -u origin <branch>` (first push), or `git push` (tracking already exists).
-- Create PR with template headings unchanged:
-- `gh pr create --base main --head <branch> --title ... --body-file ...`
-- PR title/body must be English.
-- Use `N/A` for non-applicable template sections.
-- Include verification commands and any `BLOCKED` reason clearly.
-
-## Recommended check commands
-
-- `git status --short --branch`
-- `git diff --name-only origin/main...HEAD`
-- `git diff --stat origin/main...HEAD`
-- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
-- `make pre-commit`
-
-## Output contract
-
-When using this skill, always report:
-
-- Target version.
-- Files changed.
-- Any assumptions or uncertainties requiring confirmation.
-- Verification result (`PASSED` or `BLOCKED`) with key evidence.
-- Commit message(s) used.
-- Push status and PR URL when GitHub flow is requested.

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "RustFS Release Bump"
-  short_description: "Prepare RustFS release branches like PR #2957."
-  default_prompt: "Use $rustfs-release-version-bump to prepare and verify an exact RustFS release-version bump."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -1,40 +0,0 @@
----
-name: security-advisory-lessons
-description: Perform a dedicated RustFS security/advisory review for authn/authz, IAM, RPC trust, paths, secrets, browser isolation, encryption, Object Lock, or other security boundaries. Use only when the user requests a security/advisory review or an adversarial review explicitly escalates to the full advisory map; do not auto-load solely because code touches a sensitive path.
----
-
-# RustFS Security Advisory Lessons
-
-Use this skill as the deep security lens. For a normal adversarial review with a
-matched security surface, the concise security reference under
-`adversarial-validation` is sufficient.
-
-## Workflow
-
-1. Freeze the exact diff/head and identify the changed trust boundaries.
-2. Read [advisory-patterns.md](references/advisory-patterns.md), then apply only
-   the matching sections. Useful headings are
-   auth/admin, IAM/STS/OIDC, policy/plugins, S3/copy/multipart, protocols, paths,
-   secrets/logging/RPC, browser/CORS/proxy, SSE, Object Lock, and serde.
-3. Trace unauthenticated, low-privilege, wrong-action/owner/bucket, malformed,
-   and default-config cases. Security decisions must fail closed.
-4. Require a focused negative regression test for the bypass/exploit form, not
-   only the intended success path. State residual risk when a test is impractical.
-5. Report proven vulnerabilities separately from defense-in-depth hardening.
-
-When advisory currentness matters, fetch the live inventory instead of treating
-the reference as a status mirror:
-
-```bash
-gh api repos/rustfs/rustfs/security-advisories --paginate \
-  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
-```
-
-Fetch an individual advisory only when the live summary indicates a new or
-changed lesson.
-
-## Finding Standard
-
-Each finding includes severity, `file:line`, attacker prerequisites, concrete
-input/path, impact, smallest safe fix, and a regression check. Do not exaggerate
-unauthenticated impact when the actual issue requires authenticated low privilege.

.agents/skills/security-advisory-lessons/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Security Advisory Lessons"
-  short_description: "Apply advisory lessons in reviews."
-  default_prompt: "Use $security-advisory-lessons for a dedicated RustFS security review grounded in past advisories."

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -1,147 +0,0 @@
-# RustFS Advisory Pattern Map
-
-This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
-
-When current advisory state, severity, URLs, or full text matters, fetch it live:
-
-```bash
-gh api repos/rustfs/rustfs/security-advisories --paginate \
-  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
-gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
-```
-
-Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
-
-## Pattern Index
-
-### Admin authorization and route exposure
-
-- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
-- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
-- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
-- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
-- `GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
-- `GHSA-796f-j7xp-hwf4`: `/rustfs/admin/v3/list-remote-targets` checked only that credentials existed and leaked replication target credentials. Lesson: replication target reads are privileged admin operations, and stored remote credentials require strict authz plus response redaction review.
-- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
-
-### IAM import, service accounts, and privilege boundaries
-
-- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
-- `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
-- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
-- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
-
-### STS, OIDC, and federation flows
-
-- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
-- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
-- `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
-- `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
-
-### IAM policy conditions and external policy plugins
-
-- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
-- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
-- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
-
-### S3 object actions, copy, multipart, and upload policy validation
-
-- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
-- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
-- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
-- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
-- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
-
-### Protocol frontends and IAM parity
-
-- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
-- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
-- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
-
-### Filesystem paths and object key traversal
-
-- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
-- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
-- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
-
-### Secrets, defaults, and cryptographic misuse
-
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
-- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
-- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
-- `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
-- `GHSA-m77q-r63m-pj89`: STS JWTs used the root secret key as the shared token signing key, allowing token forgery when the root secret was known. Lesson: STS signing keys need key separation, rotation, and key IDs; do not reuse root credentials for JWT/HMAC signing.
-- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
-
-### Sensitive logging and debug output
-
-- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
-- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
-- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
-
-### RPC input validation and panic safety
-
-- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
-- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
-- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
-
-### Browser, CORS, and console isolation
-
-- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
-- `GHSA-7gcx-wg4x-q9x6`: an incomplete preview fix reintroduced extension-based PDF detection and bypassed the sandboxed fallback for attacker-controlled content. Lesson: browser-surface fixes need regression tests for alternate viewers and file-type branches, and preview trust must come from validated content type plus sandboxing rather than object names.
-- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
-
-### Trusted proxy and source IP conditions
-
-- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
-
-### SSE and on-disk storage invariants
-
-- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
-
-### Object Lock and retention invariants
-
-- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
-
-### Serde deserialization and input validation
-
-- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
-- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
-- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` or clamp.
-- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
-
-## Useful Search Seeds
-
-Use these targeted searches when a diff touches security-sensitive code:
-
-```bash
-rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
-rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
-rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
-rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
-rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
-rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
-rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
-rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
-rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
-rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
-rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
-```
-
-## Minimum Regression Test Expectations
-
-- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
-- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
-- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
-- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
-- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
-- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
-- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
-- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
-- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
-- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.

.agents/skills/test-coverage-improver/SKILL.md

@@ -1,66 +0,0 @@
----
-name: test-coverage-improver
-description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
----
-
-# Test Coverage Improver
-
-Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
-
-## Usage assumptions
-- Focus scope is either changed lines/files, a module, or the whole repository.
-- Coverage artifact must be generated or provided in a supported format.
-- If required context is missing, call out assumptions explicitly before proposing work.
-
-## Workflow
-
-1. Define scope and baseline
-   - Confirm target language, framework, and branch.
-   - Confirm whether the scope is changed files only or full-repo.
-
-2. Produce coverage snapshot
-   - Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
-   - JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
-   - Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
-   - Collect total, per-file, and changed-line coverage.
-
-3. Rank highest-risk gaps
-   - Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
-   - Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
-   - Keep shortlist to 5–8 gaps.
-   - For each gap, capture: file, lines, uncovered branches, and estimated risk score.
-
-4. Propose high-impact tests
-   - For each shortlisted gap, output:
-     - Intent and expected behavior.
-     - Normal, edge, and failure scenarios.
-     - Assertions and side effects to verify.
-     - Setup needs (fixtures, mocks, integration dependencies).
-     - Estimated effort (`S/M/L`).
-
-5. Close with validation plan
-   - State which gaps remain after proposals.
-   - Provide concrete verification command and acceptance threshold.
-   - List assumptions or blockers (environment, fixtures, flaky dependencies).
-
-## Output template
-
-### Coverage Snapshot
-- total / branch coverage
-- changed-file coverage
-- top missing regions by size
-
-### Top Gaps (ranked)
-- `path:line-range` | risk score | why critical
-
-### Test Proposals
-- `path:line-range`
-  - Test name
-  - scenarios
-  - assertions
-  - effort
-
-### Validation Plan
-- command
-- pass criteria
-- remaining risk

.agents/skills/test-coverage-improver/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Test Coverage Improver"
-  short_description: "Find top uncovered risk areas and propose high-impact tests."
-  default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."

.agents/skills/test-coverage-improver/references/coverage-prioritization.md

@@ -1,25 +0,0 @@
-# Coverage Gap Prioritization Guide
-
-Use this rubric for each uncovered area.
-
-Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
-
-- Criticality:
-  - 5: authz/authn, data-loss, payment/consistency path
-  - 4: state mutation, cache invalidation, scheduling
-  - 3: error handling + fallbacks in user-visible flows
-  - 2: parsing/format conversion paths
-  - 1: logging-only or low-impact utilities
-
-- CoverageDebt:
-  - 0: 0–5 uncovered lines
-  - 1: 6–20 uncovered lines
-  - 2: 21–40 uncovered lines
-  - 3: 41+ uncovered lines
-
-- Volatility:
-  - 1: stable legacy code with few recent edits
-  - 2: changed in last 2 releases
-  - 3: touched in last 30 days or currently in active PR
-
-Sort by score descending, then by business impact.

.agents/skills/tier-debug/SKILL.md

@@ -1,34 +0,0 @@
----
-name: tier-debug
-description: Debug ILM tiering / lifecycle transition issues — NoSuchVersion on tier GET, restore failures, xl.meta inspection, remote-tier versionId tracing. Use when investigating tiered/transitioned objects, warm backends, or transition metadata.
----
-
-# Tier / ILM Debugging
-
-Full playbook: [docs/operations/tier-ilm-debugging.md](../../../docs/operations/tier-ilm-debugging.md)
-— read it before changing tier code.
-
-Quick moves:
-
-```bash
-# Inspect transition metadata on disk (one xl.meta per erasure shard disk)
-cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/{bucket}/{object}/xl.meta"
-
-# Trace what versionId is sent to the remote tier
-RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./target/debug/rustfs …
-```
-
-Interpretation:
-
-- `transition_ver_id: <none>` → correct for an unversioned tier bucket; no
-  `versionId` must be sent on tier GET/DELETE.
-- `transition_ver_id: 00000000-…` (nil) → corrupt legacy write-back; readers
-  must filter it out, never send it.
-- Empty-string `transitioned-versionID` metadata under both
-  `x-rustfs-internal-*` and `x-minio-internal-*` keys → object went to an
-  unversioned tier bucket.
-
-Code entry points: `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs`
-(ILM actions), `crates/ecstore/src/services/tier/` (warm backends),
-`crates/filemeta/src/filemeta/version.rs` (metadata read/write + regression
-tests).

.cargo/config.toml

@@ -1,33 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# RustFS Cargo configuration
-
-# NOTE: `--cfg tokio_unstable` is deliberately NOT set here.
-#
-# It used to be a global `[build] rustflags` entry so that the (default-off)
-# `dial9` telemetry feature could compile. That made every build depend on
-# Tokio's unstable, non-semver API, and it broke silently whenever a caller
-# exported their own RUSTFLAGS — an environment RUSTFLAGS replaces the value
-# from this file rather than appending to it.
-#
-# The flag is now scoped to telemetry builds, which opt in explicitly:
-#
-#     make build-profiling
-#     RUSTFLAGS="--cfg tokio_unstable" cargo build --features dial9
-#
-# `crates/obs/build.rs` fails the compile if the `dial9` feature is on without
-# the flag, so the two can no longer drift apart unnoticed.
-#
-# For CPU profiling, add `-C force-frame-pointers=yes` to that RUSTFLAGS value.

.claude/skills

@@ -1 +0,0 @@
-../.agents/skills

.config/e2e-full-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=b4ae71aa894e5c7795ae3eb8116f1777a7601d0f5db3898be2e48faf3329bd9b
-sha256-linux=433debd9d9defa832986269abdf0f1d131597b2d7a417ce930e17c1fd47d85ba

.config/e2e-nightly-selection.txt

@@ -1 +0,0 @@
-sha256=9b9bc336b43b70d0e06e0adb5455bf035bb18945d85d60936eb6fe4d48e0e680

.config/e2e-protocols-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315
-sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17

.config/e2e-repl-nightly-selection.txt

@@ -1 +0,0 @@
-sha256=655a3f3c1d042e694339d15caba7580518320322d1bac0f09450b37e6c09e2e7

.config/e2e-smoke-selection.txt

@@ -1 +0,0 @@
-sha256=ec27cde6ce6400723c4b372bfbd2ac61709c744294e4810af765e8a808d8e31d

.config/make/build-docker-buildx-dev.mak

@@ -1,64 +0,0 @@
-## —— Development/Source builds using direct buildx commands ---------------------------------------
-
-.PHONY: docker-dev
-docker-dev: ## Build dev multi-arch image (cannot load locally)
-	@echo "🏗️ Building multi-architecture development Docker images with buildx..."
-	@echo "💡 This builds from source code and is intended for local development and testing"
-	@echo "⚠️  Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
-	$(DOCKER_CLI) buildx build \
-		--platform linux/amd64,linux/arm64 \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:source-latest \
-		--tag rustfs:dev-latest \
-		.
-
-.PHONY: docker-dev-local
-docker-dev-local: ## Build dev single-arch image (local load)
-	@echo "🏗️ Building single-architecture development Docker image for local use..."
-	@echo "💡 This builds from source code for the current platform and loads locally"
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:source-latest \
-		--tag rustfs:dev-latest \
-		--load \
-		.
-
-.PHONY: docker-dev-push
-docker-dev-push: ## Build and push multi-arch development image # e.g (make docker-dev-push REGISTRY=xxx)
-	@if [ -z "$(REGISTRY)" ]; then \
-		echo "❌ Error: Please specify registry, example: make docker-dev-push REGISTRY=ghcr.io/username"; \
-		exit 1; \
-	fi
-	@echo "🚀 Building and pushing multi-architecture development Docker images..."
-	@echo "💡 Pushing to registry: $(REGISTRY)"
-	$(DOCKER_CLI) buildx build \
-		--platform linux/amd64,linux/arm64 \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag $(REGISTRY)/rustfs:source-latest \
-		--tag $(REGISTRY)/rustfs:dev-latest \
-		--push \
-		.
-
-.PHONY: dev-env-start
-dev-env-start: ## Start development container environment
-	@echo "🚀 Starting development environment..."
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:dev \
-		--load \
-		.
-	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) \
-		-p 9010:9010 -p 9000:9000 \
-		-v $(shell pwd):/workspace \
-		-it rustfs:dev
-
-.PHONY: dev-env-stop
-dev-env-stop: ## Stop development container environment
-	@echo "🛑 Stopping development environment..."
-	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
-
-.PHONY: dev-env-restart
-dev-env-restart: dev-env-stop dev-env-start ## Restart development container environment

.config/make/build-docker-buildx-production.mak

@@ -1,41 +0,0 @@
-## —— Production builds using docker buildx (for CI/CD and production) -----------------------------
-
-.PHONY: docker-buildx
-docker-buildx: ## Build production multi-arch image (no push)
-	@echo "🏗️ Building multi-architecture production Docker images with buildx..."
-	./docker-buildx.sh
-
-.PHONY: docker-buildx-push
-docker-buildx-push: ## Build and push production multi-arch image
-	@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
-	./docker-buildx.sh --push
-
-.PHONY: docker-buildx-version
-docker-buildx-version: ## Build and version production multi-arch image # e.g (make docker-buildx-version VERSION=v1.0.0)
-	@if [ -z "$(VERSION)" ]; then \
-		echo "❌ Error: Please specify version, example: make docker-buildx-version VERSION=v1.0.0"; \
-		exit 1; \
-	fi
-	@echo "🏗️ Building multi-architecture production Docker images (version: $(VERSION))..."
-	./docker-buildx.sh --release $(VERSION)
-
-.PHONY: docker-buildx-push-version
-docker-buildx-push-version: ## Build and version and push production multi-arch image # e.g (make docker-buildx-push-version VERSION=v1.0.0)
-	@if [ -z "$(VERSION)" ]; then \
-		echo "❌ Error: Please specify version, example: make docker-buildx-push-version VERSION=v1.0.0"; \
-		exit 1; \
-	fi
-	@echo "🚀 Building and pushing multi-architecture production Docker images (version: $(VERSION))..."
-	./docker-buildx.sh --release $(VERSION) --push
-
-.PHONY: docker-buildx-production-local
-docker-buildx-production-local: ## Build production single-arch image locally
-	@echo "🏗️ Building single-architecture production Docker image locally..."
-	@echo "💡 Alternative to docker-buildx.sh for local testing"
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_PRODUCTION) \
-		--tag rustfs:production-latest \
-		--tag rustfs:latest \
-		--load \
-		--build-arg RELEASE=latest \
-		.

.config/make/build-docker-production.mak

@@ -1,16 +0,0 @@
-## —— Single Architecture Docker Builds (Traditional) ----------------------------------------------
-
-.PHONY: docker-build-production
-docker-build-production: ## Build single-arch production image
-	@echo "🏗️ Building single-architecture production Docker image..."
-	@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
-	$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
-
-.PHONY: docker-build-source
-docker-build-source: ## Build single-arch source image
-	@echo "🏗️ Building single-architecture source Docker image..."
-	@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
-	DOCKER_BUILDKIT=1 $(DOCKER_CLI) build \
-		--build-arg BUILDKIT_INLINE_CACHE=1 \
-		-f $(DOCKERFILE_SOURCE) -t rustfs:source .
-

.config/make/build-docker.mak

@@ -1,22 +0,0 @@
-## —— Docker-based build (alternative approach) ----------------------------------------------------
-
-# Usage: make BUILD_OS=ubuntu22.04 build-docker
-# Output: target/ubuntu22.04/release/rustfs
-
-.PHONY: build-docker
-build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
-build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
-build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
-build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
-	@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
-	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) --load .
-	$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
-
-.PHONY: docker-inspect-multiarch
-docker-inspect-multiarch: ## Check image architecture support
-	@if [ -z "$(IMAGE)" ]; then \
-		echo "❌ Error: Please specify image, example: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
-		exit 1; \
-	fi
-	@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
-	docker buildx imagetools inspect $(IMAGE)

.config/make/build.mak

@@ -1,74 +0,0 @@
-## —— Local Native Build using build-rustfs.sh script (Recommended) --------------------------------
-
-.PHONY: build
-build: ## Build RustFS binary (includes console by default)
-	@echo "🔨 Building RustFS using build-rustfs.sh script..."
-	./build-rustfs.sh
-
-.PHONY: build-dev
-build-dev: ## Build RustFS in Development mode
-	@echo "🔨 Building RustFS in development mode..."
-	./build-rustfs.sh --dev
-
-.PHONY: build-musl
-build-musl: ## Build x86_64 musl version
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform x86_64-unknown-linux-musl
-
-.PHONY: build-gnu
-build-gnu: ## Build x86_64 GNU version
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
-
-.PHONY: build-musl-arm64
-build-musl-arm64: ## Build aarch64 musl version
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform aarch64-unknown-linux-musl
-
-.PHONY: build-gnu-arm64
-build-gnu-arm64: ## Build aarch64 GNU version
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform aarch64-unknown-linux-gnu
-
-
-## —— Profiling build (dial9 Tokio runtime telemetry) ------------------------------------------
-
-# dial9 hooks Tokio's unstable runtime instrumentation, so it needs
-# `--cfg tokio_unstable`. That flag is deliberately absent from
-# .cargo/config.toml: it is not free, and release binaries do not carry it.
-# Setting RUSTFLAGS here replaces (never appends to) the config-file value, and
-# crates/obs/build.rs fails the build if the feature and the flag disagree.
-#
-# There are no task-dump or S3-upload features — see the notes in
-# crates/obs/Cargo.toml for why.
-DIAL9_FEATURES ?= dial9
-DIAL9_RUSTFLAGS ?= --cfg tokio_unstable
-
-.PHONY: build-profiling
-build-profiling: ## Build RustFS with dial9 Tokio runtime telemetry (diagnostic builds only)
-	@echo "🔬 Building RustFS with dial9 telemetry (features: $(DIAL9_FEATURES))..."
-	@echo "⚠️  Diagnostic build: telemetry writes trace segments to disk continuously."
-	RUSTFLAGS="$(DIAL9_RUSTFLAGS)" cargo build --release --bin rustfs --features $(DIAL9_FEATURES)
-
-.PHONY: build-cross-all
-build-cross-all: core-deps ## Build binaries for all architectures
-	@echo "🔧 Building all target architectures..."
-	@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
-	@echo "🔨 Generating protobuf code..."
-	cargo run --bin gproto || true
-
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
-	./build-rustfs.sh --platform x86_64-unknown-linux-musl
-
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
-	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
-
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
-	./build-rustfs.sh --platform aarch64-unknown-linux-musl
-
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
-	./build-rustfs.sh --platform aarch64-unknown-linux-gnu

.config/make/check.mak

@@ -1,26 +0,0 @@
-## —— Check and Inform Dependencies ----------------------------------------------------------------
-
-# Fatal check
-# Checks all required dependencies and exits with error if not found
-# (e.g., cargo, rustfmt)
-check-%:
-	@command -v $* >/dev/null 2>&1 || { \
-		echo >&2 "❌ '$*' is not installed."; \
-		exit 1; \
-	}
-
-# Warning-only check
-# Checks for optional dependencies and issues a warning if not found
-warn-%:
-	@command -v $* >/dev/null 2>&1 || { \
-		echo >&2 "⚠️ '$*' is not installed."; \
-	}
-
-# For checking dependencies use check-<dep-name> or warn-<dep-name>
-#
-# NOTE: cargo-nextest is a HARD dependency of `make test`, gated inside the
-# test recipe itself (with a RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 escape hatch)
-# rather than a warn-only prerequisite here — see .config/make/tests.mak.
-.PHONY: core-deps fmt-deps
-core-deps: check-cargo ## Check core dependencies
-fmt-deps: check-rustfmt ## Check lint and formatting dependencies

.config/make/coverage.mak

@@ -1,26 +0,0 @@
-## —— Coverage --------------------------------------------------------------------------------------
-
-# Local equivalent of the weekly coverage workflow (.github/workflows/coverage.yml,
-# backlog#1153 infra-5): same measurement scope (--workspace --exclude e2e_test,
-# nextest `ci` profile) and the same per-crate table. Slow — the instrumented
-# build cannot reuse your normal target cache and then runs the whole suite.
-# Doctests are not measured (needs nightly). Outputs land in target/llvm-cov/.
-.PHONY: coverage
-coverage: core-deps ## Workspace line coverage (cargo-llvm-cov + nextest; slow, writes target/llvm-cov/)
-	@if ! command -v cargo-llvm-cov >/dev/null 2>&1; then \
-		echo >&2 "❌ cargo-llvm-cov is required for 'make coverage' but was not found."; \
-		echo >&2 "   Install it with:"; \
-		echo >&2 "     cargo install cargo-llvm-cov --locked"; \
-		echo >&2 "     rustup component add llvm-tools-preview"; \
-		exit 1; \
-	fi
-	@if ! command -v cargo-nextest >/dev/null 2>&1; then \
-		echo >&2 "❌ cargo-nextest is required for 'make coverage' (see 'make test')."; \
-		echo >&2 "   Install it with: cargo install cargo-nextest --locked"; \
-		exit 1; \
-	fi
-	NEXTEST_PROFILE=ci cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
-	@mkdir -p target/llvm-cov
-	cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
-	cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
-	python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json

.config/make/deploy.mak

@@ -1,6 +0,0 @@
-## —— Deploy using dev_deploy.sh script ------------------------------------------------------------
-
-.PHONY: deploy-dev
-deploy-dev: build-musl ## Deploy to dev server
-	@echo "🚀 Deploying to dev server: $${IP}"
-	./scripts/dev_deploy.sh $${IP}

.config/make/help.mak

@@ -1,38 +0,0 @@
-## —— Help, Help Build and Help Docker -------------------------------------------------------------
-
-
-.PHONY: help
-help: ## Shows This Help Menu
-	echo -e "$$HEADER"
-	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} /^## / {printf "\n${green}%s${reset}\n", $$0; next} {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}'
-
-.PHONY: help-build
-help-build: ## Shows RustFS build help
-	@echo ""
-	@echo "💡 build-rustfs.sh script provides more options, smart detection and binary verification"
-	@echo ""
-	@echo "🔧 Direct usage of build-rustfs.sh script:"
-	@echo ""
-	@echo "	./build-rustfs.sh --help                                # View script help"
-	@echo "	./build-rustfs.sh --no-console                          # Build without console resources"
-	@echo "	./build-rustfs.sh --force-console-update                # Force update console resources"
-	@echo "	./build-rustfs.sh --dev                                 # Development mode build"
-	@echo "	./build-rustfs.sh --sign                                # Sign binary files"
-	@echo "	./build-rustfs.sh --platform x86_64-unknown-linux-gnu   # Specify target platform"
-	@echo "	./build-rustfs.sh --skip-verification                   # Skip binary verification"
-	@echo ""
-
-.PHONY: help-docker
-help-docker: ## Shows docker environment and suggestion help
-	@echo ""
-	@echo "📋 Environment Variables:"
-	@echo "	REGISTRY                 Image registry address (required for push)"
-	@echo "	DOCKERHUB_USERNAME       Docker Hub username"
-	@echo "	DOCKERHUB_TOKEN          Docker Hub access token"
-	@echo "	GITHUB_TOKEN             GitHub access token"
-	@echo ""
-	@echo "💡 Suggestions:"
-	@echo "	Production use:          Use docker-buildx* commands (based on precompiled binaries)"
-	@echo "	Local development:       Use docker-dev* commands (build from source)"
-	@echo "	Development environment: Use dev-env-* commands to manage dev containers"
-	@echo ""

.config/make/lint-fmt.mak

@@ -1,91 +0,0 @@
-## —— Code quality and Formatting ------------------------------------------------------------------
-
-.NOTPARALLEL: fix
-
-.PHONY: fmt
-fmt: core-deps fmt-deps ## Format code
-	@echo "🔧 Formatting code..."
-	cargo fmt --all
-
-.PHONY: fmt-check
-fmt-check: core-deps fmt-deps ## Check code formatting
-	@echo "📝 Checking code formatting..."
-	cargo fmt --all --check
-
-.PHONY: clippy-check
-clippy-check: core-deps ## Run clippy checks
-	@echo "🔍 Running clippy checks..."
-	cargo clippy --all-targets -- -D warnings
-
-.PHONY: clippy-fix
-clippy-fix: core-deps ## Apply clippy fixes
-	@echo "🔧 Applying clippy fixes..."
-	cargo clippy --fix --allow-dirty
-
-.PHONY: fix
-fix: fmt clippy-fix ## Format code and apply clippy fixes
-
-.PHONY: quick-check
-quick-check: core-deps ## Run fast workspace compilation check
-	@echo "🔨 Running fast compilation check..."
-	cargo check --workspace --exclude e2e_test
-
-.PHONY: unsafe-code-check
-unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
-	@echo "🔒 Checking unsafe_code allowances..."
-	./scripts/check_unsafe_code_allowances.sh
-
-.PHONY: architecture-migration-check
-architecture-migration-check: ## Check architecture migration guardrails
-	@echo "🏗️ Checking architecture migration guardrails..."
-	./scripts/check_architecture_migration_rules.sh
-
-.PHONY: logging-guardrails-check
-logging-guardrails-check: ## Check logging guardrails for redaction and noise regressions
-	@echo "🪵 Checking logging guardrails..."
-	./scripts/check_logging_guardrails.sh
-
-.PHONY: tokio-io-uring-check
-tokio-io-uring-check: ## Check tokio io-uring runtime feature stays removed
-	@echo "🚫 Checking tokio io-uring feature guard..."
-	./scripts/check_no_tokio_io_uring.sh
-
-.PHONY: extension-schema-check
-extension-schema-check: ## Check extension-schema stays a lightweight contract crate
-	@echo "🧩 Checking extension schema boundaries..."
-	./scripts/check_extension_schema_boundaries.sh
-
-.PHONY: body-cache-whitelist-check
-body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fail-closed allow-list
-	@echo "🧱 Checking body-cache whitelist guard..."
-	./scripts/check_body_cache_whitelist.sh
-
-.PHONY: s3s-footprint-check
-s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
-	@echo "📦 Checking s3s footprint ratchet..."
-	./scripts/check_s3s_footprint.sh
-
-.PHONY: fips-wording-check
-fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
-	@echo "📣 Checking cryptographic capability wording guard..."
-	./scripts/check_fips_wording.sh
-
-.PHONY: embedded-secrets-check
-embedded-secrets-check: ## Check no private key material or credential literal is committed
-	@echo "🔑 Checking embedded secret material guard..."
-	./scripts/check_embedded_secrets.sh
-
-.PHONY: test-wiring-check
-test-wiring-check: ## Check tests stay registered and selected by their intended runners
-	@echo "🧪 Checking test wiring..."
-	python3 ./scripts/check_test_wiring.py
-
-.PHONY: log-analyzer-rules-check
-log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
-	@echo "🩺 Checking log-analyzer rule anchors..."
-	./scripts/check_log_analyzer_rules.sh
-
-.PHONY: compilation-check
-compilation-check: core-deps ## Run compilation check
-	@echo "🔨 Running compilation check..."
-	cargo check --all-targets

.config/make/pre-commit.mak

@@ -1,31 +0,0 @@
-## —— Pre Commit Checks ----------------------------------------------------------------------------
-
-.NOTPARALLEL: pre-commit pre-pr dev-check
-
-.PHONY: setup-hooks
-setup-hooks: ## Set up git hooks
-	@echo "🔧 Setting up git hooks..."
-	chmod +x .git/hooks/pre-commit
-	@echo "✅ Git hooks setup complete!"
-
-.PHONY: doc-paths-check
-doc-paths-check: ## Check that instruction/architecture docs reference existing file paths
-	@echo "📄 Checking doc path references..."
-	./scripts/check_doc_paths.sh
-
-.PHONY: planning-docs-check
-planning-docs-check: ## Check that no planning-type documents are committed
-	@echo "📄 Checking for committed planning docs..."
-	./scripts/check_no_planning_docs.sh
-
-.PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
-	@echo "✅ All pre-commit checks passed!"
-
-.PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
-	@echo "✅ All pre-PR checks passed!"
-
-.PHONY: dev-check
-dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
-	@echo "✅ Fast development checks passed!"

.config/make/tests.mak

@@ -1,86 +0,0 @@
-## —— Tests and e2e test ---------------------------------------------------------------------------
-
-TEST_THREADS ?= 1
-
-# cargo-nextest is a HARD dependency of `make test`.
-#
-# nextest changes test semantics vs plain `cargo test`: it runs every test in
-# its own process (so serial_test's #[serial] mutex does not serialize across
-# tests) and it is the only runner that honours .config/nextest.toml
-# [test-groups] (e.g. the ecstore-serial-flaky serialization guard). CI runs
-# nextest (.github/actions/setup/action.yml installs it), so a silent fallback
-# to `cargo test` would run with different serialization behaviour than CI and
-# mask (or invent) flakes.
-#
-# Installing cargo-nextest:
-#   cargo install cargo-nextest --locked          # from source
-#   # or a prebuilt binary (faster) via the taiki-e installer / get.nexte.st:
-#   #   https://nexte.st/docs/installation/
-#
-# Escape hatch: set RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to run the plain
-# `cargo test` fallback anyway. Results are NOT authoritative — semantics
-# differ from CI and .config/nextest.toml test-groups will NOT apply.
-
-.PHONY: script-tests
-script-tests: ## Run shell script tests
-	@echo "Running script tests..."
-	./scripts/test_build_rustfs_options.sh
-	./scripts/test_docker_runtime_timezone.sh
-	./scripts/test_entrypoint_credentials.sh
-	./scripts/test_internode_grpc_ab_bench.sh
-	./scripts/test_object_batch_bench_enhanced.sh
-	./scripts/test_hotpath_warp_ab_gate.sh
-	./scripts/test_hotpath_warp_abba.sh
-	./scripts/test_exact_1mib_handoff_abba.sh
-	./scripts/test_pinned_paired_abba_bench.sh
-	./scripts/test_manual_transition_runbooks.sh
-	./scripts/check_embedded_secrets.sh --self-test
-	python3 ./scripts/check_test_wiring.py --self-test
-	python3 ./scripts/s3-tests/test_report_compat.py
-	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
-	python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
-	./scripts/validate_object_data_cache_cold_stampede.sh --self-test
-
-.PHONY: test
-test: core-deps script-tests ## Run all tests (needs cargo-nextest; RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to override)
-	@echo "🧪 Running tests..."
-	@if command -v cargo-nextest >/dev/null 2>&1; then \
-		cargo nextest run --all --exclude e2e_test; \
-	elif [ "$${RUSTFS_ALLOW_CARGO_TEST_FALLBACK:-0}" = "1" ]; then \
-		echo >&2 "⚠️  ============================================================================"; \
-		echo >&2 "⚠️  cargo-nextest NOT found — running the 'cargo test' fallback (opt-in)."; \
-		echo >&2 "⚠️  TEST SEMANTICS DIFFER FROM CI; results are NOT authoritative:"; \
-		echo >&2 "⚠️    * nextest runs each test in its own process; 'cargo test' does not,"; \
-		echo >&2 "⚠️      so serial_test #[serial] serialization behaves differently."; \
-		echo >&2 "⚠️    * .config/nextest.toml [test-groups] will NOT apply (e.g. the"; \
-		echo >&2 "⚠️      ecstore-serial-flaky group), so load-sensitive tests may flake here"; \
-		echo >&2 "⚠️      but pass on CI (or vice versa)."; \
-		echo >&2 "⚠️  Install cargo-nextest and re-run before trusting these results."; \
-		echo >&2 "⚠️  ============================================================================"; \
-		cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
-	else \
-		echo >&2 "❌ cargo-nextest is required for 'make test' but was not found."; \
-		echo >&2 ""; \
-		echo >&2 "   RustFS tests run under cargo-nextest (process-per-test isolation)."; \
-		echo >&2 "   CI runs nextest and .config/nextest.toml [test-groups] only take effect"; \
-		echo >&2 "   under nextest. Plain 'cargo test' has different serialization semantics"; \
-		echo >&2 "   and is NOT a faithful substitute."; \
-		echo >&2 ""; \
-		echo >&2 "   Install it with either:"; \
-		echo >&2 "     cargo install cargo-nextest --locked"; \
-		echo >&2 "   or a prebuilt binary (faster) — see https://nexte.st/docs/installation/"; \
-		echo >&2 ""; \
-		echo >&2 "   To run the plain 'cargo test' fallback anyway (results NOT authoritative;"; \
-		echo >&2 "   serialization semantics differ from CI), re-run with:"; \
-		echo >&2 "     RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test"; \
-		exit 1; \
-	fi
-	cargo test --all --doc
-
-.PHONY: e2e-server
-e2e-server: ## Run e2e-server tests
-	sh $(shell pwd)/scripts/run.sh
-
-.PHONY: probe-e2e
-probe-e2e: ## Probe e2e tests
-	sh $(shell pwd)/scripts/probe.sh

.config/migration-gate-floor.txt

@@ -1,10 +0,0 @@
-# Committed floor for the number of tests selected by the migration-critical
-# CI gate (see scripts/check_migration_gate_count.sh, backlog#1153 infra-12).
-#
-# The floor equals the exact count of rustfs-ecstore --lib tests matching the
-# gate filter (name substrings: data_movement, rebalance, decommission,
-# source_cleanup, delete_marker) at the time this file was last updated.
-# CI fails if the selected count drops below this number, so renames or
-# removals that thin the gate must update this file in the same PR.
-# Adding tests does not require a bump, but bumping keeps the guard tight.
-571

.config/nextest.toml

@@ -1,447 +0,0 @@
-# nextest configuration for RustFS.
-#
-# Serialize the ecstore tests that share the process-wide disk registry or
-# exercise a multi-disk commit handoff across nextest process boundaries.
-#
-#   * store::bucket::tests::bucket_delete_* share process/global state (disk
-#     registry, lock client) and race make_bucket into InsufficientWriteQuorum
-#     when run concurrently with other ecstore tests.
-#   * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
-#     uses the shared multipart fixture and a deterministic uploadId-lock
-#     handoff, so it must not overlap another process mutating that fixture.
-#   * bucket::metadata_sys::tests::concurrent_config_writes_from_separate_nodes_do_not_lose_writes
-#     uses the shared transaction lock and must not overlap other ecstore tests.
-#
-# serial_test's #[serial] attribute does NOT serialize these across runs:
-# nextest executes each test in its own process, where the in-process
-# serial_test mutex has no effect. A nextest test-group with max-threads = 1 is
-# the mechanism that actually serializes across nextest's process boundary.
-#
-# ---------------------------------------------------------------------------
-# Profiles
-# ---------------------------------------------------------------------------
-# The `default` profile is what local `cargo nextest run` uses. It NEVER
-# retries: a red test locally means a real failure to investigate, not noise to
-# paper over. The `ci` profile (below) is the strict CI gate: global
-# retries = 0 so a new race's first occurrence is never masked, plus a
-# narrowly-scoped quarantine list (retries = 2) for tests with a tracked OPEN
-# flake issue. Flake policy lives in docs/testing/README.md.
-
-[test-groups]
-ecstore-serial-flaky = { max-threads = 1 }
-embedded-test-ports = { max-threads = 1 }
-e2e-vault = { max-threads = 1 }
-
-# Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
-# server and manipulate its disk directories at runtime (crates/e2e_test:
-# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13, and
-# replacement_privileged_e2e_test when explicitly run as root on Linux). They
-# are correct in isolation but resource-heavy; serialize them under nextest's
-# process boundary (serial_test's #[serial] does not cross it) so several 4-disk
-# servers never run at once. The e2e-full merge/main lane picks these up;
-# they are deliberately NOT in the fast PR `e2e-smoke` filter.
-e2e-reliability = { max-threads = 1 }
-e2e-inline-boundaries = { max-threads = 1 }
-e2e-cluster-nightly = { max-threads = 1 }
-
-# --- default profile (local): serialize the flaky groups, never retry --------
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
-# each spawns a 4-disk hermetic erasure set and drives full staged-upload +
-# commit + GET cycles — the same cross-disk-commit IO shape that made
-# concurrent_resend load-sensitive. Preventive serialization only, no retries.
-# The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
-test-group = 'ecstore-serial-flaky'
-
-# The production-handler relocation regression builds an isolated 8-disk,
-# 2-pool store and commits a 72 MiB multipart object. Keep that cross-disk IO
-# from overlapping the ecstore commit fixtures above.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Embedded integration-test binaries discover an ephemeral port and release
-# the probe listener before RustFS binds it. Serialize that cross-process
-# TOCTOU window; retries would only hide real startup failures.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
-# Serialize the durable manual-transition checkpoint test across nextest's
-# process boundary; it mutates bucket lifecycle metadata and is not quarantined.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the bucket-incarnation / lifecycle-fence tests. They drive
-# init_bucket_metadata_sys and bucket_metadata_sys_of, i.e. process-global
-# OnceLock state that serial_test's #[serial] cannot protect across nextest's
-# process boundary, and they delete+recreate buckets — the same shape that
-# raced into InsufficientWriteQuorum in backlog#937. Preventive only, no
-# retries. The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the default-off dst-dir fsync group-commit tests. They use
-# process-global test hooks/registry to deterministically freeze fsync batches;
-# no retries, just one at a time under nextest too.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the 4-disk reliability / degraded-read e2e tests (see the
-# e2e-reliability test-group note above). The matching ci-profile override is at
-# the end of the file, after [profile.ci] is declared.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
-test-group = 'e2e-reliability'
-
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
-test-group = 'e2e-inline-boundaries'
-
-# Vault KMS tests share the fixed dev-server port 8200. serial_test's #[serial]
-# does not cross nextest process boundaries, so keep these tests in one group.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
-test-group = 'e2e-vault'
-
-# ---------------------------------------------------------------------------
-# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
-# ---------------------------------------------------------------------------
-[profile.ci]
-# Strict: a new race must fail on its first occurrence, never be retried away.
-retries = 0
-# Report every failure in one run instead of bailing on the first.
-fail-fast = false
-
-[profile.ci.junit]
-# Emitted to target/nextest/ci/junit.xml; uploaded as a CI artifact.
-# Tests that pass only after a quarantine retry are marked `flaky` here — that
-# marker is the observable signal the flake policy is built around.
-path = "junit.xml"
-
-# ===========================================================================
-# QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
-#
-# RULES (enforced by review, see docs/testing/README.md):
-#   * Every entry MUST link exactly one OPEN issue tracking the flake.
-#   * An entry stays until the issue is fixed (test made robust) or the test is
-#     deleted — 30-day policy. No entry may exist without a live issue link.
-#
-# Each entry also re-declares the `ecstore-serial-flaky` test-group so the
-# serialization holds under the ci profile (nextest evaluates a named
-# profile's own overrides list, not the default profile's).
-# ===========================================================================
-
-# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
-# make_bucket into InsufficientWriteQuorum via shared global state under load.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
-test-group = 'ecstore-serial-flaky'
-retries = 2
-
-# Keep deterministic ECStore write handoffs isolated across nextest processes.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
-test-group = 'ecstore-serial-flaky'
-
-# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
-# on producer/consumer timing windows that stretch past the budget on loaded
-# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
-retries = 2
-
-# Serialize the 4-disk reliability / degraded-read e2e tests under the ci
-# profile too (see the e2e-reliability test-group note near the top). Not a
-# quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently when e2e-full runs the suite.
-[[profile.ci.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
-test-group = 'e2e-reliability'
-
-# Serialize the multipart crash-consistency scenarios under the ci profile too
-# (see the matching default-profile override near the top). Not a quarantine:
-# no retries, just serialized 4-disk cross-disk-commit IO.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
-test-group = 'ecstore-serial-flaky'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Match the default-profile embedded test isolation without quarantining or
-# retrying failures in CI.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
-# Serialize the durable manual-transition checkpoint test under the ci profile
-# too. No retries: failures stay visible.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the bucket-incarnation / lifecycle-fence tests under the ci profile
-# too (see the matching default-profile override near the top). No retries.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
-test-group = 'ecstore-serial-flaky'
-
-# ---------------------------------------------------------------------------
-# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
-# ---------------------------------------------------------------------------
-# PR smoke subset of the e2e_test crate (backlog#1149 ci-4). This profile is
-# the single wiring mechanism for e2e tests in CI: other suites join by
-# extending this filter (or a sibling profile), never by adding ad-hoc e2e
-# jobs to ci.yml. Admission criteria (see crates/e2e_test/README.md): fast,
-# single-node topology, no external dependencies (no awscurl / Vault / fixed
-# ports / pre-started server), no #[ignore].
-#
-# Each e2e test spawns its own rustfs server on a random port with an isolated
-# temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
-#
-# Replication failure harness (backlog#1147 repl-8): the first clause admits
-# its four in-process fake-target self-tests. They bind random loopback ports,
-# use no external service, and finish in under a second.
-#
-# Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
-# FAST bucket-replication tests from replication_extension_test — the
-# target-registration / replication-check / list / remove / delete admin paths
-# that validate config synchronously and never wait for asynchronous
-# replication convergence. Each spawns its own single-node rustfs server(s) on
-# random ports (source, plus an independent single-node target for the pair
-# checks — NOT a cluster), so the subset stays parallel-safe and single-digit
-# seconds. The data-plane tests that poll for convergence and all
-# `_real_dual_node` / `_real_single_node`
-# site-replication tests run in the [profile.e2e-repl-nightly] lane below, NOT
-# here. This allowlist is the single source of truth for the PR/nightly split:
-# the nightly profile derives its set as "the replication module MINUS this
-# allowlist", so any new replication test lands in nightly by default (never
-# silently unrun) until it is explicitly blessed as fast here. Keep the two
-# regexes byte-identical. The committed profile selection digests make changes
-# visible in CI; current counts live in docs/testing/e2e-suite-inventory.md.
-# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
-# (#4724) because they set a loopback (127.0.0.1) replication target that the
-# SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed —
-# the guard now honours an off-by-default opt-in and this suite's source servers
-# set it (RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) — so the allowlist below is
-# restored.
-#
-# Security negative-auth subset (backlog#1151 sec-5): the three attacker-facing
-# S3 auth-rejection suites join the first clause above by module name —
-# presigned_negative (sec-2), negative_sigv4 (sec-1, header SigV4), and
-# admin_auth (sec-4, admin gate + root-credential lifecycle). All three use
-# RustFSTestEnvironment on a random port and are parallel-safe, so they meet the
-# smoke admission criteria unchanged. This is the wiring step that makes those
-# merged suites actually execute on every PR (they were dead until listed here).
-# A rename that drops any of them out of this filter would silently thin the
-# security gate with no CI signal, so scripts/check_security_smoke_count.sh owns
-# a count-floor guard over exactly this subset (infra-12 mechanism, floor in
-# .config/security-smoke-floor.txt), invoked from the e2e-tests job in ci.yml.
-# NOT here by topology: the GHSA-3p3x FTPS/WebDAV constant-time e2e
-# (protocols::test_protocol_core_suite) binds fixed ports and needs the
-# ftps,webdav features, so it cannot join this random-port, default-feature
-# profile; its GHSA-r5qv sibling is a unit test that already runs in the
-# test-and-lint `--all --exclude e2e_test` pass. See
-# docs/testing/security-regressions.md for the full CI-execution map.
-#
-# ILM tiering main path (backlog#1148 ilm-7): the `reliant::tiering::` clause
-# admits the hermetic transition e2e. Like the fast replication pair checks it
-# spawns a second independent single-node server (the cold RustFS tier), not a
-# cluster, so it keeps the lane's parallel-safe / no-external-dependency
-# properties. The RustFS warm backend has no loopback guard (that guard is
-# replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
-#
-# Disk compression (backlog#1848): the `compression` module joins the smoke
-# lane so the multipart disk-compression roundtrips (restored after
-# rustfs/rustfs#5169 disabled them) have PR-lane signal, not just merge-gate.
-# Single-node servers on random ports with isolated temp dirs — meets the
-# admission criteria unchanged.
-[profile.e2e-smoke]
-default-filter = """
-    package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
-        | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
-        | test(/^reliant::lifecycle::/)
-        | test(/^reliant::tiering::/)
-    )
-"""
-fail-fast = false
-
-[profile.e2e-smoke.junit]
-path = "junit.xml"
-
-# The pagination boundary cases can stall when a server/listing regression
-# prevents the continuation request from completing. Keep the timeout scoped
-# to those known failure modes so legitimate lifecycle/tiering waits retain
-# their test-level timing budget.
-[[profile.e2e-smoke.overrides]]
-filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
-slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
-
-# ---------------------------------------------------------------------------
-# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
-# ---------------------------------------------------------------------------
-# backlog#1147 repl-1 (deps: ci-4). Runs the SLOW / cross-process replication
-# tests that are unfit for the per-PR e2e-smoke gate:
-#
-#   * 2 remote-target TLS validation tests.
-#   * 15 bucket-replication data-plane/helper tests — they PUT/delete objects
-#     and poll until source and target converge; two replicate over HTTPS,
-#     six pin SSE replication contracts (managed SSE-S3/SSE-KMS re-encrypt on
-#     the target incl. multipart and the resync path, SSE-C and
-#     target-without-KMS stay fail-closed), and one guards event/history
-#     observers.
-#   * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
-#     servers and drives the cross-process site-replication control plane.
-#   * 1 `_real_three_node` site-replication test.
-#   * 1 `_real_single_node` service-account round-trip test.
-#
-# The set is defined as "everything in replication_extension_test that is NOT
-# in the e2e-smoke PR allowlist above" (the negated clause is byte-identical to
-# the allowlist), so a newly added replication test automatically runs here
-# until it is explicitly promoted to the fast PR subset — no replication test
-# is ever silently left out of CI.
-#
-# #[serial] does NOT serialize under nextest (process-per-test; see the file
-# header). These tests need no cross-test serialization: each spawns its own
-# server(s) on random ports with isolated temp dirs, so they are parallel-safe
-# by construction — the same property the e2e-smoke subset relies on. If load
-# on the runner surfaces a real flake, quarantine the specific test with an
-# OPEN issue link (ci-10 / backlog#937 policy), never blanket-retry or exclude.
-#
-# Wired by .github/workflows/e2e-replication-nightly.yml (schedule +
-# workflow_dispatch), which builds the rustfs binary once, installs awscurl so
-# the STS dual-node test actually exercises its path (it skips gracefully with
-# a visible log line when awscurl is absent), and routes scheduled failures
-# through .github/actions/schedule-failure-issue (ci-8). Explicit division of
-# labor with e2e-full: these tests run only in the consolidated nightly
-# workflow, not in the merge/main lane.
-[profile.e2e-repl-nightly]
-default-filter = """
-    package(e2e_test)
-    & test(/^replication_extension_test::/)
-    & !test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
-"""
-fail-fast = false
-
-[profile.e2e-repl-nightly.junit]
-# Emitted to target/nextest/e2e-repl-nightly/junit.xml; uploaded by the nightly
-# workflow as the failure-triage artifact.
-path = "junit.xml"
-
-# ---------------------------------------------------------------------------
-# e2e-nightly profile — destructive multi-process cluster fault domains
-# ---------------------------------------------------------------------------
-# These seven modules are deliberately outside e2e-full's merge budget. Each
-# starts a real multi-process or multi-disk topology and exercises node/disk
-# loss, quorum, cleanup, notification fan-in, or admin-timeout behavior. The
-# consolidated nightly workflow runs them serially to avoid resource
-# starvation; failures are never retried.
-[profile.e2e-nightly]
-default-filter = """
-    package(e2e_test)
-    & test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
-"""
-fail-fast = false
-
-[profile.e2e-nightly.junit]
-path = "junit.xml"
-
-[[profile.e2e-nightly.overrides]]
-filter = 'package(e2e_test)'
-test-group = 'e2e-cluster-nightly'
-
-# ---------------------------------------------------------------------------
-# e2e-protocols profile — serial protocol lane
-# ---------------------------------------------------------------------------
-# The suite owns fixed ports, so the nightly workflow runs this exact profile
-# with one nextest worker.
-[profile.e2e-protocols]
-default-filter = 'package(e2e_test) & test(/^protocols::/)'
-fail-fast = false
-
-[profile.e2e-protocols.junit]
-path = "junit.xml"
-
-# ---------------------------------------------------------------------------
-# e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5)
-# ---------------------------------------------------------------------------
-# The merge gate (ci.yml `e2e-full` job: push main + merge_group +
-# workflow_dispatch). Runs the user-visible KMS, object-lock, multipart-auth,
-# quota, checksum, encryption,
-# security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
-# skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
-# --profile e2e-full` (see docs/testing/e2e-suite-inventory.md).
-#
-# The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
-#   * protocols:: — FTPS/SFTP/WebDAV, run from the dedicated protocol profile
-#     with one worker because the suite owns fixed ports.
-#   * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
-#     (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
-#     namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
-#     object_lambda) — too heavy for the merge budget; they run in the
-#     e2e-nightly serial cluster-fault lane.
-#   * replication_extension_test — repl-1 already splits it into the PR
-#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (55 slow) lanes and reserves
-#     it for those, so e2e-full does not double-run it.
-#   * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
-#     manual-localhost:9000 reliant/policy tests are ci-13's migration.
-#
-# Each e2e test spawns its own single-node rustfs server on a random port with
-# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
-# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
-# 4-disk reliability / degraded-read fault-injection tests and the fixed-port
-# Vault tests, both serialized below.
-# KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
-# 341 ran / 32 failed on the suites' first automated run ever). Deterministic
-# product failures cannot be quarantined away with retries, so each family is
-# excluded here with its tracking issue, under the same discipline as the
-# ci-profile quarantine (docs/testing/README.md): every entry MUST cite one
-# OPEN issue, and the fixing PR MUST delete the exclusion. The passing
-# negative-path siblings of each family stay in as regression guards.
-#   * rustfs#4843 — over-limit archive entry paths hard-reject the whole
-#     archive even under ignore-errors semantics.
-[profile.e2e-full]
-default-filter = """
-    package(e2e_test)
-    & !test(/^protocols::/)
-    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
-    & !test(/^replication_extension_test::/)
-    & !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
-    & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
-"""
-fail-fast = false
-
-[profile.e2e-full.junit]
-# Emitted to target/nextest/e2e-full/junit.xml; uploaded by the e2e-full job.
-path = "junit.xml"
-
-# Serialize the 4-disk reliability / degraded-read e2e tests under e2e-full too
-# (see the e2e-reliability test-group note near the top of this file). Not a
-# quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently.
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
-test-group = 'e2e-reliability'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
-test-group = 'e2e-inline-boundaries'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
-test-group = 'e2e-vault'

.config/security-smoke-floor.txt

@@ -1,12 +0,0 @@
-# Committed floor for the number of security negative-auth tests selected by the
-# e2e-smoke PR profile (see scripts/check_security_smoke_count.sh, backlog#1151
-# sec-5).
-#
-# The floor equals the exact count of e2e_test cases whose name starts with a
-# security module prefix (negative_sigv4_test, presigned_negative_test,
-# admin_auth_test) that the [profile.e2e-smoke] default-filter in
-# .config/nextest.toml selects, at the time this file was last updated. CI fails
-# if the selected count drops below this number, so a rename or removal that
-# thins the security smoke gate must update this file in the same PR.
-# Adding tests does not require a bump, but bumping keeps the guard tight.
-16

.docker/README.md

@@ -1,131 +0,0 @@
-# RustFS Docker Infrastructure
-
-This directory contains the complete Docker infrastructure for building, deploying, and monitoring RustFS. It provides ready-to-use configurations for development, testing, and production-grade observability.
-
-## 📂 Directory Structure
-
-| Directory | Description | Status |
-| :--- | :--- | :--- |
-| **[`observability/`](observability/README.md)** | **[RECOMMENDED]** Full-stack observability (Prometheus, Grafana, Tempo, Loki). | ✅ Production-Ready |
-| **[`compose/`](compose/README.md)** | Specialized setups (e.g., 4-node distributed cluster testing). | ⚠️ Testing Only |
-| **[`mqtt/`](mqtt/README.md)** | EMQX Broker configuration for MQTT integration testing. | 🧪 Development |
-| **[`openobserve-otel/`](openobserve-otel/README.md)** | Alternative lightweight observability stack using OpenObserve. | 🔄 Alternative |
-
----
-
-## 📄 Root Directory Files
-
-The following files in the project root are essential for Docker operations:
-
-### Build Scripts & Dockerfiles
-
-| File | Description | Usage |
-| :--- | :--- | :--- |
-| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
-| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
-| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
-| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
-
-### Docker Compose Configurations
-
-| File | Description | Usage |
-| :--- | :--- | :--- |
-| **`docker-compose.yml`** | **Main Development Setup**<br>Comprehensive setup with profiles for development, observability, and proxying. | `docker compose up -d`<br>`docker compose --profile observability up -d` |
-| **`docker-compose-simple.yml`** | **Quick Start Setup**<br>Minimal configuration running a single RustFS instance with 4 volumes. Perfect for first-time users. | `docker compose -f docker-compose-simple.yml up -d` |
-
----
-
-## 🌟 Observability Stack (Recommended)
-
-Located in: [`.docker/observability/`](observability/README.md)
-
-We provide a comprehensive, industry-standard observability stack designed for deep insights into RustFS performance. This is the recommended setup for both development and production monitoring.
-
-### Components
-- **Metrics**: Prometheus (Collection) + Grafana (Visualization)
-- **Traces**: Tempo (Storage) + Jaeger (UI)
-- **Logs**: Loki
-- **Ingestion**: OpenTelemetry Collector
-
-### Key Features
-- **Full Persistence**: All metrics, logs, and traces are saved to Docker volumes, ensuring no data loss on restarts.
-- **Correlation**: Seamlessly jump between Logs, Traces, and Metrics in Grafana.
-- **High Performance**: Optimized configurations for batching, compression, and memory management.
-
-### Quick Start
-```bash
-cd .docker/observability
-docker compose up -d
-```
-
----
-
-## 🧪 Specialized Environments
-
-Located in: [`.docker/compose/`](compose/README.md)
-
-These configurations are tailored for specific testing scenarios that require complex topologies.
-
-### Distributed Cluster (4-Nodes)
-Simulates a real-world distributed environment with 4 RustFS nodes running locally.
-```bash
-docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
-```
-
-### Integrated Observability Test
-A self-contained environment running 4 RustFS nodes alongside the full observability stack. Useful for end-to-end testing of telemetry.
-```bash
-docker compose -f .docker/compose/docker-compose.observability.yaml up -d
-```
-
----
-
-## 📡 MQTT Integration
-
-Located in: [`.docker/mqtt/`](mqtt/README.md)
-
-Provides an EMQX broker for testing RustFS MQTT features.
-
-### Quick Start
-```bash
-cd .docker/mqtt
-docker compose up -d
-```
-- **Dashboard**: [http://localhost:18083](http://localhost:18083) (Default: `admin` / `public`)
-- **MQTT Port**: `1883`
-
----
-
-## 👁️ Alternative: OpenObserve
-
-Located in: [`.docker/openobserve-otel/`](openobserve-otel/README.md)
-
-For users preferring a lightweight, all-in-one solution, we support OpenObserve. It combines logs, metrics, and traces into a single binary and UI.
-
-### Quick Start
-```bash
-cd .docker/openobserve-otel
-docker compose up -d
-```
-
----
-
-## 🔧 Common Operations
-
-### Cleaning Up
-To stop all containers and remove volumes (**WARNING**: deletes all persisted data):
-```bash
-docker compose down -v
-```
-
-### Viewing Logs
-To follow logs for a specific service:
-```bash
-docker compose logs -f [service_name]
-```
-
-### Checking Status
-To see the status of all running containers:
-```bash
-docker compose ps
-```

.docker/compose/README.md

@@ -1,102 +0,0 @@
-# Specialized Docker Compose Configurations
-
-This directory contains specialized Docker Compose configurations for specific testing scenarios.
-
-## ⚠️ Important Note
-
-**For Observability:**
-We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
-
-The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.
-
-## 📁 Configuration Files
-
-### Cluster Testing
-
-- **`docker-compose.cluster.yaml`**
-  - **Purpose**: Simulates a 4-node RustFS distributed cluster.
-  - **Use Case**: Testing distributed storage logic, consensus, and failover.
-  - **Nodes**: 4 RustFS instances.
-  - **Storage**: Uses local HTTP endpoints.
-
-### Legacy / Minimal Observability
-
-- **`docker-compose.observability.yaml`**
-  - **Purpose**: A minimal observability setup.
-  - **Status**: **Deprecated**. Please use `../observability/docker-compose.yml` instead.
-
-## 🚀 Usage Examples
-
-### Cluster Testing
-
-To start a 4-node cluster for distributed testing:
-
-```bash
-# From project root
-docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
-```
-
-### Script-Based 4-Node Validation (Recommended)
-
-Use the local validation script when you need local-source image build, failover checks,
-and benchmark workflow in one command:
-
-```bash
-# Default mode: WAIT_PROBE_MODE=service
-# This avoids false negatives where /health/ready remains 503 locally
-# while the service path is already available.
-./scripts/run_four_node_cluster_failover_bench.sh
-```
-
-Strict mode is available when you explicitly want `/health/ready == 200` as the gate:
-
-```bash
-WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
-```
-
-### Profiling + Trace Validation
-
-The profiling-focused 4-node compose keeps profiling enabled and points RustFS
-to an OTLP/HTTP collector endpoint:
-
-```bash
-docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
-```
-
-Important behavior notes:
-
-- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
-  traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
-- Startup usually produces logs and metrics first. That does not guarantee
-  visible traces yet.
-- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
-- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
-  many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
-  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.
-
-Minimal trace verification flow:
-
-```bash
-# 1. Start the profiling compose with richer span visibility.
-RUSTFS_OBS_LOGGER_LEVEL=debug \
-docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
-
-# 2. Generate real request traffic after startup.
-curl -I http://127.0.0.1:9000/health
-curl -I http://127.0.0.1:9000/health/ready
-
-# 3. Then inspect Tempo or Jaeger.
-# Grafana: http://localhost:3000
-# Jaeger:  http://localhost:16686
-```
-
-If logs and metrics are present but traces are sparse, the most common cause is
-"no real request traffic yet" or "`info` level filtered nested spans", not an
-OTLP routing failure.
-
-### (Deprecated) Minimal Observability
-
-```bash
-# From project root
-docker compose -f .docker/compose/docker-compose.observability.yaml up -d
-```

.docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml

@@ -1,236 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Profiling-first 4-node local-build compose.
-#
-# Goals:
-# - force linux/amd64 runtime/build on Apple Silicon hosts;
-# - enable RustFS built-in CPU profiling;
-# - keep all tuning knobs host-overridable via env.
-#
-# Observability notes:
-# - `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS appends
-#   `/v1/traces`, `/v1/metrics`, and `/v1/logs` automatically.
-# - Logs and metrics usually appear during startup. Traces mainly appear after
-#   real HTTP/S3/gRPC requests create spans.
-# - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request trace span but
-#   filters many `debug`-level nested spans. Use `debug` when validating trace
-#   richness rather than collector reachability.
-
-services:
-  node1:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node1
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node1_data_0:/data/rustfs0
-      - node1_data_1:/data/rustfs1
-      - node1_data_2:/data/rustfs2
-      - node1_data_3:/data/rustfs3
-    ports:
-      - "9000:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node2:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node2
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node2_data_0:/data/rustfs0
-      - node2_data_1:/data/rustfs1
-      - node2_data_2:/data/rustfs2
-      - node2_data_3:/data/rustfs3
-    ports:
-      - "9001:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node3:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node3
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node3_data_0:/data/rustfs0
-      - node3_data_1:/data/rustfs1
-      - node3_data_2:/data/rustfs2
-      - node3_data_3:/data/rustfs3
-    ports:
-      - "9002:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node4:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node4
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node4_data_0:/data/rustfs0
-      - node4_data_1:/data/rustfs1
-      - node4_data_2:/data/rustfs2
-      - node4_data_3:/data/rustfs3
-    ports:
-      - "9003:9000"
-    networks:
-      - rustfs-cluster-net
-
-volumes:
-  node1_data_0:
-  node1_data_1:
-  node1_data_2:
-  node1_data_3:
-  node2_data_0:
-  node2_data_1:
-  node2_data_2:
-  node2_data_3:
-  node3_data_0:
-  node3_data_1:
-  node3_data_2:
-  node3_data_3:
-  node4_data_0:
-  node4_data_1:
-  node4_data_2:
-  node4_data_3:
-
-networks:
-  rustfs-cluster-net:
-    driver: bridge

.docker/compose/docker-compose.cluster.local-build.yml

@@ -1,220 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  node1:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node1
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node1_data_0:/data/rustfs0
-      - node1_data_1:/data/rustfs1
-      - node1_data_2:/data/rustfs2
-      - node1_data_3:/data/rustfs3
-    ports:
-      - "9000:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node2:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node2
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node2_data_0:/data/rustfs0
-      - node2_data_1:/data/rustfs1
-      - node2_data_2:/data/rustfs2
-      - node2_data_3:/data/rustfs3
-    ports:
-      - "9001:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node3:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node3
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node3_data_0:/data/rustfs0
-      - node3_data_1:/data/rustfs1
-      - node3_data_2:/data/rustfs2
-      - node3_data_3:/data/rustfs3
-    ports:
-      - "9002:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node4:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node4
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node4_data_0:/data/rustfs0
-      - node4_data_1:/data/rustfs1
-      - node4_data_2:/data/rustfs2
-      - node4_data_3:/data/rustfs3
-    ports:
-      - "9003:9000"
-    networks:
-      - rustfs-cluster-net
-
-volumes:
-  node1_data_0:
-  node1_data_1:
-  node1_data_2:
-  node1_data_3:
-  node2_data_0:
-  node2_data_1:
-  node2_data_2:
-  node2_data_3:
-  node3_data_0:
-  node3_data_1:
-  node3_data_2:
-  node3_data_3:
-  node4_data_0:
-  node4_data_1:
-  node4_data_2:
-  node4_data_3:
-
-networks:
-  rustfs-cluster-net:
-    driver: bridge

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-latency-stable.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-throughput-max.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60

.docker/compose/docker-compose.cluster.perf-round1-linux.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40

.docker/compose/docker-compose.cluster.perf-round1.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30

.docker/compose/docker-compose.cluster.yaml

@@ -1,82 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  node0:
-    image: rustfs/rustfs:latest # Replace with your image name and label
-    container_name: node0
-    hostname: node0
-    environment:
-      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-    platform: linux/amd64
-    ports:
-      - "9000:9000" # Map port 9001 of the host to port 9000 of the container
-    volumes:
-      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
-    command: "/app/rustfs"
-
-  node1:
-    image: rustfs/rustfs:latest
-    container_name: node1
-    hostname: node1
-    environment:
-      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-    platform: linux/amd64
-    ports:
-      - "9001:9000" # Map port 9002 of the host to port 9000 of the container
-    volumes:
-      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
-    command: "/app/rustfs"
-
-  node2:
-    image: rustfs/rustfs:latest
-    container_name: node2
-    hostname: node2
-    environment:
-      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-    platform: linux/amd64
-    ports:
-      - "9002:9000" # Map port 9003 of the host to port 9000 of the container
-    volumes:
-      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
-    command: "/app/rustfs"
-
-  node3:
-    image: rustfs/rustfs:latest
-    container_name: node3
-    hostname: node3
-    environment:
-      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-    platform: linux/amd64
-    ports:
-      - "9003:9000" # Map port 9004 of the host to port 9000 of the container
-    volumes:
-      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
-    command: "/app/rustfs"

.docker/compose/docker-compose.observability.yaml

@@ -1,282 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  # --- Observability Stack ---
-
-  tempo-init:
-    image: busybox:latest
-    command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
-    volumes:
-      - tempo-data:/var/tempo
-    user: root
-    networks:
-      - rustfs-network
-    restart: "no"
-
-  tempo:
-    image: grafana/tempo:2.10.5
-    user: "10001"
-    command: [ "-config.file=/etc/tempo.yaml" ]
-    volumes:
-      - ../../.docker/observability/tempo.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200" # tempo
-      - "4317"      # otlp grpc
-      - "4318"      # otlp http
-      - "7946"      # memberlist
-    restart: unless-stopped
-    networks:
-      - rustfs-network
-    depends_on:
-      tempo-init:
-        condition: service_completed_successfully
-    healthcheck:
-      test: [ "CMD", "/tempo", "-version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
-    environment:
-      - TZ=Asia/Shanghai
-    volumes:
-      - ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
-    ports:
-      - "1888:1888"   # pprof
-      - "8888:8888"   # Prometheus metrics for Collector
-      - "8889:8889"   # Prometheus metrics for application indicators
-      - "13133:13133" # health check
-      - "4317:4317"   # OTLP gRPC
-      - "4318:4318"   # OTLP HTTP
-      - "55679:55679" # zpages
-    networks:
-      - rustfs-network
-    depends_on:
-      - tempo
-      - jaeger
-      - prometheus
-      - loki
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/otelcol-contrib", "--version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  jaeger:
-    image: jaegertracing/jaeger:latest
-    environment:
-      - TZ=Asia/Shanghai
-      - SPAN_STORAGE_TYPE=badger
-      - BADGER_EPHEMERAL=false
-      - BADGER_DIRECTORY_VALUE=/badger/data
-      - BADGER_DIRECTORY_KEY=/badger/key
-      - COLLECTOR_OTLP_ENABLED=true
-    volumes:
-      - ../../.docker/observability/jaeger.yaml:/etc/jaeger/config.yml:ro
-      - jaeger-data:/badger
-    ports:
-      - "16686:16686" # Web UI
-      - "14269:14269" # Admin/Metrics
-      - "4317"        # otlp grpc
-      - "4318"        # otlp http
-    command: [ "--config", "/etc/jaeger/config.yml" ]
-    networks:
-      - rustfs-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  prometheus:
-    image: prom/prometheus:latest
-    environment:
-      - TZ=Asia/Shanghai
-    volumes:
-      - ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
-      - ../../.docker/observability/prometheus-rules:/etc/prometheus/rules:ro
-      - prometheus-data:/prometheus
-    ports:
-      - "9090:9090"
-    command:
-      - '--config.file=/etc/prometheus/prometheus.yml'
-      - '--web.enable-otlp-receiver'
-      - '--web.enable-remote-write-receiver'
-      - '--enable-feature=promql-experimental-functions'
-      - '--storage.tsdb.path=/prometheus'
-      - '--storage.tsdb.retention.time=30d'
-    networks:
-      - rustfs-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  loki:
-    image: grafana/loki:latest
-    environment:
-      - TZ=Asia/Shanghai
-    volumes:
-      - ../../.docker/observability/loki.yaml:/etc/loki/loki.yaml:ro
-      - loki-data:/loki
-    ports:
-      - "3100:3100"
-    command: -config.file=/etc/loki/loki.yaml
-    networks:
-      - rustfs-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
-      interval: 15s
-      timeout: 10s
-      retries: 5
-      start_period: 60s
-
-  pyroscope:
-    image: grafana/pyroscope:latest
-    ports:
-      - "4040:4040"
-    command:
-      - -self-profiling.disable-push=true
-    networks:
-      - rustfs-network
-    restart: unless-stopped
-
-  grafana:
-    image: grafana/grafana:latest
-    ports:
-      - "3000:3000" # Web UI
-    environment:
-      - GF_SECURITY_ADMIN_PASSWORD=admin
-      - GF_SECURITY_ADMIN_USER=admin
-      - TZ=Asia/Shanghai
-      - GF_INSTALL_PLUGINS=grafana-pyroscope-datasource
-      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
-    networks:
-      - rustfs-network
-    volumes:
-      - ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
-      - ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
-      - grafana-data:/var/lib/grafana
-    depends_on:
-      - prometheus
-      - tempo
-      - loki
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- RustFS Cluster ---
-
-  node1:
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    container_name: node1
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - RUSTFS_OBS_LOGGER_LEVEL=debug
-    platform: linux/amd64
-    ports:
-      - "9001:9000"
-    networks:
-      - rustfs-network
-    depends_on:
-      - otel-collector
-
-  node2:
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    container_name: node2
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - RUSTFS_OBS_LOGGER_LEVEL=debug
-    platform: linux/amd64
-    ports:
-      - "9002:9000"
-    networks:
-      - rustfs-network
-    depends_on:
-      - otel-collector
-
-  node3:
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    container_name: node3
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - RUSTFS_OBS_LOGGER_LEVEL=debug
-    platform: linux/amd64
-    ports:
-      - "9003:9000"
-    networks:
-      - rustfs-network
-    depends_on:
-      - otel-collector
-
-  node4:
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    container_name: node4
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - RUSTFS_OBS_LOGGER_LEVEL=debug
-    platform: linux/amd64
-    ports:
-      - "9004:9000"
-    networks:
-      - rustfs-network
-    depends_on:
-      - otel-collector
-
-volumes:
-  prometheus-data:
-  tempo-data:
-  loki-data:
-  jaeger-data:
-  grafana-data:
-
-networks:
-  rustfs-network:
-    driver: bridge
-    name: "network_rustfs_config"
-    driver_opts:
-      com.docker.network.enable_ipv6: "true"

.docker/mqtt/README.md

@@ -1,30 +0,0 @@
-# MQTT Broker (EMQX)
-
-This directory contains the configuration for running an EMQX MQTT broker, which can be used for testing RustFS's MQTT integration.
-
-## 🚀 Quick Start
-
-To start the EMQX broker:
-
-```bash
-docker compose up -d
-```
-
-## 📊 Access
-
-- **Dashboard**: [http://localhost:18083](http://localhost:18083)
-- **Default Credentials**: `admin` / `public`
-- **MQTT Port**: `1883`
-- **WebSocket Port**: `8083`
-
-## 🛠️ Configuration
-
-The `docker-compose.yml` file sets up a single-node EMQX instance.
-
-- **Persistence**: Data is not persisted by default (for testing).
-- **Network**: Uses the default bridge network.
-
-## 📝 Notes
-
-- This setup is intended for development and testing purposes.
-- For production deployments, please refer to the official [EMQX Documentation](https://www.emqx.io/docs/en/latest/).

.docker/mqtt/config/emqx.conf

@@ -1,37 +0,0 @@
-# 节点配置
-node.name = "emqx@127.0.0.1"
-node.cookie = "aBcDeFgHiJkLmNoPqRsTuVwXyZ012345"
-node.data_dir = "/opt/emqx/data"
-
-# 日志配置
-log.console = {level = info, enable = true}
-log.file = {path = "/opt/emqx/log/emqx.log", enable = true, level = info}
-
-# MQTT TCP 监听器
-listeners.tcp.default = {bind = "0.0.0.0:1883", max_connections = 1000000, enable = true}
-
-# MQTT SSL 监听器
-listeners.ssl.default = {bind = "0.0.0.0:8883", enable = false}
-
-# MQTT WebSocket 监听器
-listeners.ws.default = {bind = "0.0.0.0:8083", enable = true}
-
-# MQTT WebSocket SSL 监听器
-listeners.wss.default = {bind = "0.0.0.0:8084", enable = false}
-
-# 管理控制台
-dashboard.listeners.http = {bind = "0.0.0.0:18083", enable = true}
-
-# HTTP API
-management.listeners.http = {bind = "0.0.0.0:8081", enable = true}
-
-# 认证配置
-authentication = [
-  {enable = true, mechanism = password_based, backend = built_in_database, user_id_type = username}
-]
-
-# 授权配置
-authorization.sources = [{type = built_in_database, enable = true}]
-
-# 持久化消息存储
-message.storage.backend = built_in_database

.docker/mqtt/config/vm.args

@@ -1,9 +0,0 @@
--name emqx@127.0.0.1
--setcookie aBcDeFgHiJkLmNoPqRsTuVwXyZ012345
-+P 2097152
-+t 1048576
-+zdbbl 32768
--kernel inet_dist_listen_min 6000
--kernel inet_dist_listen_max 6100
--smp enable
--mnesia dir "/opt/emqx/data/mnesia"

.docker/mqtt/docker-compose-more.yml

@@ -1,74 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  emqx:
-    image: emqx/emqx:latest
-    container_name: emqx
-    restart: unless-stopped
-    environment:
-      - EMQX_NODE__NAME=emqx@127.0.0.1
-      - EMQX_NODE__COOKIE=aBcDeFgHiJkLmNoPqRsTuVwXyZ012345
-      - EMQX_NODE__DATA_DIR=/opt/emqx/data
-      - EMQX_LOG__CONSOLE__LEVEL=info
-      - EMQX_LOG__CONSOLE__ENABLE=true
-      - EMQX_LOG__FILE__PATH=/opt/emqx/log/emqx.log
-      - EMQX_LOG__FILE__LEVEL=info
-      - EMQX_LOG__FILE__ENABLE=true
-      - EMQX_LISTENERS__TCP__DEFAULT__BIND=0.0.0.0:1883
-      - EMQX_LISTENERS__TCP__DEFAULT__MAX_CONNECTIONS=1000000
-      - EMQX_LISTENERS__TCP__DEFAULT__ENABLE=true
-      - EMQX_LISTENERS__SSL__DEFAULT__BIND=0.0.0.0:8883
-      - EMQX_LISTENERS__SSL__DEFAULT__ENABLE=false
-      - EMQX_LISTENERS__WS__DEFAULT__BIND=0.0.0.0:8083
-      - EMQX_LISTENERS__WS__DEFAULT__ENABLE=true
-      - EMQX_LISTENERS__WSS__DEFAULT__BIND=0.0.0.0:8084
-      - EMQX_LISTENERS__WSS__DEFAULT__ENABLE=false
-      - EMQX_DASHBOARD__LISTENERS__HTTP__BIND=0.0.0.0:18083
-      - EMQX_DASHBOARD__LISTENERS__HTTP__ENABLE=true
-      - EMQX_MANAGEMENT__LISTENERS__HTTP__BIND=0.0.0.0:8081
-      - EMQX_MANAGEMENT__LISTENERS__HTTP__ENABLE=true
-      - EMQX_AUTHENTICATION__1__ENABLE=true
-      - EMQX_AUTHENTICATION__1__MECHANISM=password_based
-      - EMQX_AUTHENTICATION__1__BACKEND=built_in_database
-      - EMQX_AUTHENTICATION__1__USER_ID_TYPE=username
-      - EMQX_AUTHORIZATION__SOURCES__1__TYPE=built_in_database
-      - EMQX_AUTHORIZATION__SOURCES__1__ENABLE=true
-    ports:
-      - "1883:1883"    # MQTT TCP
-      - "8883:8883"    # MQTT SSL
-      - "8083:8083"    # MQTT WebSocket
-      - "8084:8084"    # MQTT WebSocket SSL
-      - "18083:18083"  # Web 管理控制台
-      - "8081:8081"    # HTTP API
-    volumes:
-      - ./data:/opt/emqx/data
-      - ./log:/opt/emqx/log
-      - ./config:/opt/emqx/etc
-    networks:
-      - mqtt-net
-    healthcheck:
-      test: [ "CMD", "/opt/emqx/bin/emqx_ctl", "status" ]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-    logging:
-      driver: "json-file"
-      options:
-        max-size: "100m"
-        max-file: "3"
-
-networks:
-  mqtt-net:
-    driver: bridge

.docker/mqtt/docker-compose.yml

@@ -1,29 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  emqx:
-    image: emqx/emqx:latest
-    container_name: emqx
-    ports:
-      - "1883:1883"
-      - "8083:8083"
-      - "8084:8084"
-      - "8883:8883"
-      - "18083:18083"
-    restart: unless-stopped
-
-networks:
-  default:
-    driver: bridge

.docker/nginx/nginx.conf

@@ -1,93 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-worker_processes auto;
-pid /var/run/nginx.pid;
-
-events {
-    worker_connections 1024;
-}
-
-http {
-    include /etc/nginx/mime.types;
-    default_type application/octet-stream;
-
-    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
-                    '$status $body_bytes_sent "$http_referer" '
-                    '"$http_user_agent" "$http_x_forwarded_for"';
-
-    access_log /var/log/nginx/access.log main;
-    error_log /var/log/nginx/error.log warn;
-
-    sendfile on;
-    keepalive_timeout 65;
-
-    # RustFS Server Block
-    server {
-        listen 80;
-        server_name localhost;
-
-        # Redirect HTTP to HTTPS (optional, uncomment if SSL is configured)
-        # return 301 https://$host$request_uri;
-
-        location / {
-            proxy_pass http://rustfs:9000;
-            proxy_set_header Host $host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto $scheme;
-
-            # S3 specific headers
-            proxy_set_header X-Amz-Date $http_x_amz_date;
-            proxy_set_header Authorization $http_authorization;
-
-            # Disable buffering for large uploads
-            proxy_request_buffering off;
-            client_max_body_size 0;
-        }
-
-        location /rustfs/console {
-            proxy_pass http://rustfs:9001;
-            proxy_set_header Host $host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto $scheme;
-        }
-    }
-
-    # SSL Configuration (Example)
-    # server {
-    #     listen 443 ssl;
-    #     server_name localhost;
-    #
-    #     ssl_certificate /etc/nginx/ssl/server.crt;
-    #     ssl_certificate_key /etc/nginx/ssl/server.key;
-    #
-    #     # Restrict to modern TLS versions and ciphers. Operators copying this
-    #     # example must keep at least these directives — without them, nginx
-    #     # may negotiate older protocol versions that have known weaknesses.
-    #     ssl_protocols TLSv1.2 TLSv1.3;
-    #     ssl_prefer_server_ciphers on;
-    #     ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
-    #     ssl_session_timeout 1d;
-    #     ssl_session_cache shared:SSL:10m;
-    #     ssl_session_tickets off;
-    #     # add_header Strict-Transport-Security "max-age=63072000" always;
-    #
-    #     location / {
-    #         proxy_pass http://rustfs:9000;
-    #         ...
-    #     }
-    # }
-}

.docker/observability/.gitignore

@@ -1,5 +0,0 @@
-jaeger-data/*
-loki-data/*
-prometheus-data/*
-tempo-data/*
-grafana-data/*

.docker/observability/README.md

@@ -1,221 +0,0 @@
-# RustFS Observability Stack
-
-This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.
-
-## Components
-
-The stack is composed of the following best-in-class open-source components:
-
-- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
-- **Grafana** (v11.1.0): The leading platform for observability visualization.
-- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
-- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
-- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
-- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
-
-By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
-If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
-
-## Architecture
-
-1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
-2. **Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
-    - **Traces** -> **Tempo** (Primary) & **Jaeger** (Secondary/Optional)
-    - **Metrics** -> **Prometheus** (via scraping the Collector's exporter)
-    - **Logs** -> **Loki**
-3. **Visualization**: **Grafana** connects to all backends (Prometheus, Tempo, Loki, Jaeger) to provide a unified dashboard experience.
-
-## Features
-
-- **Full Persistence**: All data (Metrics, Logs, Traces) is persisted to Docker volumes, ensuring no data loss on restart.
-- **Correlation**: Seamless navigation between Metrics, Logs, and Traces in Grafana.
-  - Jump from a Metric spike to relevant Traces.
-  - Jump from a Trace to relevant Logs.
-- **High Performance**: Optimized configurations for batching, compression, and memory management.
-- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
-
-## GET Performance Optimization Dashboards
-
-Three pre-built Grafana dashboards are included for monitoring RustFS GET performance optimization rollout:
-
-### Available Dashboards
-
-| Dashboard | File | Description |
-|-----------|------|-------------|
-| **GET Rollout Health** | `grafana-get-rollout-health.json` | Monitors optimization rollout: latency by reader path, early-stop hit rate, codec streaming usage, pipeline failures |
-| **GET Data Integrity** | `grafana-get-data-integrity.json` | Monitors data safety: bitrot verify failures, decode errors, short reads, shard read outcomes |
-| **GET Resource Impact** | `grafana-get-resource-impact.json` | Monitors resource usage: concurrent requests, IO queue utilization, disk permit wait, RSS trend |
-| **Object Data Cache** | `grafana-object-data-cache.json` | Monitors the GET body cache (`rustfs_object_data_cache_*`): hit ratio, lookup/plan/fill outcomes, fill duration quantiles, hit vs fill throughput, entries/weighted bytes, inflight fills, memory-pressure skips, invalidations, and size-class breakdowns |
-
-### Prometheus Alert Rules
-
-The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-configured alerting rules:
-
-| Alert | Severity | Condition |
-|-------|----------|-----------|
-| `GetP99Regression` | Critical | GET p99 latency > 2x baseline for 10m |
-| `PipelineFailureSpike` | Critical | Pipeline failure rate > 5x baseline for 5m |
-| `BitrotMismatchSpike` | Critical | Bitrot mismatch rate > 3x baseline for 5m |
-| `EarlyStopInsufficientQuorum` | Warning | Early-stop insufficient quorum rate > 0.1/s for 5m |
-| `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
-| `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
-
-The file `prometheus-rules/rustfs-kms-alerts.yml` contains alerting rules for the KMS backend operation metrics. Thresholds are conservative defaults pending staging baseline calibration; response procedures live in `docs/operations/kms-observability-runbook.md`, and the matching dashboard is `deploy/observability/grafana/rustfs-kms-observability.json`.
-
-| Alert | Severity | Condition |
-|-------|----------|-----------|
-| `KmsBackendFatalErrors` | Critical | Fatal (non-retryable) attempt failures > 0 for 5m |
-| `KmsBackendHighErrorRate` | Critical | Non-success operation ratio > 5% for 10m (with traffic guard) |
-| `KmsBackendP99LatencyHigh` | Warning | Operation p99 duration (incl. retries) > 2s for 10m |
-| `KmsBackendAttemptFailureSpike` | Warning | Attempt failure rate > 0.5/s for 10m |
-| `KmsBackendRetryBudgetExhausted` | Warning | budget_exhausted / deadline_exceeded outcomes > 0.05/s for 10m |
-
-### Enabling Alert Rules
-
-Add the alert rules file to your Prometheus configuration:
-
-```yaml
-# prometheus.yml
-rule_files:
-  - "/etc/prometheus/rules/*.yml"
-
-# Or mount the file in docker-compose.yml:
-# volumes:
-#   - ./prometheus-rules:/etc/prometheus/rules
-```
-
-### Dashboard Usage
-
-The dashboards are automatically provisioned when Grafana starts. They use the `${DS_PROMETHEUS}` datasource variable, so you need a Prometheus datasource configured in Grafana.
-
-Key panels to monitor during optimization rollout:
-
-1. **GET Latency by Reader Path** - Compare `codec_streaming` vs `legacy_duplex` latency
-2. **Early-Stop Hit Rate** - Verify early-stop is triggering effectively
-3. **Pipeline Failure Rate** - Detect any new failure modes introduced by optimizations
-4. **Bitrot Verify Failures** - Ensure data integrity is maintained
-
-## Quick Start
-
-### Prerequisites
-
-- Docker
-- Docker Compose
-
-### Deploy
-
-Run the following command to start the entire stack:
-
-```bash
-docker compose up -d
-```
-
-### High Availability Tempo
-
-The default `docker-compose.yml` is the single-node stack.
-If you need the Kafka-backed HA Tempo configuration, start it with:
-
-```bash
-docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
-```
-
-### Access Dashboards
-
-| Service        | URL                                              | Credentials       | Description                    |
-| :------------- | :----------------------------------------------- | :---------------- | :----------------------------- |
-| **Grafana**    | [http://localhost:3000](http://localhost:3000)   | `admin` / `admin` | Main visualization hub.        |
-| **Prometheus** | [http://localhost:9090](http://localhost:9090)   | -                 | Metric queries and status.     |
-| **Jaeger UI**  | [http://localhost:16686](http://localhost:16686) | -                 | Secondary trace visualization. |
-| **Tempo**      | [http://localhost:3200](http://localhost:3200)   | -                 | Tempo status/metrics.          |
-
-## Configuration
-
-### Data Persistence
-
-Data is stored in the following Docker volumes:
-
-- `prometheus-data`: Prometheus metrics
-- `tempo-data`: Tempo traces (WAL and Blocks)
-- `loki-data`: Loki logs (Chunks and Rules)
-- `jaeger-data`: Jaeger traces (Badger DB)
-
-To clear all data:
-
-```bash
-docker compose down -v
-```
-
-### Customization
-
-- **Prometheus**: Edit `prometheus.yml` to add scrape targets or alerting rules.
-- **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
-- **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
-
-### Verifying RustFS Traces
-
-When RustFS points `RUSTFS_OBS_ENDPOINT` at this stack, treat the value as the
-OTLP/HTTP base URL, for example:
-
-```bash
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-```
-
-RustFS automatically expands that base URL to:
-
-- `/v1/traces`
-- `/v1/metrics`
-- `/v1/logs`
-
-Important behavior notes:
-
-- Logs and metrics usually appear during startup, so seeing those two signals
-  first is expected.
-- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
-  those attributes as structured metadata, and the Collector also mirrors the
-  common troubleshooting fields into the log line so simple line filters can
-  find them.
-- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
-  startup, because request-path spans are created on demand.
-- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
-  many nested `debug` spans. If Tempo or Jaeger looks sparse, retry with
-  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting collector or Tempo issues.
-
-Minimal validation flow:
-
-```bash
-# 1. Start this observability stack.
-docker compose up -d
-
-# 2. Start RustFS with OTLP/HTTP export and richer span visibility.
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-export RUSTFS_OBS_LOGGER_LEVEL=debug
-
-# 3. Generate real request traffic.
-curl -I http://127.0.0.1:9000/health
-curl -I http://127.0.0.1:9000/health/ready
-
-# 4. Inspect Grafana or Jaeger.
-# Grafana: http://localhost:3000
-# Jaeger:  http://localhost:16686
-```
-
-For a structured RustFS log such as an inter-node RPC authentication failure,
-the Loki line now includes fields such as `event`, `component`, `subsystem`,
-`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
-LogQL checks:
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
-If logs and metrics are present but traces are sparse, the most common cause is
-"no real request traffic yet" or "`info` level filtered nested spans", not an
-OTLP routing failure.
-
-## Troubleshooting
-
-- **Service Health**: Check the health of services using `docker compose ps`.
-- **Logs**: View logs for a specific service using `docker compose logs -f <service_name>`.
-- **Otel Collector**: Check `http://localhost:13133` for health status and `http://localhost:1888/debug/pprof/` for profiling.

.docker/observability/README_ZH.md

@@ -1,211 +0,0 @@
-# RustFS 可观测性技术栈
-
-本目录包含 RustFS 的全面可观测性技术栈，旨在提供对应用程序性能、日志和追踪的深入洞察。
-
-## 组件
-
-该技术栈由以下一流的开源组件组成：
-
-- **Prometheus** (v2.53.1): 行业标准的指标收集和告警工具。
-- **Grafana** (v11.1.0): 领先的可观测性可视化平台。
-- **Loki** (v3.1.0): 水平可扩展、高可用、多租户的日志聚合系统。
-- **Tempo** (v2.5.0): 高吞吐量、最小依赖的分布式追踪后端。
-- **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
-- **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
-
-默认情况下，这套技术栈使用 Tempo 单二进制模式，不依赖 Kafka/Redpanda。
-如果需要基于 Kafka 的 HA Tempo 路径，请使用 `docker-compose-example-for-rustfs.yml` 配合 `docker-compose-tempo-ha-override.yml`。
-
-## 架构
-
-1.  **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
-2.  **处理与导出**: Collector 处理数据（批处理、内存限制）并将其导出到相应的后端：
-    -   **追踪** -> **Tempo** (主要) & **Jaeger** (辅助/可选)
-    -   **指标** -> **Prometheus** (通过抓取 Collector 的导出器)
-    -   **日志** -> **Loki**
-3.  **可视化**: **Grafana** 连接到所有后端（Prometheus, Tempo, Loki, Jaeger），提供统一的仪表盘体验。
-
-## 特性
-
--   **完全持久化**: 所有数据（指标、日志、追踪）都持久化到 Docker 卷，确保重启后无数据丢失。
--   **关联性**: 在 Grafana 中实现指标、日志和追踪之间的无缝导航。
-    -   从指标峰值跳转到相关追踪。
-    -   从追踪跳转到相关日志。
--   **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
--   **标准化协议**: 完全基于 OpenTelemetry 标准构建。
-
-## GET 性能优化仪表盘
-
-包含三个预构建的 Grafana 仪表盘，用于监控 RustFS GET 性能优化发布：
-
-### 可用仪表盘
-
-| 仪表盘 | 文件 | 描述 |
-|--------|------|------|
-| **GET 发布健康度** | `grafana-get-rollout-health.json` | 监控优化发布：按 reader path 的延迟、early-stop 命中率、codec streaming 使用率、pipeline 失败率 |
-| **GET 数据完整性** | `grafana-get-data-integrity.json` | 监控数据安全：bitrot 校验失败、decode 错误、short read、shard 读取结果 |
-| **GET 资源影响** | `grafana-get-resource-impact.json` | 监控资源使用：并发请求数、IO 队列利用率、disk permit 等待、RSS 趋势 |
-| **对象数据缓存** | `grafana-object-data-cache.json` | 监控 GET body 缓存（`rustfs_object_data_cache_*`）：命中率、查找/规划/填充结果、填充耗时分位、命中 vs 填充吞吐、条目数/加权字节、在途填充、内存压力拒绝、失效、按尺寸档拆分 |
-
-### Prometheus 告警规则
-
-文件 `prometheus-rules/rustfs-get-optimization-alerts.yaml` 包含预配置的告警规则：
-
-| 告警 | 级别 | 条件 |
-|------|------|------|
-| `GetP99Regression` | 严重 | GET p99 延迟 > 2x 基线，持续 10 分钟 |
-| `PipelineFailureSpike` | 严重 | Pipeline 失败率 > 5x 基线，持续 5 分钟 |
-| `BitrotMismatchSpike` | 严重 | Bitrot 不匹配率 > 3x 基线，持续 5 分钟 |
-| `EarlyStopInsufficientQuorum` | 警告 | Early-stop quorum 不足率 > 0.1/s，持续 5 分钟 |
-| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线，持续 10 分钟 |
-| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%，持续 5 分钟 |
-
-文件 `prometheus-rules/rustfs-kms-alerts.yml` 包含 KMS 后端操作指标的告警规则。阈值为保守默认值，待 staging 基线校准；响应流程见 `docs/operations/kms-observability-runbook.md`，配套仪表盘为 `deploy/observability/grafana/rustfs-kms-observability.json`。
-
-| 告警 | 级别 | 条件 |
-|------|------|------|
-| `KmsBackendFatalErrors` | 严重 | fatal（不可重试）尝试失败 > 0，持续 5 分钟 |
-| `KmsBackendHighErrorRate` | 严重 | 非 success 操作占比 > 5%，持续 10 分钟（含流量下限保护） |
-| `KmsBackendP99LatencyHigh` | 警告 | 操作 p99 耗时（含重试）> 2s，持续 10 分钟 |
-| `KmsBackendAttemptFailureSpike` | 警告 | 尝试失败率 > 0.5/s，持续 10 分钟 |
-| `KmsBackendRetryBudgetExhausted` | 警告 | budget_exhausted / deadline_exceeded 结果 > 0.05/s，持续 10 分钟 |
-
-### 启用告警规则
-
-在 Prometheus 配置中添加告警规则文件：
-
-```yaml
-# prometheus.yml
-rule_files:
-  - "/etc/prometheus/rules/*.yml"
-
-# 或在 docker-compose.yml 中挂载文件：
-# volumes:
-#   - ./prometheus-rules:/etc/prometheus/rules
-```
-
-### 仪表盘使用
-
-仪表盘在 Grafana 启动时自动预置。它们使用 `${DS_PROMETHEUS}` 数据源变量，因此需要在 Grafana 中配置 Prometheus 数据源。
-
-优化发布期间需要关注的关键面板：
-
-1. **GET 延迟按 Reader Path** - 对比 `codec_streaming` vs `legacy_duplex` 延迟
-2. **Early-Stop 命中率** - 验证 early-stop 是否有效触发
-3. **Pipeline 失败率** - 检测优化引入的新故障模式
-4. **Bitrot 校验失败** - 确保数据完整性
-
-## 快速开始
-
-### 前置条件
-
--   Docker
--   Docker Compose
-
-### 部署
-
-运行以下命令启动整个技术栈：
-
-```bash
-docker compose up -d
-```
-
-### Tempo 高可用模式
-
-默认的 `docker-compose.yml` 对应单机栈。
-如果需要基于 Kafka 的 HA Tempo 配置，请使用：
-
-```bash
-docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
-```
-
-### 访问仪表盘
-
-| 服务 | URL | 凭据 | 描述 |
-| :--- | :--- | :--- | :--- |
-| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | 主要可视化中心。 |
-| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | 指标查询和状态。 |
-| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | 辅助追踪可视化。 |
-| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo 状态/指标。 |
-
-## 配置
-
-### 数据持久化
-
-数据存储在以下 Docker 卷中：
-
--   `prometheus-data`: Prometheus 指标
--   `tempo-data`: Tempo 追踪 (WAL 和 Blocks)
--   `loki-data`: Loki 日志 (Chunks 和 Rules)
--   `jaeger-data`: Jaeger 追踪 (Badger DB)
-
-要清除所有数据：
-
-```bash
-docker compose down -v
-```
-
-### 自定义
-
--   **Prometheus**: 编辑 `prometheus.yml` 以添加抓取目标或告警规则。
--   **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
--   **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
-
-### 验证 RustFS Trace
-
-当 RustFS 将 `RUSTFS_OBS_ENDPOINT` 指向这套技术栈时，应将该值视为
-OTLP/HTTP 的基础 URL，例如：
-
-```bash
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-```
-
-RustFS 会自动在该基础 URL 后补全：
-
-- `/v1/traces`
-- `/v1/metrics`
-- `/v1/logs`
-
-需要注意：
-
-- 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
-- OpenTelemetry bridge 会把 `tracing` 字段作为日志 attributes 发送。Loki 会将这些 attributes 存为 structured metadata，同时 Collector 会把常用排障字段镜像进日志行，方便用简单的行内容过滤直接查到。
-- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
-- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
-  如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
-
-最小验证流程：
-
-```bash
-# 1. 启动本目录下的可观测性技术栈。
-docker compose up -d
-
-# 2. 以 OTLP/HTTP 导出方式启动 RustFS，并提高 span 可见性。
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-export RUSTFS_OBS_LOGGER_LEVEL=debug
-
-# 3. 产生真实请求流量。
-curl -I http://127.0.0.1:9000/health
-curl -I http://127.0.0.1:9000/health/ready
-
-# 4. 到 Grafana 或 Jaeger 中检查。
-# Grafana: http://localhost:3000
-# Jaeger:  http://localhost:16686
-```
-
-对于 RustFS 结构化日志，例如节点间 RPC 鉴权失败，Loki 日志行现在会包含 `event`、`component`、`subsystem`、`failure_reason`、`rpc_service`、`rpc_method`、`expected_audience` 等字段。常用 LogQL 检查：
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
-如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
-“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
-
-## 故障排除
-
--   **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
--   **日志**: 使用 `docker compose logs -f <service_name>` 查看特定服务的日志。
--   **Otel Collector**: 检查 `http://localhost:13133` 获取健康状态，检查 `http://localhost:1888/debug/pprof/` 进行性能分析。

.docker/observability/docker-compose-example-for-rustfs.yml

@@ -1,268 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  rustfs:
-    security_opt:
-      - "no-new-privileges:true"
-    image: rustfs/rustfs:latest
-    container_name: rustfs-server
-    ports:
-      - "9000:9000" # S3 API port
-      - "9001:9001" # Console port
-    environment:
-      - RUSTFS_VOLUMES=/data/rustfs
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_CORS_ALLOWED_ORIGINS=*
-      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
-      - RUSTFS_OBS_LOGGER_LEVEL=info
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
-    volumes:
-      - rustfs-data:/data/rustfs
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test:
-        [
-          "CMD",
-          "sh",
-          "-c",
-          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
-        ]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 40s
-    depends_on:
-      otel-collector:
-        condition: service_started
-
-  rustfs-init:
-    image: alpine
-    container_name: rustfs-init
-    volumes:
-      - rustfs-data:/data
-    networks:
-      - otel-network
-    command: >
-      sh -c "
-        chown -R 10001:10001 /data &&
-        echo 'Volume Permissions fixed' &&
-        exit 0
-      "
-    restart: no
-
-  # --- Tracing ---
-
-  tempo:
-    image: grafana/tempo:latest
-    container_name: tempo
-    command: [ "-config.file=/etc/tempo.yaml" ]
-    volumes:
-      - ./tempo.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200" # tempo
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/tempo", "-version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  redpanda:
-    image: redpandadata/redpanda:latest # for tempo ingest
-    container_name: redpanda
-    ports:
-      - "9092:9092"
-    networks:
-      - otel-network
-    restart: unless-stopped
-    command: >
-      redpanda start --overprovisioned
-      --mode=dev-container
-      --kafka-addr=PLAINTEXT://0.0.0.0:9092
-      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
-
-  jaeger:
-    image: jaegertracing/jaeger:latest
-    container_name: jaeger
-    environment:
-      - SPAN_STORAGE_TYPE=badger
-      - BADGER_EPHEMERAL=false
-      - BADGER_DIRECTORY_VALUE=/badger/data
-      - BADGER_DIRECTORY_KEY=/badger/key
-      - COLLECTOR_OTLP_ENABLED=true
-    volumes:
-      - ./jaeger.yaml:/etc/jaeger/config.yml
-      - jaeger-data:/badger
-    ports:
-      - "16686:16686" # Web UI
-      - "14269:14269" # Admin/Metrics
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-    command: [ "--config", "/etc/jaeger/config.yml" ]
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  # --- Metrics ---
-
-  prometheus:
-    image: prom/prometheus:latest
-    container_name: prometheus
-    volumes:
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
-      - prometheus-data:/prometheus
-    ports:
-      - "9090:9090"
-    command:
-      - "--config.file=/etc/prometheus/prometheus.yml"
-      - "--web.enable-otlp-receiver" # Enable OTLP
-      - "--web.enable-remote-write-receiver" # Enable remote write
-      - "--enable-feature=promql-experimental-functions" # Enable info()
-      - "--storage.tsdb.retention.time=30d"
-    restart: unless-stopped
-    networks:
-      - otel-network
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Logging ---
-
-  loki:
-    image: grafana/loki:latest
-    container_name: loki
-    volumes:
-      - ./loki.yaml:/etc/loki/loki.yaml:ro
-      - loki-data:/loki
-    ports:
-      - "3100:3100"
-    command: -config.file=/etc/loki/loki.yaml
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
-      interval: 15s
-      timeout: 10s
-      retries: 5
-      start_period: 60s
-
-  # --- Collection ---
-
-  otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
-    volumes:
-      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
-    ports:
-      - "1888:1888" # pprof
-      - "8888:8888" # Prometheus metrics for Collector
-      - "8889:8889" # Prometheus metrics for application indicators
-      - "13133:13133" # health check
-      - "4317:4317" # OTLP gRPC
-      - "4318:4318" # OTLP HTTP
-      - "55679:55679" # zpages
-    networks:
-      - otel-network
-    restart: unless-stopped
-    depends_on:
-      - tempo
-      - jaeger
-      - prometheus
-      - loki
-    healthcheck:
-      test: [ "CMD", "/otelcol-contrib", "--version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Profiles ---
-
-  pyroscope:
-    image: grafana/pyroscope:latest
-    container_name: pyroscope
-    ports:
-      - "4040:4040"
-    command:
-      - -self-profiling.disable-push=true
-    networks:
-      - otel-network
-    restart: unless-stopped
-
-  # --- Visualization ---
-
-  grafana:
-    image: grafana/grafana:latest
-    container_name: grafana
-    ports:
-      - "3000:3000"
-    environment:
-      - GF_SECURITY_ADMIN_PASSWORD=admin
-      - GF_SECURITY_ADMIN_USER=admin
-    volumes:
-      - ./grafana/provisioning:/etc/grafana/provisioning:ro
-      - ./grafana/dashboards:/etc/grafana/dashboards:ro
-      - grafana-data:/var/lib/grafana
-    networks:
-      - otel-network
-    restart: unless-stopped
-    depends_on:
-      - prometheus
-      - tempo
-      - loki
-    healthcheck:
-      test:
-        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-volumes:
-  rustfs-data:
-  tempo-data:
-  jaeger-data:
-  prometheus-data:
-  loki-data:
-  grafana-data:
-
-networks:
-  otel-network:
-    driver: bridge
-    name: "network_otel"
-    ipam:
-      config:
-        - subnet: 172.28.0.0/16
-    driver_opts:
-      com.docker.network.enable_ipv6: "true"

.docker/observability/docker-compose-tempo-ha-override.yml

@@ -1,62 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Docker Compose override file for High Availability Tempo setup
-#
-# Usage:
-# docker-compose -f docker-compose-example-for-rustfs.yml \
-#                -f docker-compose-tempo-ha-override.yml up
-
-services:
-  # Override Tempo to use high-availability configuration
-  tempo:
-    volumes:
-      - ./tempo-ha.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200"   # Tempo HTTP
-      - "4317:4317"   # OTLP gRPC
-      - "4318:4318"   # OTLP HTTP
-      - "7946:7946"   # Memberlist
-      - "14250:14250" # Jaeger gRPC
-      - "14268:14268" # Jaeger Thrift HTTP
-      - "9411:9411"   # Zipkin
-    environment:
-      - TEMPO_MEMBERLIST_BIND_PORT=7946
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
-      interval: 10s
-      timeout: 5s
-      retries: 5
-      start_period: 30s
-    depends_on:
-      - redpanda
-
-volumes:
-  tempo-data:
-    driver: local
-    driver_opts:
-      type: tmpfs
-      device: tmpfs
-      o: "size=4g"  # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
-
-# Network configuration remains the same
-# networks:
-#   otel-network:
-#     driver: bridge
-#     name: "network_otel"
-#     ipam:
-#       config:
-#         - subnet: 172.28.0.0/16
-

.docker/observability/docker-compose.yml

@@ -1,211 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-
-  # --- Tracing ---
-  tempo:
-    image: grafana/tempo:2.10.5
-    container_name: tempo
-    command: [ "-config.file=/etc/tempo.yaml" ]
-    volumes:
-      - ./tempo.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200" # tempo
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-      - "7946" # memberlist
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/tempo", "-version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  vulture:
-    image: grafana/tempo-vulture:latest
-    restart: always
-    command:
-      [
-        "-prometheus-listen-address=:8080",
-        "-tempo-query-url=http://tempo:3200",
-        "-tempo-push-url=http://tempo:4317",
-      ]
-    depends_on:
-      - tempo
-
-  jaeger:
-    image: jaegertracing/jaeger:latest
-    container_name: jaeger
-    environment:
-      - SPAN_STORAGE_TYPE=badger
-      - BADGER_EPHEMERAL=false
-      - BADGER_DIRECTORY_VALUE=/badger/data
-      - BADGER_DIRECTORY_KEY=/badger/key
-      - COLLECTOR_OTLP_ENABLED=true
-    volumes:
-      - ./jaeger.yaml:/etc/jaeger/config.yml
-      - jaeger-data:/badger
-    ports:
-      - "16686:16686" # Web UI
-      - "18888:8888" # Metrics
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-    command: [ "--config", "/etc/jaeger/config.yml" ]
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:8888/metrics" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  # --- Metrics ---
-
-  prometheus:
-    image: prom/prometheus:latest
-    container_name: prometheus
-    volumes:
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
-      - ./prometheus-rules:/etc/prometheus/rules:ro
-      - prometheus-data:/prometheus
-    ports:
-      - "9090:9090"
-    command:
-      - "--config.file=/etc/prometheus/prometheus.yml"
-      - "--web.enable-otlp-receiver" # Enable OTLP
-      - "--web.enable-remote-write-receiver" # Enable remote write
-      - "--enable-feature=promql-experimental-functions" # Enable info()
-      - "--storage.tsdb.retention.time=30d"
-    restart: unless-stopped
-    networks:
-      - otel-network
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Logging ---
-
-  loki:
-    image: grafana/loki:latest
-    container_name: loki
-    volumes:
-      - ./loki.yaml:/etc/loki/loki.yaml:ro
-      - loki-data:/loki
-    ports:
-      - "3100:3100"
-    command: -config.file=/etc/loki/loki.yaml
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/usr/bin/loki", "--version" ]
-      interval: 15s
-      timeout: 10s
-      retries: 5
-      start_period: 60s
-
-  # --- Collection ---
-
-  otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
-    volumes:
-      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
-    ports:
-      - "1888:1888" # pprof
-      - "8888:8888" # Prometheus metrics for Collector
-      - "8889:8889" # Prometheus metrics for application indicators
-      - "13133:13133" # health check
-      - "4317:4317" # OTLP gRPC
-      - "4318:4318" # OTLP HTTP
-      - "55679:55679" # zpages
-    networks:
-      - otel-network
-    restart: unless-stopped
-    depends_on:
-      - tempo
-      - jaeger
-      - prometheus
-      - loki
-    healthcheck:
-      test: [ "CMD", "/otelcol-contrib", "--version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Profiles ---
-
-  pyroscope:
-    image: grafana/pyroscope:latest
-    container_name: pyroscope
-    ports:
-      - "4040:4040"
-    command:
-      - -self-profiling.disable-push=true
-    networks:
-      - otel-network
-    restart: unless-stopped
-
-  # --- Visualization ---
-
-  grafana:
-    image: grafana/grafana:latest
-    container_name: grafana
-    ports:
-      - "3000:3000"
-    environment:
-      - GF_SECURITY_ADMIN_PASSWORD=admin
-      - GF_SECURITY_ADMIN_USER=admin
-    volumes:
-      - ./grafana/provisioning:/etc/grafana/provisioning:ro
-      - ./grafana/dashboards:/etc/grafana/dashboards:ro
-      - grafana-data:/var/lib/grafana
-    networks:
-      - otel-network
-    restart: unless-stopped
-    depends_on:
-      - prometheus
-      - tempo
-      - loki
-    healthcheck:
-      test:
-        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-volumes:
-  tempo-data:
-  jaeger-data:
-  prometheus-data:
-  loki-data:
-  grafana-data:
-
-networks:
-  otel-network:
-    driver: bridge
-    name: "network_otel"
-    ipam:
-      config:
-        - subnet: 172.28.0.0/16
-    driver_opts:
-      com.docker.network.enable_ipv6: "true"

.docker/observability/grafana/dashboards/grafana-get-data-integrity.json

@@ -1,500 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max", "sum"]
-        }
-      },
-      "title": "Bitrot Verify Failures",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{stage=\"bitrot_verify\"}[$__rate_interval])) by (path, reason)",
-          "legendFormat": "bitrot_fail {{path}} / {{reason}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"bitrot_verify_failed\"}[$__rate_interval])) by (path, stage)",
-          "legendFormat": "verify_failed {{path}} / {{stage}}",
-          "refId": "B"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Bitrot Verify Duration (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p50",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p95",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p99",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max", "sum"]
-        }
-      },
-      "title": "Decode Errors",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"decode_error\"}[$__rate_interval])) by (path, stage)",
-          "legendFormat": "decode_error {{path}} / {{stage}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max", "sum"]
-        }
-      },
-      "title": "Short Reads",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"short_read\"}[$__rate_interval])) by (path, stage)",
-          "legendFormat": "short_read {{path}} / {{stage}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Shard Read Outcomes by Role",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_shard_read_total[$__rate_interval])) by (path, role, outcome)",
-          "legendFormat": "{{path}} {{role}}/{{outcome}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Request Result Status Rate",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_request_results_total[$__rate_interval])) by (status)",
-          "legendFormat": "{{status}}",
-          "refId": "A"
-        }
-      ]
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["rustfs", "get-optimization", "data-integrity"],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "",
-  "title": "RustFS GET Data Integrity",
-  "uid": "rustfs-get-data-integrity",
-  "version": 1
-}

.docker/observability/grafana/dashboards/grafana-get-performance-attribution.json

@@ -1,716 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_strategy_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{strategy}} / {{mode}}",
-          "refId": "A"
-        }
-      ],
-      "title": "GET Reader Setup Strategy Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_strategy_by_size_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{path}} / {{strategy}} / {{size_bucket}}",
-          "refId": "A"
-        }
-      ],
-      "title": "GET Reader Setup Strategy by Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "none",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "scheduled {{strategy}} / {{mode}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "attempted {{strategy}} / {{mode}}",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "ready {{strategy}} / {{mode}}",
-          "refId": "C"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "deferred {{strategy}} / {{mode}}",
-          "refId": "D"
-        }
-      ],
-      "title": "GET Reader Setup Fanout Average",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "none",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "scheduled {{path}} / {{strategy}} / {{size_bucket}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "ready {{path}} / {{strategy}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Reader Setup Fanout by Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{path}} / {{stage}} / {{size_bucket}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p99 {{path}} / {{stage}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Hot Stage Latency by Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{path}} / {{stage}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p99 {{path}} / {{stage}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Reader, Handler and Body Latency",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 24
-      },
-      "id": 7,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (subpath) (rate(rustfs_io_get_object_direct_memory_subpath_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{subpath}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (subpath, size_bucket) (rate(rustfs_io_get_object_direct_memory_subpath_by_size_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{subpath}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Direct-Memory Subpath Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 24
-      },
-      "id": 8,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_mmap_page_faults_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "mmap {{path}} / {{stage}} / {{kind}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_direct_read_page_faults_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "direct-read {{path}} / {{stage}} / {{kind}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET mmap vs Direct-Read Page Fault Rate",
-      "type": "timeseries"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": [
-    "rustfs",
-    "get",
-    "performance",
-    "attribution"
-  ],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "",
-  "title": "RustFS GET Performance Attribution",
-  "uid": "rustfs-get-performance-attribution",
-  "version": 1,
-  "weekStart": ""
-}

.docker/observability/grafana/dashboards/grafana-get-resource-impact.json

@@ -1,525 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "none",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Concurrent GET Requests",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_get_object_concurrent_requests",
-          "legendFormat": "concurrent GETs",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent",
-          "min": 0,
-          "max": 100,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 70
-              },
-              {
-                "color": "red",
-                "value": 90
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "IO Queue Utilization",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_queue_utilization_percent",
-          "legendFormat": "queue utilization %",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_queue_permits_in_use",
-          "legendFormat": "permits in use",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_queue_permits_available",
-          "legendFormat": "permits available",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Disk Permit Wait Duration (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p50",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p95",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p99",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "bytes",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["lastNotNull"]
-        }
-      },
-      "title": "RSS Trend (Process Memory)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "process_resident_memory_bytes{job=~\"rustfs.*\"}",
-          "legendFormat": "RSS {{instance}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_process_memory_bytes",
-          "legendFormat": "RustFS memory {{instance}}",
-          "refId": "B"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "IO Queue Operations by Priority",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_queue_operations[$__rate_interval])) by (operation, priority)",
-          "legendFormat": "{{operation}} / {{priority}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["sum"]
-        }
-      },
-      "title": "IO Queue Congestion Events",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_queue_congestion_total[$__rate_interval]))",
-          "legendFormat": "congestion events/s",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_timeout_total[$__rate_interval])) by (stage)",
-          "legendFormat": "timeout {{stage}}",
-          "refId": "B"
-        }
-      ]
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["rustfs", "get-optimization", "resource-impact"],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "",
-  "title": "RustFS GET Resource Impact",
-  "uid": "rustfs-get-resource-impact",
-  "version": 1
-}

.docker/observability/grafana/dashboards/grafana-get-rollout-health.json

@@ -1,530 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "axisLabel": "",
-            "axisColorMode": "text",
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "GET Latency by Reader Path (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
-          "legendFormat": "p50 {{path}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
-          "legendFormat": "p95 {{path}}",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
-          "legendFormat": "p99 {{path}}",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percentunit",
-          "min": 0,
-          "max": 1,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.3
-              },
-              {
-                "color": "red",
-                "value": 0.7
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Early-Stop Hit Rate",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_metadata_early_stop_total{decision=\"hit\"}[$__rate_interval])) by (path) / clamp_min(sum(rate(rustfs_io_get_object_metadata_early_stop_total[$__rate_interval])) by (path), 1)",
-          "legendFormat": "hit rate {{path}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percentunit",
-          "min": 0,
-          "max": 1,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Codec Streaming Usage Rate",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])) by (decision) / clamp_min(sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])), 1)",
-          "legendFormat": "codec_streaming {{decision}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])) by (strategy) / clamp_min(sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])), 1)",
-          "legendFormat": "stream_strategy {{strategy}}",
-          "refId": "B"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "bars",
-            "fillOpacity": 80,
-            "lineWidth": 0,
-            "pointSize": 5,
-            "showPoints": "never",
-            "spanNulls": false,
-            "stacking": {
-              "mode": "normal",
-              "group": "A"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["sum"]
-        }
-      },
-      "title": "Pipeline Failure Rate by Stage",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total[$__rate_interval])) by (stage, reason)",
-          "legendFormat": "{{stage}} / {{reason}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "reqps",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "GET Request Rate by Path",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_reader_path_total[$__rate_interval])) by (path)",
-          "legendFormat": "{{path}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "GET Total Latency (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p50",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p95",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p99",
-          "refId": "C"
-        }
-      ]
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["rustfs", "get-optimization"],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "",
-  "title": "RustFS GET Rollout Health",
-  "uid": "rustfs-get-rollout-health",
-  "version": 1
-}

.docker/observability/grafana/dashboards/grafana-put-performance-attribution.json

@@ -1,534 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, eager_status, size_bucket, buffer_bucket, large_concurrency_tuning) (rate(rustfs_s3_put_object_diagnostics_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{path}} / {{eager_status}} / {{size_bucket}} / {{buffer_bucket}} / large={{large_concurrency_tuning}}",
-          "refId": "A"
-        }
-      ],
-      "title": "PUT Diagnostics Decision Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path) (rate(rustfs_s3_put_object_path_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{path}}",
-          "refId": "A"
-        }
-      ],
-      "title": "PUT Path Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ms",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{stage}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p99 {{stage}}",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Hot Stage Latency",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "bytes",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p50 {{path}} / {{size_bucket}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{path}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Selected Buffer Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percentunit",
-          "min": 0,
-          "max": 1,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_s3_put_object_zero_copy_eligible_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "eligible / total",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_s3_put_object_zero_copy_enabled_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "enabled / total",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Zero-Copy Eligibility Ratio",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_s3_put_object_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "PUT total",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (stage) (rate(rustfs_system_storage_erasure_write_quorum_failures_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "write quorum failure {{stage}}",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Throughput and Quorum Failure Signals",
-      "type": "timeseries"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": [
-    "rustfs",
-    "put",
-    "performance",
-    "attribution"
-  ],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "",
-  "title": "RustFS PUT Performance Attribution",
-  "uid": "rustfs-put-performance-attribution",
-  "version": 1,
-  "weekStart": ""
-}

.docker/observability/grafana/provisioning/dashboards/dashboard.yml

@@ -1,25 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-apiVersion: 1
-
-providers:
-  - name: "default"
-    orgId: 1
-    folder: ""
-    type: file
-    disableDeletion: false
-    updateIntervalSeconds: 10
-    options:
-      path: /etc/grafana/dashboards

.docker/observability/grafana/provisioning/datasources/datasources.yaml

@@ -1,98 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-apiVersion: 1
-
-datasources:
-  - name: Prometheus
-    type: prometheus
-    uid: prometheus
-    url: http://prometheus:9090
-    access: proxy
-    isDefault: true
-    editable: false
-    jsonData:
-      httpMethod: GET
-      exemplarTraceIdDestinations:
-        - name: trace_id
-          datasourceUid: tempo
-
-  - name: Tempo
-    type: tempo
-    uid: tempo
-    access: proxy
-    url: http://tempo:3200
-    isDefault: false
-    editable: false
-    jsonData:
-      httpMethod: GET
-      serviceMap:
-        datasourceUid: prometheus
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '-1h'
-        spanEndTimeShift: '1h'
-        filterByTraceID: true
-        filterBySpanID: false
-      tracesToMetrics:
-        datasourceUid: prometheus
-        tags: [ { key: 'service.name' }, { key: 'job' } ]
-        queries:
-          - name: 'Service-Level Latency'
-            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
-          - name: 'Service-Level Calls'
-            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
-          - name: 'Service-Level Errors'
-            query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
-      nodeGraph:
-        enabled: true
-
-  - name: Loki
-    type: loki
-    uid: loki
-    url: http://loki:3100
-    basicAuth: false
-    isDefault: false
-    editable: false
-    jsonData:
-      derivedFields:
-        - datasourceUid: tempo
-          matcherRegex: 'trace_id=(\w+)'
-          name: 'TraceID'
-          url: '$${__value.raw}'
-
-  - name: Jaeger
-    type: jaeger
-    uid: jaeger
-    url: http://jaeger:16686
-    access: proxy
-    isDefault: false
-    editable: false
-    jsonData:
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '1s'
-        spanEndTimeShift: '-1s'
-        filterByTraceID: true
-        filterBySpanID: false
-
-  - name: Pyroscope
-    type: grafana-pyroscope-datasource
-    url: http://pyroscope:4040
-    jsonData:
-      minStep: '15s'

.docker/observability/jaeger.yaml

@@ -1,74 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-service:
-  extensions: [jaeger_storage, jaeger_query]
-  pipelines:
-    traces:
-      receivers: [otlp]
-      processors: [batch]
-      exporters: [jaeger_storage_exporter, spanmetrics]
-    metrics/spanmetrics:
-      receivers: [spanmetrics]
-      exporters: [prometheus]
-  telemetry:
-    resource:
-      service.name: jaeger
-    metrics:
-      level: detailed
-      readers:
-        - pull:
-            exporter:
-              prometheus:
-                host: 0.0.0.0
-                port: 8888
-    logs:
-      level: DEBUG
-
-extensions:
-  jaeger_query:
-    storage:
-      traces: some_storage
-      metrics: some_metrics_storage
-  jaeger_storage:
-    backends:
-      some_storage:
-        memory:
-          max_traces: 100000
-    metric_backends:
-      some_metrics_storage:
-        prometheus:
-          endpoint: http://prometheus:9090
-          normalize_calls: true
-          normalize_duration: true
-
-connectors:
-  spanmetrics:
-
-receivers:
-  otlp:
-    protocols:
-      grpc:
-        endpoint: "0.0.0.0:4317"
-      http:
-        endpoint: "0.0.0.0:4318"
-
-processors:
-  batch:
-
-exporters:
-  jaeger_storage_exporter:
-    trace_storage: some_storage
-  prometheus:
-    endpoint: "0.0.0.0:8889"

.docker/observability/loki.yaml

@@ -1,63 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-auth_enabled: false
-
-server:
-  http_listen_port: 3100
-  grpc_listen_port: 9095
-  log_level: info
-  grpc_server_max_concurrent_streams: 1000
-
-common:
-  instance_addr: 127.0.0.1
-  path_prefix: /loki
-  storage:
-    filesystem:
-      chunks_directory: /loki/chunks
-      rules_directory: /loki/rules
-  replication_factor: 1
-  ring:
-    kvstore:
-      store: inmemory
-
-query_range:
-  results_cache:
-    cache:
-      embedded_cache:
-        enabled: true
-        max_size_mb: 100
-
-schema_config:
-  configs:
-    - from: 2020-10-24
-      store: tsdb
-      object_store: filesystem
-      schema: v13
-      index:
-        prefix: index_
-        period: 24h
-
-limits_config:
-  reject_old_samples: true
-  reject_old_samples_max_age: 168h
-  allow_structured_metadata: true
-  max_line_size: 256KB
-
-pattern_ingester:
-  enabled: true
-  metric_aggregation:
-    loki_address: localhost:3100
-
-frontend:
-  encoding: protobuf

.docker/observability/otel-collector-config.yaml

@@ -1,132 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-receivers:
-  otlp:
-    protocols:
-      grpc:
-        endpoint: 0.0.0.0:4317
-      http:
-        endpoint: 0.0.0.0:4318
-
-processors:
-  batch:
-    timeout: 1s
-    send_batch_size: 1024
-  memory_limiter:
-    check_interval: 1s
-    limit_mib: 1024
-    spike_limit_mib: 256
-  transform/logs:
-    error_mode: ignore
-    log_statements:
-      - context: log
-        statements:
-          - set(attributes["message"], body.string) where IsString(body)
-          - set(attributes["log.body"], body.string) where IsString(body)
-          - set(body, Concat([body, " event=", attributes["event"]], "")) where IsString(body) and attributes["event"] != nil
-          - set(body, Concat([body, " component=", attributes["component"]], "")) where IsString(body) and attributes["component"] != nil
-          - set(body, Concat([body, " subsystem=", attributes["subsystem"]], "")) where IsString(body) and attributes["subsystem"] != nil
-          - set(body, Concat([body, " state=", attributes["state"]], "")) where IsString(body) and attributes["state"] != nil
-          - set(body, Concat([body, " result=", attributes["result"]], "")) where IsString(body) and attributes["result"] != nil
-          - set(body, Concat([body, " reason=", attributes["reason"]], "")) where IsString(body) and attributes["reason"] != nil
-          - set(body, Concat([body, " failure_reason=", attributes["failure_reason"]], "")) where IsString(body) and attributes["failure_reason"] != nil
-          - set(body, Concat([body, " rpc_path=", attributes["rpc_path"]], "")) where IsString(body) and attributes["rpc_path"] != nil
-          - set(body, Concat([body, " rpc_service=", attributes["rpc_service"]], "")) where IsString(body) and attributes["rpc_service"] != nil
-          - set(body, Concat([body, " rpc_method=", attributes["rpc_method"]], "")) where IsString(body) and attributes["rpc_method"] != nil
-          - set(body, Concat([body, " expected_audience=", attributes["expected_audience"]], "")) where IsString(body) and attributes["expected_audience"] != nil
-          - set(body, Concat([body, " peer_addr=", attributes["peer_addr"]], "")) where IsString(body) and attributes["peer_addr"] != nil
-          - set(body, Concat([body, " replay_scope_bootstrap_allowed=", attributes["replay_scope_bootstrap_allowed"]], "")) where IsString(body) and attributes["replay_scope_bootstrap_allowed"] != nil
-          - set(body, Concat([body, " error=", attributes["error"]], "")) where IsString(body) and attributes["error"] != nil
-          - set(body, Concat([body, " exception_message=", attributes["exception.message"]], "")) where IsString(body) and attributes["exception.message"] != nil
-
-exporters:
-  otlp/tempo:
-    endpoint: "tempo:4317"
-    tls:
-      insecure: true
-    compression: gzip
-    retry_on_failure:
-      enabled: true
-      initial_interval: 1s
-      max_interval: 30s
-      max_elapsed_time: 300s
-    sending_queue:
-      enabled: true
-      num_consumers: 10
-      queue_size: 5000
-
-  otlp/jaeger:
-    endpoint: "jaeger:4317"
-    tls:
-      insecure: true
-    compression: gzip
-    retry_on_failure:
-      enabled: true
-      initial_interval: 1s
-      max_interval: 30s
-      max_elapsed_time: 300s
-    sending_queue:
-      enabled: true
-      num_consumers: 10
-      queue_size: 5000
-
-  prometheus:
-    endpoint: "0.0.0.0:8889"
-    send_timestamps: true
-    metric_expiration: 5m
-    resource_to_telemetry_conversion:
-      enabled: true
-
-  otlphttp/loki:
-    endpoint: "http://loki:3100/otlp"
-    tls:
-      insecure: true
-    compression: gzip
-
-extensions:
-  health_check:
-    endpoint: 0.0.0.0:13133
-  pprof:
-    endpoint: 0.0.0.0:1888
-  zpages:
-    endpoint: 0.0.0.0:55679
-
-service:
-  extensions: [ health_check, pprof, zpages ]
-  pipelines:
-    traces:
-      receivers: [ otlp ]
-      processors: [ memory_limiter, batch ]
-      exporters: [ otlp/tempo, otlp/jaeger ]
-    metrics:
-      receivers: [ otlp ]
-      processors: [ batch ]
-      exporters: [ prometheus ]
-    logs:
-      receivers: [ otlp ]
-      processors: [ batch, transform/logs ]
-      exporters: [ otlphttp/loki ]
-  telemetry:
-    logs:
-      level: "info"
-      encoding: "json"
-    metrics:
-      level: "normal"
-      readers:
-        - pull:
-            exporter:
-              prometheus:
-                host: '0.0.0.0'
-                port: 8888

.docker/observability/prometheus-rules/rustfs-dashboard.yml

@@ -1,53 +0,0 @@
-groups:
-  - name: rustfs-dashboard
-    interval: 30s
-    rules:
-      - record: rustfs:http_server_requests:rate5m
-        expr: sum by (job) (rate(rustfs_http_server_requests_total[5m]))
-
-      - record: rustfs:http_server_request_duration_seconds:p50_5m
-        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
-      - record: rustfs:http_server_request_duration_seconds:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
-      - record: rustfs:http_server_request_duration_seconds:p99_5m
-        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
-
-      - record: rustfs:http_server_response_body_size_bytes:p50_5m
-        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
-      - record: rustfs:http_server_response_body_size_bytes:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
-      - record: rustfs:http_server_response_body_size_bytes:p99_5m
-        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
-
-      - record: rustfs:log_cleaner_runs:rate15m
-        expr: sum by (job) (rate(rustfs_log_cleaner_runs_total[15m]))
-      - record: rustfs:log_cleaner_failure_ratio:rate5m
-        expr: sum by (job) (rate(rustfs_log_cleaner_run_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_runs_total[5m])), 1e-9)
-      - record: rustfs:log_cleaner_rotation_failure_ratio:rate5m
-        expr: sum by (job) (rate(rustfs_log_cleaner_rotation_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_rotation_total[5m])), 1e-9)
-      - record: rustfs:log_cleaner_rotation_duration_seconds:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_rotation_duration_seconds_bucket[5m])))
-      - record: rustfs:log_cleaner_compress_duration_seconds:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])))
-
-      - record: rustfs:scanner_objects_scanned:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_objects_scanned_total[5m]))
-      - record: rustfs:scanner_directories_scanned:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_directories_scanned_total[5m]))
-      - record: rustfs:scanner_buckets_scanned:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
-      - record: rustfs:scanner_cycles_success:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))
-
-      - record: rustfs:log_chain_op_event_mismatch:rate5m
-        expr: sum by (job) (rate(rustfs_log_chain_op_event_mismatch_total[5m]))
-
-      - alert: RustFSLogChainOpEventMismatchDetected
-        expr: rustfs:log_chain_op_event_mismatch:rate5m > 0
-        for: 10m
-        labels:
-          severity: warning
-          component: s3-log-chain
-        annotations:
-          summary: "RustFS log-chain op/event mismatch detected"
-          description: "job={{ $labels.job }} has non-zero rustfs_log_chain_op_event_mismatch_total rate for more than 10m. Check s3 op/event mapping changes."

.docker/observability/prometheus-rules/rustfs-get-optimization-alerts.yaml

@@ -1,260 +0,0 @@
-# =============================================================================
-# RustFS GET Optimization — Prometheus Alerting Rules
-# =============================================================================
-#
-# Import into Prometheus:
-#   1. Copy this file to your Prometheus rules directory
-#   2. Add to prometheus.yml:
-#        rule_files:
-#          - "prometheus-alert-rules.yaml"
-#   3. Validate: promtool check rules prometheus-alert-rules.yaml
-#   4. Reload:   curl -X POST http://localhost:9090/-/reload
-#
-# All metric names match those registered in crates/io-metrics/src/lib.rs
-# and documented in crates/ecstore/src/diagnostics/get.rs.
-#
-# Baseline comparison uses "offset 1d" — adjust to "offset 7d" for weekly
-# seasonality if your traffic pattern varies by day of week.
-# =============================================================================
-
-groups:
-  # ==========================================================================
-  # Critical alerts — immediate action required
-  # ==========================================================================
-  - name: rustfs-get-optimization-critical
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 1. GetP99Regression
-      #    GET p99 latency exceeds 2x the baseline (same time yesterday)
-      #    sustained for 10 minutes.
-      #    Action: Roll back the GET optimization immediately.
-      # ------------------------------------------------------------------
-      - alert: GetP99Regression
-        expr: |
-          histogram_quantile(0.99,
-            sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m])) by (le)
-          )
-          >
-          2
-          *
-          histogram_quantile(0.99,
-            sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m] offset 1d)) by (le)
-          )
-        for: 10m
-        labels:
-          severity: critical
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "GET p99 latency regression detected (>2x baseline for 10m)"
-          description: >-
-            The 99th-percentile GET object latency is {{ $value | humanizeDuration }}
-            which is more than double the baseline measured 24 hours ago.
-            This indicates a severe performance regression introduced by
-            a recent GET optimization change.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/get-p99-regression"
-          action: >
-            1. Verify the regression is not caused by external factors
-               (disk health, network, load spike).
-            2. If confirmed optimization-related, roll back:
-               - Set RUSTFS_GET_CODEC_STREAMING=0
-               - Set RUSTFS_GET_METADATA_EARLY_STOP=0
-               - Restart affected nodes.
-            3. Collect flamegraphs and open a P0 incident.
-
-      # ------------------------------------------------------------------
-      # 2. PipelineFailureSpike
-      #    Pipeline failure rate exceeds 5x the baseline sustained for
-      #    5 minutes.  Covers all failure reasons: bitrot_mismatch,
-      #    decode_error, downstream_closed, io, read_quorum, timeout, etc.
-      #    Action: Investigate pipeline health and roll back if needed.
-      # ------------------------------------------------------------------
-      - alert: PipelineFailureSpike
-        expr: |
-          sum(rate(rustfs_io_get_object_pipeline_failures_total[5m]))
-          >
-          5
-          *
-          sum(rate(rustfs_io_get_object_pipeline_failures_total[5m] offset 1d))
-        for: 5m
-        labels:
-          severity: critical
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "GET pipeline failure rate spike (>5x baseline for 5m)"
-          description: >-
-            The GET pipeline failure rate is {{ $value | printf "%.2f" }}/s,
-            more than 5x the baseline from 24 hours ago.
-            Failure reasons may include: bitrot_mismatch, decode_error,
-            downstream_closed, io, range_or_length_invalid, read_quorum,
-            short_read, timeout, unknown.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/pipeline-failure-spike"
-          action: >
-            1. Check Grafana "GET Data Integrity" dashboard for failure
-               breakdown by reason label.
-            2. If decode_error or bitrot_mismatch dominates, stop
-               optimization and investigate data integrity.
-            3. If io or timeout dominates, check disk and network health.
-            4. Roll back optimization if failures persist.
-
-      # ------------------------------------------------------------------
-      # 3. BitrotMismatchSpike
-      #    Bitrot verification mismatch rate exceeds 3x baseline for
-      #    5 minutes.  This is a data-integrity signal — shard checksums
-      #    do not match after read.
-      #
-      #    The "bitrot_mismatch" reason is recorded on the
-      #    rustfs_io_get_object_pipeline_failures_total counter when a
-      #    StorageError::FileCorrupt or DiskError::FileCorrupt /
-      #    DiskError::PartMissingOrCorrupt is classified during the GET
-      #    pipeline (see classify_storage_error / classify_disk_error in
-      #    crates/ecstore/src/diagnostics/get.rs).
-      #
-      #    Action: Stop optimization, investigate data integrity urgently.
-      # ------------------------------------------------------------------
-      - alert: BitrotMismatchSpike
-        expr: |
-          sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m]))
-          >
-          3
-          *
-          sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m] offset 1d))
-        for: 5m
-        labels:
-          severity: critical
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "Bitrot mismatch rate spike (>3x baseline for 5m)"
-          description: >-
-            The rate of pipeline failures classified as bitrot_mismatch is
-            {{ $value | printf "%.2f" }}/s, more than 3x the baseline from
-            24 hours ago.  This indicates shard checksum verification
-            failures (FileCorrupt / PartMissingOrCorrupt) which may point
-            to data corruption introduced by the GET optimization pipeline
-            (e.g., incorrect decode, buffer reuse bug).
-          runbook_url: "https://internal.wiki/runbooks/rustfs/bitrot-mismatch-spike"
-          action: >
-            1. Immediately disable codec streaming:
-               RUSTFS_GET_CODEC_STREAMING=0
-            2. Run "mc admin scan" on affected buckets to verify on-disk
-               integrity independent of the GET path.
-            3. Compare xl.meta checksums across erasure shards.
-            4. If corruption confirmed, initiate data recovery from parity.
-            5. Do NOT re-enable optimization until root cause is identified.
-
-  # ==========================================================================
-  # Warning alerts — investigation needed
-  # ==========================================================================
-  - name: rustfs-get-optimization-warning
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 4. EarlyStopInsufficientQuorum
-      #    The metadata early-stop path is hitting "insufficient_quorum"
-      #    at a rate above 0.1/s for 5 minutes.  This means too many
-      #    disks are failing to return valid metadata in time.
-      #    Action: Check disk health and metadata fanout latency.
-      # ------------------------------------------------------------------
-      - alert: EarlyStopInsufficientQuorum
-        expr: |
-          sum(rate(rustfs_io_get_object_metadata_early_stop_total{reason="insufficient_quorum"}[5m]))
-          > 0.1
-        for: 5m
-        labels:
-          severity: warning
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "Early-stop insufficient quorum rate elevated (>0.1/s for 5m)"
-          description: >-
-            The metadata early-stop path is returning "insufficient_quorum"
-            at {{ $value | printf "%.3f" }}/s.  This means the bounded
-            metadata fanout cannot gather enough valid responses before
-            the quorum deadline, suggesting disk or network issues.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/early-stop-quorum"
-          action: >
-            1. Check disk health: mc admin info --json | jq '.disks'
-            2. Review rustfs_io_get_object_metadata_response_total by
-               outcome (error, timeout, disk_not_found) in Grafana.
-            3. Check rustfs_io_disk_permit_wait_duration_seconds for
-               I/O scheduler saturation.
-            4. If disks are healthy, consider increasing the early-stop
-               timeout or temporarily disabling early-stop.
-
-      # ------------------------------------------------------------------
-      # 5. CodecStreamingFallbackSpike
-      #    The codec streaming fallback rate is >10x baseline for 10
-      #    minutes.  This means the optimized codec streaming path is
-      #    being bypassed much more often than expected.
-      #    Action: Check fallback reasons and object eligibility.
-      # ------------------------------------------------------------------
-      - alert: CodecStreamingFallbackSpike
-        expr: |
-          sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
-          >
-          10
-          *
-          sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m] offset 1d))
-        for: 10m
-        labels:
-          severity: warning
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "Codec streaming fallback rate spike (>10x baseline for 10m)"
-          description: >-
-            The codec streaming fallback rate is {{ $value | printf "%.2f" }}/s,
-            more than 10x the baseline from 24 hours ago.  Fallback reasons
-            are labeled by "reason" — check Grafana for breakdown.
-            Common reasons: object too small, multipart not supported,
-            unsupported erasure layout, feature flag disabled.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/codec-fallback-spike"
-          action: >
-            1. Query by reason label:
-               sum by (reason) (rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
-            2. If dominated by a single reason, investigate why that
-               condition became more frequent (e.g., workload change,
-               configuration drift).
-            3. Cross-reference with rustfs_io_get_object_reader_path_total
-               to verify the fallback path (legacy_duplex) is healthy.
-            4. If fallback is expected (e.g., workload shifted to small
-               objects), update the alert baseline.
-
-      # ------------------------------------------------------------------
-      # 6. IoQueueSaturation
-      #    I/O queue utilization exceeds 90% for 5 minutes.  High
-      #    utilization causes disk permit wait latency to increase and
-      #    can cascade into pipeline timeouts.
-      #    Action: Check disk load and consider reducing concurrency.
-      # ------------------------------------------------------------------
-      - alert: IoQueueSaturation
-        expr: |
-          rustfs_io_queue_utilization_percent > 90
-        for: 5m
-        labels:
-          severity: warning
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "I/O queue utilization >90% for 5m"
-          description: >-
-            The I/O queue utilization is {{ $value | printf "%.1f" }}%,
-            sustained above 90% for 5 minutes.  This indicates the disk
-            I/O scheduler is near saturation, which will increase
-            rustfs_io_disk_permit_wait_duration_seconds and may trigger
-            pipeline timeouts.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/io-queue-saturation"
-          action: >
-            1. Check disk I/O metrics (iostat, node_exporter) for
-               individual disk saturation.
-            2. Review rustfs_io_queue_permits_in_use vs
-               rustfs_io_queue_permits_available for permit exhaustion.
-            3. Check rustfs_io_starvation_events for priority starvation.
-            4. If GET optimization increased concurrency, consider:
-               - Reducing RUSTFS_GET_PIPELINE_PARALLELISM
-               - Lowering RUSTFS_IO_QUEUE_PERMITS
-            5. If caused by background operations (ILM, healing), throttle
-               those before adjusting GET concurrency.

.docker/observability/prometheus-rules/rustfs-kms-alerts.yml

@@ -1,251 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# =============================================================================
-# RustFS KMS backend — Prometheus alerting rules
-# =============================================================================
-#
-# Metric source: the KMS operation-policy choke point in
-# crates/kms/src/policy.rs, except KmsKeyRotationOverdue, which reads the
-# label-less key-lifecycle gauge published by the deletion worker's sweep
-# (crates/kms/src/deletion_worker.rs). All label values are bounded static
-# strings (operation, op_class, outcome, error_class, backend, scope); key
-# identifiers, key material, and tokens never appear in labels.
-#
-# Response procedures: docs/operations/kms-observability-runbook.md
-#
-# IMPORTANT — threshold status: every numeric threshold below is a
-# conservative default chosen without a production baseline. Calibrate against
-# a staging baseline before relying on these alerts for paging, and prefer
-# loosening over tightening until the baseline exists. Formal SLO targets are
-# deliberately not encoded here (see rustfs/backlog#1584).
-#
-# NOTE: prometheus.yml loads /etc/prometheus/rules/*.yml — keep the .yml
-# extension or the file is silently ignored by the docker-compose stack.
-#
-# Validate: promtool check rules rustfs-kms-alerts.yml
-# =============================================================================
-
-groups:
-  # ==========================================================================
-  # Critical alerts — immediate action required
-  # ==========================================================================
-  - name: rustfs-kms-critical
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 1. KmsBackendFatalErrors
-      #    Any attempt failure classified as fatal (non-retryable): auth
-      #    or permission errors, malformed requests, missing keys. The
-      #    policy never retries these, so even a low rate means real
-      #    operations are failing right now.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendFatalErrors
-        expr: |
-          sum by (operation) (rate(rustfs_kms_backend_attempt_failures_total{error_class="fatal"}[5m])) > 0
-        for: 5m
-        labels:
-          severity: critical
-          component: kms
-        annotations:
-          summary: "KMS backend fatal errors on operation {{ $labels.operation }}"
-          description: >-
-            Attempt failures classified as fatal are occurring at
-            {{ $value | printf "%.3f" }}/s on operation
-            {{ $labels.operation }}. Fatal failures are not retried:
-            each one is a KMS backend call that failed permanently
-            (authentication, permissions, malformed request, or a
-            missing key/version).
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendfatalerrors"
-
-      # ------------------------------------------------------------------
-      # 2. KmsBackendHighErrorRate
-      #    Sustained share of operations terminating without success
-      #    (fatal, budget/deadline exhaustion, admission backpressure,
-      #    or an open circuit). The cancelled outcome is excluded because
-      #    shutdowns legitimately produce it.
-      #    The traffic guard keeps a single failure on a near-idle
-      #    cluster from firing the alert.
-      #    Threshold: 5% for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendHighErrorRate
-        expr: |
-          (
-            sum(rate(rustfs_kms_backend_operations_total{outcome!~"success|cancelled"}[5m]))
-            /
-            clamp_min(sum(rate(rustfs_kms_backend_operations_total[5m])), 1e-9)
-          ) > 0.05
-          and
-          sum(rate(rustfs_kms_backend_operations_total[5m])) > 0.02
-        for: 10m
-        labels:
-          severity: critical
-          component: kms
-        annotations:
-          summary: "KMS backend non-success ratio above 5% for 10m"
-          description: >-
-            {{ $value | humanizePercentage }} of KMS backend operations
-            are terminating in fatal, budget_exhausted,
-            deadline_exceeded, backpressure_timeout,
-            backpressure_rejected, or circuit_open. Object encryption
-            and decryption paths depending on the KMS are degraded or
-            failing.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
-
-  # ==========================================================================
-  # Warning alerts — investigation needed
-  # ==========================================================================
-  - name: rustfs-kms-warning
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 3. KmsBackendP99LatencyHigh
-      #    p99 wall-clock duration of whole operations (attempts plus
-      #    backoff) is sustained above 2 seconds. Because the histogram
-      #    includes retries, a high p99 usually means the retry policy
-      #    is absorbing backend failures, not that every call is slow.
-      #    Threshold: 2s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendP99LatencyHigh
-        expr: |
-          histogram_quantile(0.99,
-            sum by (le) (rate(rustfs_kms_backend_operation_duration_seconds_bucket[5m]))
-          ) > 2
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend operation p99 latency above 2s for 10m"
-          description: >-
-            The 99th-percentile KMS backend operation duration is
-            {{ $value | humanizeDuration }}, including retries and
-            backoff. Encryption and decryption latency is leaking into
-            S3 request latency.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendp99latencyhigh"
-
-      # ------------------------------------------------------------------
-      # 4. KmsBackendAttemptFailureSpike
-      #    Aggregate attempt-failure rate (all error classes) sustained
-      #    above an absolute floor. An absolute threshold is used instead
-      #    of an offset-1d baseline ratio because fresh deployments have
-      #    no baseline and an empty offset vector would keep a ratio
-      #    alert from ever firing; switch to a baseline-relative form
-      #    (see rustfs-get-optimization-alerts.yaml for the pattern)
-      #    once a stable staging baseline exists.
-      #    Threshold: 0.5/s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendAttemptFailureSpike
-        expr: |
-          sum(rate(rustfs_kms_backend_attempt_failures_total[5m])) > 0.5
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend attempt failures above 0.5/s for 10m"
-          description: >-
-            KMS backend attempts are failing at
-            {{ $value | printf "%.2f" }}/s across all error classes.
-            The retry policy may still be masking these from callers —
-            check the error-class breakdown before it stops absorbing
-            them.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendattemptfailurespike"
-
-      # ------------------------------------------------------------------
-      # 5. KmsBackendRetryBudgetExhausted
-      #    Operations are running out of retry budget (budget_exhausted)
-      #    or operation deadline (deadline_exceeded). These surface to
-      #    callers as failed KMS operations even though every individual
-      #    failure was retryable — the backend is unhealthy for longer
-      #    than the policy can bridge.
-      #    Threshold: 0.05/s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendRetryBudgetExhausted
-        expr: |
-          sum by (outcome) (rate(rustfs_kms_backend_operations_total{outcome=~"budget_exhausted|deadline_exceeded"}[5m])) > 0.05
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend operations exhausting retry budget ({{ $labels.outcome }})"
-          description: >-
-            KMS backend operations are terminating as
-            {{ $labels.outcome }} at {{ $value | printf "%.3f" }}/s.
-            Retryable failures are outlasting the retry budget, so
-            callers are seeing hard failures.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
-
-      # ------------------------------------------------------------------
-      # 6. KmsBackendCircuitOpen
-      #    Direct circuit-state signal, independent of operation traffic.
-      #    A transient open can recover on its first half-open probe; alert
-      #    only when the circuit remains open or half-open for one minute.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendCircuitOpen
-        expr: |
-          rustfs_kms_backend_circuit_open > 0
-        for: 1m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend circuit open ({{ $labels.backend }}/{{ $labels.scope }})"
-          description: >-
-            The KMS backend circuit for {{ $labels.backend }} scope
-            {{ $labels.scope }} has remained open or half-open for one
-            minute. Operations in this scope can terminate as
-            circuit_open until the half-open probe succeeds or returns
-            a non-retryable failure.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"
-
-      # ------------------------------------------------------------------
-      # 7. KmsKeyRotationOverdue
-      #    The least recently rotated usable key has gone more than 400
-      #    days without a rotation (measured from creation for keys with
-      #    no recorded rotation). Direct gauge state published by the
-      #    deletion worker's sweep, so no traffic guard applies; the
-      #    one-hour hold only bridges scrape gaps. The worker runs only
-      #    on backends with the schedule_deletion capability, so on the
-      #    Static backend the series never exists and this alert cannot
-      #    fire — that backend cannot rotate either; see the rotation
-      #    driver matrix in docs/operations/kms-backend-security.md.
-      #    Threshold: 400 days — conservative default sitting above a
-      #    one-year rotation policy. Align it with the rotation period
-      #    your compliance policy requires, and with
-      #    RUSTFS_KMS_ROTATION_MAX_AGE_SECS so the per-key rotation_due
-      #    verdict and this aggregate alert agree.
-      # ------------------------------------------------------------------
-      - alert: KmsKeyRotationOverdue
-        expr: |
-          rustfs_kms_oldest_key_rotation_age_seconds > (400 * 86400)
-        for: 1h
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "Oldest KMS key unrotated for more than 400 days"
-          description: >-
-            The least recently rotated usable KMS key was last rotated
-            {{ $value | humanizeDuration }} ago (measured from creation
-            for keys with no recorded rotation). List keys through the
-            admin API and read rotation_due / rotation_due_reason for
-            the per-key verdict; an "unsupported" reason means the
-            backend cannot rotate at all.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmskeyrotationoverdue"

.docker/observability/prometheus.yml

@@ -1,84 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-global:
-  scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
-  evaluation_interval: 15s
-  external_labels:
-    cluster: 'rustfs-dev'  # Label to identify the cluster
-    replica: '1'  # Replica identifier
-
-rule_files:
-  - /etc/prometheus/rules/*.yml
-
-scrape_configs:
-  - job_name: 'otel-collector'
-    static_configs:
-      - targets: [ 'otel-collector:8888' ]  # Scrape metrics from Collector
-    scrape_interval: 10s
-
-  - job_name: 'rustfs-app-metrics'
-    static_configs:
-      - targets: [ 'otel-collector:8889' ]  # Application indicators
-    scrape_interval: 15s
-    metric_relabel_configs:
-      - source_labels: [ __name__ ]
-        regex: 'go_.*'
-        action: drop  # Drop Go runtime metrics if not needed
-
-  - job_name: 'tempo'
-    static_configs:
-      - targets: [ 'tempo:3200' ]  # Scrape metrics from Tempo
-
-  - job_name: 'jaeger'
-    static_configs:
-      - targets: [ 'jaeger:14269' ]  # Jaeger admin port (14269 is standard for admin/metrics)
-
-  - job_name: 'loki'
-    static_configs:
-      - targets: [ 'loki:3100' ]
-
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: [ 'localhost:9090' ]
-
-  - job_name: 'vulture'
-    static_configs:
-      - targets:
-          - 'vulture:8080'
-
-otlp:
-  promote_resource_attributes:
-    - service.instance.id
-    - service.name
-    - service.namespace
-    - cloud.availability_zone
-    - cloud.region
-    - container.name
-    - deployment.environment.name
-    - k8s.cluster.name
-    - k8s.container.name
-    - k8s.cronjob.name
-    - k8s.daemonset.name
-    - k8s.deployment.name
-    - k8s.job.name
-    - k8s.namespace.name
-    - k8s.pod.name
-    - k8s.replicaset.name
-    - k8s.statefulset.name
-  translation_strategy: NoUTF8EscapingWithSuffixes
-
-storage:
-  tsdb:
-    out_of_order_time_window: 30m

.docker/observability/redpanda-console.yaml

@@ -1,3 +0,0 @@
-kafka:
-  brokers:
-    - redpanda:9092

.docker/observability/tempo-ha.yaml

@@ -1,286 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# High Availability Tempo Configuration for docker-compose-example-for-rustfs.yml
-# Features:
-# - Distributed architecture with multiple components
-# - Kafka-based ingestion for fault tolerance
-# - Replication factor of 3 for data resilience
-# - Query frontend for load balancing
-# - Metrics generation from traces
-# - WAL for durability
-
-partition_ring_live_store: true
-stream_over_http_enabled: true
-
-server:
-  http_listen_port: 3200
-  http_server_read_timeout: 30s
-  http_server_write_timeout: 30s
-  grpc_server_max_recv_msg_size: 4194304  # 4MB
-  grpc_server_max_send_msg_size: 4194304
-  log_level: info
-  log_format: json
-
-# Memberlist configuration for distributed mode
-memberlist:
-  node_name: tempo
-  bind_port: 7946
-  join_members:
-    - tempo:7946
-  retransmit_factor: 4
-  node_timeout: 15s
-  retransmit_interval: 300ms
-  dead_node_reclaim_time: 30s
-
-# Distributor configuration - receives traces and routes to ingesters
-distributor:
-  ingester_write_path_enabled: true
-  kafka_write_path_enabled: true
-  rate_limit_bytes: 10MB
-  rate_limit_enabled: true
-  receivers:
-    otlp:
-      protocols:
-        grpc:
-          endpoint: "0.0.0.0:4317"
-          max_concurrent_streams: 0
-          max_receive_message_size: 4194304
-        http:
-          endpoint: "0.0.0.0:4318"
-          cors:
-            allowed_origins:
-              - "*"
-            max_age: 86400
-    jaeger:
-      protocols:
-        grpc:
-          endpoint: "0.0.0.0:14250"
-        thrift_http:
-          endpoint: "0.0.0.0:14268"
-    zipkin:
-      endpoint: "0.0.0.0:9411"
-  ring:
-    kvstore:
-      store: memberlist
-    heartbeat_timeout: 5s
-    replication_factor: 3
-    heartbeat_interval: 5s
-
-# Ingester configuration - stores traces and querying
-ingester:
-  lifecycler:
-    address: tempo
-    ring:
-      kvstore:
-        store: memberlist
-      replication_factor: 3
-      max_cache_freshness_per_sec: 10s
-      heartbeat_interval: 5s
-      heartbeat_timeout: 5s
-    num_tokens: 128
-    tokens_file_path: /var/tempo/tokens.json
-    claim_on_rollout: true
-  trace_idle_period: 20s
-  max_block_bytes: 10_000_000
-  max_block_duration: 10m
-  chunk_size_bytes: 1_000_000
-  chunk_encoding: snappy
-  wal:
-    checkpoint_duration: 5s
-    max_wal_blocks: 4
-  metrics:
-    enabled: true
-    level: block
-    target_info_duration: 15m
-
-# WAL configuration for data durability
-wal:
-  checkpoint_duration: 5s
-  flush_on_shutdown: true
-  path: /var/tempo/wal
-
-# Kafka ingestion configuration - for high throughput scenarios
-ingest:
-  enabled: true
-  kafka:
-    brokers: [ redpanda:9092 ]
-    topic: tempo-ingest
-    encoding: protobuf
-    consumer_group: tempo-ingest-consumer
-    session_timeout: 10s
-    rebalance_timeout: 1m
-    partition: auto
-    verbosity: 2
-
-# Query frontend configuration - distributed querying
-query_frontend:
-  compression: gzip
-  downstream_url: http://localhost:3200
-  log_queries_longer_than: 5s
-  cache_uncompressed_bytes: 100MB
-  max_outstanding_requests_per_tenant: 100
-  max_query_length: 48h
-  max_query_lookback: 30d
-  default_result_cache_ttl: 1m
-  result_cache:
-    cache:
-      enable_fifocache: true
-      default_validity: 1m
-  rf1_after: "1999-01-01T00:00:00Z"
-  mcp_server:
-    enabled: true
-
-# Querier configuration - queries traces
-querier:
-  frontend_worker:
-    frontend_address: localhost:3200
-    grpc_client_config:
-      max_recv_msg_size: 104857600
-  max_concurrent_queries: 20
-  max_metric_bytes_per_trace: 1MB
-
-# Query scheduler configuration - for distributed querying
-query_scheduler:
-  use_scheduler_ring: false
-
-# Metrics generator configuration - generates metrics from traces
-metrics_generator:
-  enabled: true
-  registry:
-    enabled: true
-    external_labels:
-      source: tempo
-      cluster: rustfs-docker-ha
-      environment: production
-  storage:
-    path: /var/tempo/generator/wal
-    remote_write:
-      - url: http://prometheus:9090/api/v1/write
-        send_exemplars: true
-        resource_to_telemetry_conversion:
-          enabled: true
-  processor:
-    batch:
-      timeout: 10s
-      send_batch_size: 1024
-    memory_limiter:
-      check_interval: 5s
-      limit_mib: 512
-      spike_limit_mib: 128
-  processors:
-    - span-metrics
-    - local-blocks
-    - service-graphs
-  generate_native_histograms: both
-
-# Backend worker configuration
-backend_worker:
-  backend_scheduler_addr: localhost:3200
-  compaction:
-    block_retention: 24h
-    compacted_block_retention: 1h
-  ring:
-    kvstore:
-      store: memberlist
-
-# Backend scheduler configuration
-backend_scheduler:
-  enabled: true
-  provider:
-    compaction:
-      compaction:
-        block_retention: 24h
-        compacted_block_retention: 1h
-        concurrency: 25
-        v2_out_path: /var/tempo/blocks/compaction
-
-# Storage configuration - local backend with proper retention
-storage:
-  trace:
-    backend: local
-    wal:
-      path: /var/tempo/wal
-      checkpoint_duration: 5s
-      flush_on_shutdown: true
-    local:
-      path: /var/tempo/blocks
-      bloom_filter_false_positive: 0.05
-      bloom_shift: 4
-      index:
-        downsample_bytes: 1000000
-        page_size_bytes: 0
-        cache_size_bytes: 0
-    pool:
-      max_workers: 400
-      queue_depth: 10000
-
-# Compactor configuration - manages block compaction
-compactor:
-  compaction:
-    block_retention: 168h  # 7 days
-    compacted_block_retention: 1h
-    concurrency: 25
-    v2_out_path: /var/tempo/blocks/compaction
-    shard_count: 32
-    max_block_bytes: 107374182400  # 100GB
-    max_compaction_objects: 6000000
-    max_time_per_tenant: 5m
-    block_size_bytes: 107374182400
-  ring:
-    kvstore:
-      store: memberlist
-    heartbeat_interval: 5s
-    heartbeat_timeout: 5s
-
-# Limits configuration - rate limiting and quotas
-limits:
-  max_traces_per_user: 10000
-  max_bytes_per_trace: 10485760  # 10MB
-  max_search_bytes_per_trace: 0
-  forgiving_oversize_traces: true
-  rate_limit_bytes: 10MB
-  rate_limit_enabled: true
-  ingestion_burst_size_bytes: 20MB
-  ingestion_rate_limit_bytes: 10MB
-  max_bytes_per_second: 10485760
-  metrics_generator_max_active_series: 10000
-  metrics_generator_max_churned_series: 10000
-  metrics_generator_forta_out_of_order_ttl: 5m
-
-# Override configuration
-overrides:
-  defaults:
-    metrics_generator:
-      processors:
-        - span-metrics
-        - local-blocks
-        - service-graphs
-      generate_native_histograms: both
-      max_active_series: 10000
-      max_churned_series: 10000
-
-# Usage reporting configuration
-usage_report:
-  reporting_enabled: false
-
-# Tracing configuration for debugging
-tracing:
-  enabled: true
-  jaeger:
-    sampler:
-      name: probabilistic
-      param: 0.1
-    reporter_log_spans: false
-

.docker/observability/tempo.yaml

@@ -1,72 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-stream_over_http_enabled: true
-
-server:
-  http_listen_port: 3200
-  log_level: info
-
-memberlist:
-  node_name: tempo
-  bind_port: 7946
-  join_members:
-    - tempo:7946
-
-distributor:
-  receivers:
-    otlp:
-      protocols:
-        grpc:
-          endpoint: "0.0.0.0:4317"
-        http:
-          endpoint: "0.0.0.0:4318"
-
-ingester:
-  max_block_duration: 5m
-
-metrics_generator:
-  registry:
-    external_labels:
-      source: tempo
-      cluster: docker-compose
-  traces_storage:
-    path: /var/tempo/generator/traces
-  storage:
-    path: /var/tempo/generator/wal
-    remote_write:
-      - url: http://prometheus:9090/api/v1/write
-        send_exemplars: true
-
-query_frontend:
-  rf1_after: "1999-01-01T00:00:00Z"
-  mcp_server:
-    enabled: true
-
-storage:
-  trace:
-    backend: local
-    wal:
-      path: /var/tempo/wal # where to store the wal locally
-    local:
-      path: /var/tempo/blocks # where to store the traces locally
-
-overrides:
-  defaults:
-    metrics_generator:
-      processors: [ "span-metrics", "service-graphs", "local-blocks" ]
-      generate_native_histograms: both
-
-usage_report:
-  reporting_enabled: false

.docker/openobserve-otel/README.md

@@ -1,61 +0,0 @@
-# OpenObserve + OpenTelemetry Collector
-
-[![OpenObserve](https://img.shields.io/badge/OpenObserve-OpenSource-blue.svg)](https://openobserve.org)
-[![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-Collector-green.svg)](https://opentelemetry.io/)
-
-English | [中文](README_ZH.md)
-
-This directory contains the configuration for an **alternative** observability stack using OpenObserve.
-
-## ⚠️ Note
-
-For the **recommended** observability stack (Prometheus, Grafana, Tempo, Loki), please see `../observability/`.
-
-## 🌟 Overview
-
-OpenObserve is a lightweight, all-in-one observability platform that handles logs, metrics, and traces in a single binary. This setup is ideal for:
-- Resource-constrained environments.
-- Quick setup and testing.
-- Users who prefer a unified UI.
-
-## 🚀 Quick Start
-
-### 1. Start Services
-
-```bash
-cd .docker/openobserve-otel
-docker compose up -d
-```
-
-### 2. Access Dashboard
-
-- **URL**: [http://localhost:5080](http://localhost:5080)
-- **Username**: `root@rustfs.com`
-- **Password**: `rustfs123`
-
-## 🛠️ Configuration
-
-### OpenObserve
-
-- **Persistence**: Data is persisted to a Docker volume.
-- **Ports**:
-    - `5080`: HTTP API and UI
-    - `5081`: OTLP gRPC
-
-### OpenTelemetry Collector
-
-- **Receivers**: OTLP (gRPC `4317`, HTTP `4318`)
-- **Exporters**: Sends data to OpenObserve.
-
-## 🔗 Integration
-
-Configure your application to send OTLP data to the collector:
-
-- **Endpoint**: `http://localhost:4318` (HTTP) or `localhost:4317` (gRPC)
-
-Example for RustFS:
-
-```bash
-export RUSTFS_OBS_ENDPOINT=http://localhost:4318
-export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
-```

.docker/openobserve-otel/README_ZH.md

@@ -1,61 +0,0 @@
-# OpenObserve + OpenTelemetry Collector
-
-[![OpenObserve](https://img.shields.io/badge/OpenObserve-OpenSource-blue.svg)](https://openobserve.org)
-[![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-Collector-green.svg)](https://opentelemetry.io/)
-
-[English](README.md) | 中文
-
-本目录包含使用 OpenObserve 的**替代**可观测性技术栈配置。
-
-## ⚠️ 注意
-
-对于**推荐**的可观测性技术栈（Prometheus, Grafana, Tempo, Loki），请参阅 `../observability/`。
-
-## 🌟 概览
-
-OpenObserve 是一个轻量级、一体化的可观测性平台，在一个二进制文件中处理日志、指标和追踪。此设置非常适合：
-- 资源受限的环境。
-- 快速设置和测试。
-- 喜欢统一 UI 的用户。
-
-## 🚀 快速开始
-
-### 1. 启动服务
-
-```bash
-cd .docker/openobserve-otel
-docker compose up -d
-```
-
-### 2. 访问仪表盘
-
-- **URL**: [http://localhost:5080](http://localhost:5080)
-- **用户名**: `root@rustfs.com`
-- **密码**: `rustfs123`
-
-## 🛠️ 配置
-
-### OpenObserve
-
-- **持久化**: 数据持久化到 Docker 卷。
-- **端口**:
-    - `5080`: HTTP API 和 UI
-    - `5081`: OTLP gRPC
-
-### OpenTelemetry Collector
-
-- **接收器**: OTLP (gRPC `4317`, HTTP `4318`)
-- **导出器**: 将数据发送到 OpenObserve。
-
-## 🔗 集成
-
-配置您的应用程序将 OTLP 数据发送到收集器：
-
-- **端点**: `http://localhost:4318` (HTTP) 或 `localhost:4317` (gRPC)
-
-RustFS 示例：
-
-```bash
-export RUSTFS_OBS_ENDPOINT=http://localhost:4318
-export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
-```

.docker/openobserve-otel/docker-compose.yml

@@ -1,87 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  openobserve:
-    image: public.ecr.aws/zinclabs/openobserve:latest
-    restart: unless-stopped
-    environment:
-      ZO_ROOT_USER_EMAIL: "root@rustfs.com"
-      ZO_ROOT_USER_PASSWORD: "rustfs123"
-      ZO_TRACING_HEADER_KEY: "Authorization"
-      ZO_TRACING_HEADER_VALUE: "Basic cm9vdEBydXN0ZnMuY29tOmQ4SXlCSEJTUkk3RGVlcEQ="
-      ZO_DATA_DIR: "/data"
-      ZO_MEMORY_CACHE_ENABLED: "true"
-      ZO_MEMORY_CACHE_MAX_SIZE: "256"
-      RUST_LOG: "info"
-      TZ: Asia/Shanghai
-    ports:
-      - "5080:5080"
-      - "5081:5081"
-    volumes:
-      - ./data:/data
-    healthcheck:
-      test: [ "CMD", "curl", "-f", "http://localhost:5080/health" ]
-      start_period: 60s
-      interval: 10s
-      timeout: 5s
-      retries: 6
-    networks:
-      - otel-network
-    deploy:
-      resources:
-        limits:
-          memory: 1024M
-        reservations:
-          memory: 512M
-
-  otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
-    restart: unless-stopped
-    environment:
-      - TZ=Asia/Shanghai
-    volumes:
-      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
-    ports:
-      - "4317:4317" # OTLP gRPC
-      - "4318:4318" # OTLP HTTP
-      - "13133:13133" # Health check
-      - "1777:1777" # pprof
-      - "55679:55679" # zpages
-      - "1888:1888" # Metrics
-      - "8888:8888" # Prometheus metrics
-      - "8889:8889" # Additional metrics endpoint
-    depends_on:
-      - openobserve
-    networks:
-      - otel-network
-    deploy:
-      resources:
-        limits:
-          memory: 10240M
-        reservations:
-          memory: 512M
-
-networks:
-  otel-network:
-    driver: bridge
-    name: otel-network
-    ipam:
-      config:
-        - subnet: 172.28.0.0/16
-          gateway: 172.28.0.1
-    labels:
-      com.example.description: "Network for OpenObserve and OpenTelemetry Collector"
-volumes:
-  data:

.docker/openobserve-otel/otel-collector-config.yaml

@@ -1,92 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-receivers:
-  otlp:
-    protocols:
-      grpc:
-        endpoint: 0.0.0.0:4317
-      http:
-        endpoint: 0.0.0.0:4318
-  filelog:
-    include: [ "/var/log/app/*.log" ]
-    start_at: end
-
-processors:
-  batch:
-    timeout: 1s
-    send_batch_size: 1024
-  memory_limiter:
-    check_interval: 1s
-    limit_mib: 400
-    spike_limit_mib: 100
-
-exporters:
-  otlphttp/openobserve:
-    endpoint: http://openobserve:5080/api/default # http://127.0.0.1:5080/api/default
-    headers:
-      Authorization: "Basic cm9vdEBydXN0ZnMuY29tOmQ4SXlCSEJTUkk3RGVlcEQ="
-      stream-name: default
-      organization: default
-    compression: gzip
-    retry_on_failure:
-      enabled: true
-      initial_interval: 5s
-      max_interval: 30s
-      max_elapsed_time: 300s
-    timeout: 10s
-  otlp/openobserve:
-    endpoint: openobserve:5081 # http://127.0.0.1:5080/api/default
-    headers:
-      Authorization: "Basic cm9vdEBydXN0ZnMuY29tOmQ4SXlCSEJTUkk3RGVlcEQ="
-      stream-name: default
-      organization: default
-    compression: gzip
-    retry_on_failure:
-      enabled: true
-      initial_interval: 5s
-      max_interval: 30s
-      max_elapsed_time: 300s
-    timeout: 10s
-    tls:
-      insecure: true
-
-extensions:
-  health_check:
-    endpoint: 0.0.0.0:13133
-  pprof:
-    endpoint: 0.0.0.0:1777
-  zpages:
-    endpoint: 0.0.0.0:55679
-
-service:
-  extensions: [ health_check, pprof, zpages ]
-  pipelines:
-    traces:
-      receivers: [ otlp ]
-      processors: [ memory_limiter, batch ]
-      exporters: [ otlp/openobserve ]
-    metrics:
-      receivers: [ otlp ]
-      processors: [ memory_limiter, batch ]
-      exporters: [ otlp/openobserve ]
-    logs:
-      receivers: [ otlp, filelog ]
-      processors: [ memory_limiter, batch ]
-      exporters: [ otlp/openobserve ]
-  telemetry:
-    logs:
-      level: "info"  # Collector 日志级别
-    metrics:
-      address: "0.0.0.0:8888"  # Collector 自身指标暴露

.docker/test/compat/README.md

@@ -1,91 +0,0 @@
-# Rio compatibility compose files
-
-These compose files prepare 4-node, 4-disk clusters for rio/rio-v2 storage format compatibility checks. All disks are bind-mounted under `.docker/compat/data` so the on-disk files remain available on the host.
-
-## Clusters
-
-```bash
-docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d --build
-docker compose -f .docker/compat/docker-compose.minio.yml up -d
-docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
-```
-
-Default API endpoints:
-
-- RustFS `1.0.0-beta.5`: `http://127.0.0.1:9100`
-- MinIO: `http://127.0.0.1:9200`
-- current main with `rio-v2`: `http://127.0.0.1:9300`
-
-## Reading old datasets with rio-v2
-
-Stop the writer cluster before mounting its disks into the rio-v2 cluster.
-
-```bash
-docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml down
-RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
-  docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
-
-docker compose -f .docker/compat/docker-compose.minio.yml down
-RUSTFS_RIO_V2_DATASET=./data/minio \
-  docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
-```
-
-## 200G object mix
-
-Use the same bucket/object matrix against the beta5 and MinIO endpoints, then read it back through the rio-v2 endpoint. A practical 200G mix is:
-
-- 1 KiB x 1024
-- 1 MiB x 1024
-- 64 MiB x 512
-- 1 GiB x 64
-- 8 GiB x 12
-- 6 GiB x 1
-
-Compression is enabled by default for RustFS and MinIO. Server-side KMS/SSE settings are intentionally left to environment variables or mounted key directories so real key material is not committed. For SSE-C cases, run the clusters with TLS because MinIO requires HTTPS for SSE-C.
-
-## High-concurrency write/read stress
-
-Use `run_rw_compat_stress.sh` to generate a manifest on an old endpoint, then verify the same objects through the rio-v2 endpoint after mounting the old disks.
-
-```bash
-.docker/compat/run_rw_compat_stress.sh \
-  --mode write \
-  --endpoint http://127.0.0.1:9100 \
-  --access-key rustfsadmin \
-  --secret-key rustfsadmin \
-  --bucket compat-beta5 \
-  --concurrency 96
-
-RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
-  docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
-
-.docker/compat/run_rw_compat_stress.sh \
-  --mode verify \
-  --endpoint http://127.0.0.1:9300 \
-  --access-key rustfsadmin \
-  --secret-key rustfsadmin \
-  --bucket compat-beta5 \
-  --concurrency 96 \
-  --manifest target/compat/rw-stress-YYYYmmdd-HHMMSS/manifest.csv
-```
-
-For encrypted datasets, add `--encryption sse-s3`, `--encryption sse-kms --sse-kms-key-id <key-id>`, or `--encryption sse-c --sse-c-key-file <raw-32-byte-key-file>` to both the write and verify commands.
-
-## 5 GiB encrypted compatibility run
-
-The `5g` profile covers 1 KiB, 1 MiB, 16 MiB, 64 MiB, and 1 GiB objects and totals exactly 5 GiB. Generate `compat-key.key` under `.docker/compat/kms/rustfs-compat`, enable local KMS on both RustFS clusters, then use the same encryption arguments while writing with beta5 and verifying with rio-v2. Set non-default local test credentials first because distributed listeners reject the built-in default credentials.
-
-```bash
-export COMPAT_ACCESS_KEY='<non-default-access-key>'
-export COMPAT_SECRET_KEY='<non-default-secret-key>'
-
-RUSTFS_ACCESS_KEY="$COMPAT_ACCESS_KEY" RUSTFS_SECRET_KEY="$COMPAT_SECRET_KEY" \
-  RUSTFS_KMS_ENABLE=true RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true \
-  docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d
-
-.docker/compat/run_rw_compat_stress.sh \
-  --mode write --endpoint http://127.0.0.1:9100 \
-  --access-key "$COMPAT_ACCESS_KEY" --secret-key "$COMPAT_SECRET_KEY" \
-  --bucket compat-beta5-sse-s3 --profile 5g --concurrency 16 \
-  --encryption sse-s3
-```

.docker/test/compat/docker-compose.minio.yml

@@ -1,88 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0.
-
-x-minio-env: &minio-env
-  MINIO_ROOT_USER: "${MINIO_ROOT_USER:-minioadmin}"
-  MINIO_ROOT_PASSWORD: "${MINIO_ROOT_PASSWORD:-minioadmin}"
-  MINIO_COMPRESSION_ENABLE: "${MINIO_COMPRESSION_ENABLE:-on}"
-  MINIO_COMPRESSION_ALLOW_ENCRYPTION: "${MINIO_COMPRESSION_ALLOW_ENCRYPTION:-on}"
-  MINIO_COMPRESSION_EXTENSIONS: "${MINIO_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
-  MINIO_COMPRESSION_MIME_TYPES: "${MINIO_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
-
-x-minio-node: &minio-node
-  image: "${MINIO_IMAGE:-quay.io/minio/minio:latest}"
-  command: server --console-address ":9001" "http://minio{1...4}:9000/data/disk{1...4}"
-  environment: *minio-env
-  networks:
-    - minio-compat-net
-  restart: unless-stopped
-
-services:
-  minio-permission-helper:
-    image: alpine:3.23
-    command: sh -c "mkdir -p /compat-data && chown -R 1000:1000 /compat-data"
-    volumes:
-      - ./data/minio:/compat-data
-    restart: "no"
-
-  minio1:
-    <<: *minio-node
-    hostname: minio1
-    depends_on:
-      minio-permission-helper:
-        condition: service_completed_successfully
-    volumes:
-      - ./data/minio/node1/disk1:/data/disk1
-      - ./data/minio/node1/disk2:/data/disk2
-      - ./data/minio/node1/disk3:/data/disk3
-      - ./data/minio/node1/disk4:/data/disk4
-    ports:
-      - "${MINIO_API_PORT:-9200}:9000"
-      - "${MINIO_CONSOLE_PORT:-9201}:9001"
-
-  minio2:
-    <<: *minio-node
-    hostname: minio2
-    depends_on:
-      minio-permission-helper:
-        condition: service_completed_successfully
-    volumes:
-      - ./data/minio/node2/disk1:/data/disk1
-      - ./data/minio/node2/disk2:/data/disk2
-      - ./data/minio/node2/disk3:/data/disk3
-      - ./data/minio/node2/disk4:/data/disk4
-    ports:
-      - "${MINIO_NODE2_PORT:-9202}:9000"
-
-  minio3:
-    <<: *minio-node
-    hostname: minio3
-    depends_on:
-      minio-permission-helper:
-        condition: service_completed_successfully
-    volumes:
-      - ./data/minio/node3/disk1:/data/disk1
-      - ./data/minio/node3/disk2:/data/disk2
-      - ./data/minio/node3/disk3:/data/disk3
-      - ./data/minio/node3/disk4:/data/disk4
-    ports:
-      - "${MINIO_NODE3_PORT:-9203}:9000"
-
-  minio4:
-    <<: *minio-node
-    hostname: minio4
-    depends_on:
-      minio-permission-helper:
-        condition: service_completed_successfully
-    volumes:
-      - ./data/minio/node4/disk1:/data/disk1
-      - ./data/minio/node4/disk2:/data/disk2
-      - ./data/minio/node4/disk3:/data/disk3
-      - ./data/minio/node4/disk4:/data/disk4
-    ports:
-      - "${MINIO_NODE4_PORT:-9204}:9000"
-
-networks:
-  minio-compat-net:
-    driver: bridge

.docker/test/compat/docker-compose.rustfs-beta5.yml

@@ -1,104 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0.
-
-x-rustfs-env: &rustfs-env
-  RUSTFS_VOLUMES: "http://rustfs-beta5-node{1...4}:9000/data/disk{1...4}"
-  RUSTFS_ADDRESS: ":9000"
-  RUSTFS_CONSOLE_ADDRESS: ":9001"
-  RUSTFS_CONSOLE_ENABLE: "true"
-  RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
-  RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
-  RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
-  RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
-  RUSTFS_OBS_LOG_DIRECTORY: "/logs"
-  RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
-  RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
-  RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
-  RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
-  RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
-  RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
-  RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
-  RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
-  RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
-
-x-rustfs-node: &rustfs-node
-  image: "${RUSTFS_BETA5_IMAGE:-rustfs/rustfs:1.0.0-beta.5}"
-  build:
-    context: ../..
-    dockerfile: Dockerfile
-    args:
-      RELEASE: "1.0.0-beta.5"
-  environment: *rustfs-env
-  depends_on:
-    rustfs-beta5-permission-helper:
-      condition: service_completed_successfully
-  networks:
-    - rustfs-beta5-net
-  restart: unless-stopped
-
-services:
-  rustfs-beta5-permission-helper:
-    image: alpine:3.23
-    command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
-    volumes:
-      - ./data/rustfs-beta5:/compat-data
-      - ./kms/rustfs-compat:/kms
-    restart: "no"
-
-  rustfs-beta5-node1:
-    <<: *rustfs-node
-    hostname: rustfs-beta5-node1
-    volumes:
-      - ./data/rustfs-beta5/node1/disk1:/data/disk1
-      - ./data/rustfs-beta5/node1/disk2:/data/disk2
-      - ./data/rustfs-beta5/node1/disk3:/data/disk3
-      - ./data/rustfs-beta5/node1/disk4:/data/disk4
-      - ./data/rustfs-beta5/logs/node1:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_BETA5_API_PORT:-9100}:9000"
-      - "${RUSTFS_BETA5_CONSOLE_PORT:-9101}:9001"
-
-  rustfs-beta5-node2:
-    <<: *rustfs-node
-    hostname: rustfs-beta5-node2
-    volumes:
-      - ./data/rustfs-beta5/node2/disk1:/data/disk1
-      - ./data/rustfs-beta5/node2/disk2:/data/disk2
-      - ./data/rustfs-beta5/node2/disk3:/data/disk3
-      - ./data/rustfs-beta5/node2/disk4:/data/disk4
-      - ./data/rustfs-beta5/logs/node2:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_BETA5_NODE2_PORT:-9102}:9000"
-
-  rustfs-beta5-node3:
-    <<: *rustfs-node
-    hostname: rustfs-beta5-node3
-    volumes:
-      - ./data/rustfs-beta5/node3/disk1:/data/disk1
-      - ./data/rustfs-beta5/node3/disk2:/data/disk2
-      - ./data/rustfs-beta5/node3/disk3:/data/disk3
-      - ./data/rustfs-beta5/node3/disk4:/data/disk4
-      - ./data/rustfs-beta5/logs/node3:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_BETA5_NODE3_PORT:-9103}:9000"
-
-  rustfs-beta5-node4:
-    <<: *rustfs-node
-    hostname: rustfs-beta5-node4
-    volumes:
-      - ./data/rustfs-beta5/node4/disk1:/data/disk1
-      - ./data/rustfs-beta5/node4/disk2:/data/disk2
-      - ./data/rustfs-beta5/node4/disk3:/data/disk3
-      - ./data/rustfs-beta5/node4/disk4:/data/disk4
-      - ./data/rustfs-beta5/logs/node4:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_BETA5_NODE4_PORT:-9104}:9000"
-
-networks:
-  rustfs-beta5-net:
-    driver: bridge