mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-22 12:26:37 +00:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 copilot-swe-agent[bot]
|
fe637d2087 |
@@ -1,45 +0,0 @@
|
|||||||
---
|
|
||||||
name: adversarial-validation
|
|
||||||
description: Review a final RustFS diff adversarially when the user requests adversarial review, the root AGENTS.md classifies the change as high risk, or a substantial PR is being reviewed. Do not use for ordinary questions, diagnosis, planning, status, documentation-only work, or routine low-risk implementation.
|
|
||||||
---
|
|
||||||
|
|
||||||
# RustFS Adversarial Validation
|
|
||||||
|
|
||||||
Use the risk tier and review shape defined in the root `AGENTS.md`. This skill
|
|
||||||
routes a review to RustFS-specific probes without loading unrelated domains.
|
|
||||||
|
|
||||||
## Select Lenses
|
|
||||||
|
|
||||||
Read only the references required by the diff:
|
|
||||||
|
|
||||||
| Lens | When to read |
|
|
||||||
|---|---|
|
|
||||||
| [Correctness](references/correctness.md) | Every non-exempt adversarial review |
|
|
||||||
| [Simplicity](references/simplicity.md) | Mechanical/standard changes and production growth |
|
|
||||||
| [Test coverage](references/test-coverage.md) | Behavior or test changes |
|
|
||||||
| [Security](references/security.md) | Authn/authz, IAM, RPC trust, paths, secrets, parsing, browser, encryption |
|
|
||||||
| [Concurrency/durability](references/concurrency-durability.md) | Async shared state, locks, storage commit, cancellation, persisted queues |
|
|
||||||
| [Compatibility](references/compatibility.md) | S3 surface, MinIO interop, metadata, wire/disk formats, mixed versions |
|
|
||||||
| [Performance](references/performance.md) | Request/object hot paths, allocation, blocking work, fsync, fan-out |
|
|
||||||
|
|
||||||
Do not read all references as a precaution. A path name alone is insufficient;
|
|
||||||
the changed behavior must touch the lens's domain.
|
|
||||||
|
|
||||||
For a dedicated security audit or advisory analysis, use
|
|
||||||
`security-advisory-lessons` instead of loading it automatically during every
|
|
||||||
adversarial review.
|
|
||||||
|
|
||||||
## Review Protocol
|
|
||||||
|
|
||||||
1. Freeze the exact final diff/head and list the selected lenses.
|
|
||||||
2. Run the review shape required by root `AGENTS.md`.
|
|
||||||
3. For each selected lens, either report a concrete finding or a null verdict
|
|
||||||
naming the attacks performed.
|
|
||||||
4. A finding needs `file:line`, a triggering input/state/interleaving, the wrong
|
|
||||||
outcome, and a focused fix or missing regression check.
|
|
||||||
5. Fix or rebut every finding with code-path, test, or invariant evidence.
|
|
||||||
6. After a non-trivial edit, rerun only lenses affected by that edit against the
|
|
||||||
new exact diff.
|
|
||||||
|
|
||||||
Do not turn a null verdict into a long checklist. Record concise evidence that
|
|
||||||
the relevant failure classes were attacked.
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
# Compatibility Lens
|
|
||||||
|
|
||||||
- Internal metadata uses `metadata_compat` helpers for dual RustFS/MinIO keys,
|
|
||||||
including mixed casing and removal of both twins.
|
|
||||||
- Binary UUID metadata treats absent, empty, and nil as no value. Unversioned
|
|
||||||
remote tiers receive no `versionId`; versioned purge requests retain the real
|
|
||||||
version ID.
|
|
||||||
- `xl.meta` changes preserve supported header/meta versions, recompute
|
|
||||||
signatures, decode legacy fixtures, and remain readable by old RustFS/MinIO.
|
|
||||||
- Foreign/corrupt metadata validates parallel array lengths and missing fields;
|
|
||||||
it returns a decode error rather than indexing, panicking, or fabricating data.
|
|
||||||
- Do not “correct” byte-for-byte MinIO ports without legacy fixture evidence.
|
|
||||||
Bitrot framing, shard math, distribution, and inline prefixes are contracts.
|
|
||||||
- Client-visible metadata/events strip both internal prefixes
|
|
||||||
case-insensitively.
|
|
||||||
- Proto fields are appended, never reused/renumbered; FlatBuffers tables extend
|
|
||||||
compatibly and absent new fields fail closed where authorization/quorum is
|
|
||||||
involved.
|
|
||||||
- Replay real client request shapes and exact pagination boundaries for S3
|
|
||||||
handler changes.
|
|
||||||
- Bucket metadata/IAM/config parsing remains compatible with pinned real MinIO
|
|
||||||
fixtures and encrypted migration data.
|
|
||||||
- Compatibility shims use `RUSTFS_COMPAT_TODO(<task-id>)`, have a removal
|
|
||||||
condition, and default toward reading old data safely.
|
|
||||||
@@ -1,23 +0,0 @@
|
|||||||
# Concurrency and Durability Lens
|
|
||||||
|
|
||||||
- For every changed lock, enumerate overlapping lock sets and construct the
|
|
||||||
ABBA interleaving. Multiple-lock order must be documented and consistent.
|
|
||||||
- Mark guard lifetimes and every `.await`, disk, and RPC call inside them.
|
|
||||||
Estimate contention and timeout behavior under concurrent requests.
|
|
||||||
- Object commits remain fenced if the distributed lock is lost after shard
|
|
||||||
writes and before metadata rename.
|
|
||||||
- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent
|
|
||||||
-> sync required ancestors`; simulate a crash after each step and honor the
|
|
||||||
configured durability gate.
|
|
||||||
- Multi-disk fan-out counts every result. Quorum-minus-one cannot become success;
|
|
||||||
heal remains best-effort per target where that is the established contract.
|
|
||||||
- At every new cancellable await between mutation and cleanup/commit, drop the
|
|
||||||
future and inspect leftover files, counters, permits, and replay state.
|
|
||||||
- Multipart operations on the same upload ID are serialized where required;
|
|
||||||
abort/complete/list races cannot delete parts before durable commit.
|
|
||||||
- Post-commit cleanup is best-effort, retry-safe, and cannot fail an already
|
|
||||||
committed write or delete the last surviving copy.
|
|
||||||
- Persisted read-modify-write uses serialization/CAS. Queue replay is crash-safe
|
|
||||||
and duplicate delivery has an idempotency contract.
|
|
||||||
- Streaming reconstruction failures after partial output surface as errors, not
|
|
||||||
successful EOF.
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# Correctness Lens
|
|
||||||
|
|
||||||
Attack the changed behavior, not every subsystem in the repository.
|
|
||||||
|
|
||||||
- Trace new error paths to the caller. Inject the ignored/wildcard variants and
|
|
||||||
verify they cannot become success, not-found, or a plausible default.
|
|
||||||
- Exercise zero/empty/missing, maximum, and exact-boundary inputs for every
|
|
||||||
changed count, size, index, page limit, or optional value.
|
|
||||||
- For aggregation/quorum changes, test exactly quorum and quorum-minus-one with
|
|
||||||
mixed disk errors and nil/placeholder entries.
|
|
||||||
- For listing/pagination, test `n == max`, `n == max + 1`, delimiter folding,
|
|
||||||
continuation markers, and object/prefix name collisions.
|
|
||||||
- For EC/read/streaming changes, inject failure after partial output and verify
|
|
||||||
the client receives an error rather than a clean truncated body. Assert exact
|
|
||||||
bytes and length.
|
|
||||||
- For multipart/object commits, fail before/after rename and cleanup; committed
|
|
||||||
data must remain readable and pre-commit cleanup must not destroy parts.
|
|
||||||
- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
|
|
||||||
versions, and deterministic tie-breaking.
|
|
||||||
- For directory-object behavior, trace `__XLDIR__` at the store layer; branches
|
|
||||||
below the layer that sees trailing slashes are dead.
|
|
||||||
- For binary UUID metadata, absent, empty, and nil all mean no value. Never send
|
|
||||||
nil/empty `versionId` to an unversioned tier.
|
|
||||||
- For agent rules/skill routers, test a trigger matrix covering ordinary
|
|
||||||
inquiry, low-risk implementation, explicit review, high-risk code, PR
|
|
||||||
creation, release, and post-PR monitoring. Each case must select only the
|
|
||||||
intended workflow and retain required safety/authorization boundaries.
|
|
||||||
|
|
||||||
Null verdicts name only the probes relevant to the diff.
|
|
||||||
@@ -1,20 +0,0 @@
|
|||||||
# Performance Lens
|
|
||||||
|
|
||||||
- For added clones/allocations on request/object/block paths, quantify copied
|
|
||||||
data and frequency. Recommend borrowing, move, `Bytes`/`Arc`, `Cow`, or
|
|
||||||
capacity reservation only for a concrete repeated cost.
|
|
||||||
- Route every new sync/flush through the durability-mode and bucket override
|
|
||||||
gates; mode `none` must not pay the new fsync.
|
|
||||||
- Keep blocking filesystem/CPU work off async runtime threads, but do not split
|
|
||||||
one small operation into many `spawn_blocking` round trips.
|
|
||||||
- Measure lock hold time across I/O and compare acquisition order for ABBA.
|
|
||||||
- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
|
|
||||||
section when they need not be there.
|
|
||||||
- Detect per-item serial I/O/RPC in batch APIs and accidental quadratic scans;
|
|
||||||
use a gate or bounded concurrency when the concrete fan-out warrants it.
|
|
||||||
- Count buffer growth and byte copies in EC/bitrot paths; preserve pool gauge
|
|
||||||
balance and avoid repeated metadata decode/fetch per object.
|
|
||||||
- Repetitive success logs stay at `trace`; metrics/instrumentation on hot paths
|
|
||||||
require an existing gate.
|
|
||||||
- Claims of no impact on PUT/GET/commit/erasure paths need relevant benchmark or
|
|
||||||
A/B evidence, especially for 4 KiB objects.
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
# Security Lens
|
|
||||||
|
|
||||||
Use `security-advisory-lessons` only for a dedicated advisory/security audit.
|
|
||||||
For an ordinary matched diff, attack these boundaries:
|
|
||||||
|
|
||||||
- Admin routes: route registration, whitelist, handler authn, and the exact
|
|
||||||
`AdminAction` must agree. Read-only diagnostics still require admin authz.
|
|
||||||
- IAM/service accounts: treat parent, claims, keys, groups, status, and policy
|
|
||||||
names as attacker-controlled; prove ownership/root authority before writes.
|
|
||||||
- Protocol frontends: every changed/sibling command authorizes the matching S3
|
|
||||||
action before reaching storage.
|
|
||||||
- Secrets/signatures: use constant-time comparison, normalize public failures,
|
|
||||||
keep RPC/root/STS keys independent, and fail closed when secrets are absent.
|
|
||||||
- RPC: bind signatures to the exact method/path and timestamp; reject replay,
|
|
||||||
stale, malformed, truncated, and invalid-enum payloads without panic.
|
|
||||||
- Paths/object/archive entries: reject traversal, absolute/platform escapes,
|
|
||||||
and normalization differences between authz and storage.
|
|
||||||
- Copy/multipart/presigned POST: enforce source, destination, version-aware
|
|
||||||
actions, copy-source conditions, and every signed policy condition.
|
|
||||||
- Logging/errors: never expose credentials, tokens, expected signatures, raw
|
|
||||||
secret-bearing input, or merged configs—including via `Debug` and parse errors.
|
|
||||||
- Untrusted serde: reject unknown fields where compatible and validate
|
|
||||||
security-critical defaults/ranges before numeric conversion.
|
|
||||||
- SSE/browser/CORS/trusted proxy: inspect stored ciphertext and wrapper order;
|
|
||||||
isolate user content; never reflect credentialed arbitrary origins or trust
|
|
||||||
forwarded identity from direct clients.
|
|
||||||
- Object Lock: unreadable/fabricated/unparsable metadata fails closed across
|
|
||||||
foreground, lifecycle, scanner, and force-delete paths.
|
|
||||||
|
|
||||||
Security findings distinguish unauthenticated compromise from a
|
|
||||||
low-privileged authenticated bypass.
|
|
||||||
@@ -1,22 +0,0 @@
|
|||||||
# Simplicity Lens
|
|
||||||
|
|
||||||
- Compare the production diff with the smallest equivalent local edit. Fewer
|
|
||||||
lines alone are not evidence; the replacement must preserve correctness,
|
|
||||||
compatibility, readability, and real boundaries.
|
|
||||||
- Search the touched crate, domain owner, `crates/utils`, `crates/common`, and
|
|
||||||
relevant dependencies for each new helper, constant, wrapper, or fixture.
|
|
||||||
- Reject forced reuse when normalization, error, deadline, or durability
|
|
||||||
semantics differ.
|
|
||||||
- Require a concrete trigger for every new defensive branch. Keep boundary
|
|
||||||
checks for disk/RPC/version data and checks immediately before destructive
|
|
||||||
actions.
|
|
||||||
- Flag one-caller helpers only when they merely forward or split a short linear
|
|
||||||
flow without adding domain naming, invariant isolation, or useful context.
|
|
||||||
- Ensure a replacement removes the superseded in-scope path or keeps one
|
|
||||||
canonical core behind a documented compatibility adapter.
|
|
||||||
- Remove narration/change-history comments; preserve concise safety, lock,
|
|
||||||
durability, and compatibility invariants.
|
|
||||||
- Treat tests, fixtures, generated code, and documentation separately from
|
|
||||||
production growth. Do not optimize away meaningful regression coverage.
|
|
||||||
|
|
||||||
A finding must include a concrete smaller design, not a style preference.
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
# Test-Coverage Lens
|
|
||||||
|
|
||||||
- For every behavior claim, name the focused test/check that fails if the
|
|
||||||
changed hunk is reverted. If none is practical, require the reason and
|
|
||||||
residual risk.
|
|
||||||
- Confirm tests exercise the real production path and assert returned values,
|
|
||||||
exact bytes, stored state, or the specific error variant—not only success,
|
|
||||||
`is_err()`, or no panic.
|
|
||||||
- For new flags/modes, verify each branch and ask which test fails if the branch
|
|
||||||
is inverted.
|
|
||||||
- For new error propagation, inject the failure and assert the caller observes
|
|
||||||
it; mentally replacing `?`/`return Err` with success must break a test.
|
|
||||||
- Streaming GET tests assert the complete body and length under degraded reads.
|
|
||||||
- Disk/wire-format tests use pinned foreign/legacy fixtures; same-code
|
|
||||||
round-trips are insufficient for compatibility.
|
|
||||||
- Concurrency tests use readiness polling, isolate global state, and avoid fixed
|
|
||||||
sleeps or unrealistically short timeouts. Use nextest groups when process-level
|
|
||||||
serialization is required.
|
|
||||||
- Internal metadata tests assert both RustFS and MinIO keys, not only read-back
|
|
||||||
through a helper that prefers one key.
|
|
||||||
- Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
|
|
||||||
absent vs empty vs nil UUID.
|
|
||||||
- A focused test proves only the targets/features it builds. Add compilation or
|
|
||||||
Clippy only for uncovered changed targets.
|
|
||||||
@@ -1,64 +0,0 @@
|
|||||||
---
|
|
||||||
name: arch-checks
|
|
||||||
description: Resolve failures from the repository's architecture guard scripts — check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_unsafe_code_allowances.sh, check_logging_guardrails.sh, check_doc_paths.sh. Use when make pre-commit / pre-pr or CI fails on one of these checks.
|
|
||||||
---
|
|
||||||
|
|
||||||
# Architecture Guard Checks
|
|
||||||
|
|
||||||
All five run in `make pre-commit` / `make pre-pr` and in CI. Fix the cause;
|
|
||||||
never weaken a check to get green.
|
|
||||||
|
|
||||||
## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
|
|
||||||
|
|
||||||
Enforces `composition (server, startup/init) → interface (admin,
|
|
||||||
storage/ecfs, storage/s3_api) → app → infra`; no upward imports. Server source
|
|
||||||
files are composition roots, while imports of their exported HTTP contracts
|
|
||||||
are classified as interface dependencies. Known legacy violations live in
|
|
||||||
`scripts/layer-dependency-baseline.txt`.
|
|
||||||
|
|
||||||
Dedicated `*_test.rs` and `tests/` modules are outside this production guard.
|
|
||||||
Inline `#[cfg(test)]` imports remain checked under their source file's layer;
|
|
||||||
move architecture-crossing test scaffolding into a dedicated test module.
|
|
||||||
|
|
||||||
- **New violation**: restructure your change so the dependency points
|
|
||||||
downward (move the shared type/function to the lower layer).
|
|
||||||
- **You legitimately removed a baseline entry**: run
|
|
||||||
`./scripts/check_layer_dependencies.sh --update-baseline` and commit the
|
|
||||||
shrunken baseline. Never add new entries to the baseline to make a new
|
|
||||||
violation pass.
|
|
||||||
|
|
||||||
## `check_architecture_migration_rules.sh` — required doc sections
|
|
||||||
|
|
||||||
Asserts that the core docs under `docs/architecture/` (overview,
|
|
||||||
crate-boundaries, runtime-lifecycle, readiness-matrix,
|
|
||||||
storage-control-data-plane, global-state-crate-split-plan,
|
|
||||||
ecstore-module-split-plan, …) still contain specific headings and exact
|
|
||||||
source lines. If it fails after a doc edit, you reworded or removed a
|
|
||||||
guarded line — restore the wording or update the script deliberately in the
|
|
||||||
same PR, with rationale.
|
|
||||||
|
|
||||||
## `check_unsafe_code_allowances.sh`
|
|
||||||
|
|
||||||
Every `#[allow(unsafe_code)]` needs a `SAFETY:` comment within a few lines.
|
|
||||||
Write the actual safety argument; don't add a placeholder.
|
|
||||||
|
|
||||||
## `check_logging_guardrails.sh`
|
|
||||||
|
|
||||||
A fixed list of security-sensitive files (auth, IAM, KMS, admin handlers…)
|
|
||||||
is scanned for logging violations. If you created a new sensitive file,
|
|
||||||
consider adding it to the script's `checked_files` list.
|
|
||||||
|
|
||||||
## `check_doc_paths.sh`
|
|
||||||
|
|
||||||
Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
|
|
||||||
`docs/architecture/*.md`) must not reference repo file paths that no longer
|
|
||||||
exist. If your refactor moved code, update the docs that point at it — the
|
|
||||||
error message lists `doc -> stale-path` pairs.
|
|
||||||
|
|
||||||
## `check_no_planning_docs.sh`
|
|
||||||
|
|
||||||
Planning-type documents must not be committed (see AGENTS.md "Sources of
|
|
||||||
Truth"). The guard fails if anything is tracked under `docs/superpowers/` —
|
|
||||||
`.gitignore` already ignores it, but `git add -f` bypasses that, so this closes
|
|
||||||
the hole. Fix by removing the listed file(s) with `git rm`; keep the plan or
|
|
||||||
spec in the issue tracker or a local worktree instead.
|
|
||||||
@@ -1,82 +0,0 @@
|
|||||||
---
|
|
||||||
name: code-change-verification
|
|
||||||
description: Review a commit, PR, or merged patch when the user requests ordinary code-change verification. Do not combine with adversarial-validation; use that skill instead for explicitly adversarial, substantial, or high-risk RustFS reviews.
|
|
||||||
---
|
|
||||||
|
|
||||||
# Code Change Verification
|
|
||||||
|
|
||||||
Use this skill for an ordinary requested review. If the root policy or user calls
|
|
||||||
for adversarial validation, use `adversarial-validation` instead of running both.
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
1. Read the scope: commit, PR, patch, or file list.
|
|
||||||
2. Map each changed area by risk and user impact.
|
|
||||||
3. Inspect each risky change in context.
|
|
||||||
4. Report findings first, ordered by severity.
|
|
||||||
5. Close with residual risks and verification recommendations.
|
|
||||||
|
|
||||||
## Core Workflow
|
|
||||||
|
|
||||||
### 1) Scope and assumptions
|
|
||||||
- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
|
|
||||||
- If context is missing, state assumptions before deeper analysis.
|
|
||||||
- Focus only on requested scope; avoid reviewing unrelated files.
|
|
||||||
|
|
||||||
### 2) Risk map
|
|
||||||
- Prioritize in this order:
|
|
||||||
- Data correctness and user-visible behavior
|
|
||||||
- API/contract compatibility
|
|
||||||
- Security and authz/authn boundaries
|
|
||||||
- Concurrency and lifecycle correctness
|
|
||||||
- Performance and resource usage
|
|
||||||
- Give higher priority to stateful paths, migration logic, defaults, and error handling.
|
|
||||||
|
|
||||||
### 3) Evidence-based inspection
|
|
||||||
- Read each modified hunk with neighboring context.
|
|
||||||
- Trace call paths and call-site expectations.
|
|
||||||
- Check for:
|
|
||||||
- invariant breaks and missing guards
|
|
||||||
- unchecked assumptions and null/empty/error-path handling
|
|
||||||
- stale tests, fixtures, and configs
|
|
||||||
- hidden coupling to shared helpers/constants/features
|
|
||||||
- If a point is uncertain, mark it as an open question instead of guessing.
|
|
||||||
|
|
||||||
#### Rust-specific checks (apply to all Rust changes)
|
|
||||||
|
|
||||||
Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — the canonical Rust review checklist for the unwrap/casting/cloning/locking/recursion/error-type/serde/test rules and the reuse-and-necessity checks (duplicated helpers, defensive branches without a nameable trigger, redundant error wrapping). Do not restate those rules here; carry its P0–P3 ratings over unchanged and use this skill's output format.
|
|
||||||
|
|
||||||
### 4) Findings-first output
|
|
||||||
- Order findings by severity:
|
|
||||||
- P0: critical failure, security breach, or data loss risk
|
|
||||||
- P1: high-impact regression
|
|
||||||
- P2: medium risk correctness gap
|
|
||||||
- P3: low risk/quality debt
|
|
||||||
- For each finding include:
|
|
||||||
- Severity
|
|
||||||
- `path:line` reference
|
|
||||||
- concise issue statement
|
|
||||||
- impact and likely failure mode
|
|
||||||
- specific fix or mitigation
|
|
||||||
- validation step to confirm
|
|
||||||
- If no issues exist, explicitly state `No findings` and why.
|
|
||||||
|
|
||||||
### 5) Close
|
|
||||||
- Report assumptions and unknowns.
|
|
||||||
- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
|
|
||||||
|
|
||||||
## Output Template
|
|
||||||
|
|
||||||
1. Findings
|
|
||||||
2. No findings (if applicable)
|
|
||||||
3. Assumptions / Unknowns
|
|
||||||
4. Recommended verification steps
|
|
||||||
|
|
||||||
## Finding Template
|
|
||||||
|
|
||||||
- `[P1] Missing timeout for downstream call`
|
|
||||||
- Location: `path/to/file.rs:123`
|
|
||||||
- Issue: ...
|
|
||||||
- Impact: ...
|
|
||||||
- Fix suggestion: ...
|
|
||||||
- Validation: ...
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
interface:
|
|
||||||
display_name: "Code Change Verification"
|
|
||||||
short_description: "Prioritize risks and verify code changes before merge."
|
|
||||||
default_prompt: "Use $code-change-verification for an ordinary requested diff review with prioritized findings."
|
|
||||||
@@ -1,83 +0,0 @@
|
|||||||
---
|
|
||||||
name: plugin-contract-guard
|
|
||||||
description: Invariants and change procedure for the target-plugin / extension system — plugin manifests, admin plugin/extension catalog and instance APIs, secret redaction, external-plugin install policy. Use when editing crates/targets (manifest, plugin, control_plane, catalog, runtime), crates/extension-schema, or rustfs/src/admin plugin_contract.rs / plugins_*.rs / extensions.rs / target_descriptor.rs.
|
|
||||||
---
|
|
||||||
|
|
||||||
# Plugin & Extension Contract Guard
|
|
||||||
|
|
||||||
The "plugin system" spans four surfaces that must stay consistent:
|
|
||||||
|
|
||||||
| Surface | Location |
|
|
||||||
|---|---|
|
|
||||||
| Manifests & registry | `crates/targets/src/{manifest,plugin}.rs` |
|
|
||||||
| Install/enable planning (control plane) | `crates/targets/src/control_plane.rs` |
|
|
||||||
| Extension schemas | `crates/extension-schema/src/lib.rs`, `crates/targets/src/catalog/extension.rs` |
|
|
||||||
| Admin API contract | `rustfs/src/admin/plugin_contract.rs`, `handlers/{plugins_catalog,plugins_instances,extensions,target_descriptor}.rs` |
|
|
||||||
|
|
||||||
## Hard invariants (verify before merging)
|
|
||||||
|
|
||||||
1. **Secrets have one source of truth.** Secret config keys are declared only
|
|
||||||
in the plugin manifest (`TargetPluginManifest.secret_fields`,
|
|
||||||
`crates/targets/src/manifest.rs`) and flow to admin via
|
|
||||||
`AdminTargetSpec.secret_fields`. Never add a hand-maintained per-service
|
|
||||||
secret table in a handler; if redaction misses a field, fix the manifest.
|
|
||||||
|
|
||||||
2. **Redaction must round-trip.** Instance GET responses replace secret values
|
|
||||||
with `***redacted***` (`REDACTED_SECRET_VALUE` in `plugins_instances.rs`).
|
|
||||||
Instance PUT restores the stored secret when it receives that placeholder
|
|
||||||
back (`restore_redacted_secret_values`). Any new read or write path for
|
|
||||||
target config must keep both halves: redact on the way out, restore the
|
|
||||||
placeholder on the way in. The placeholder literal must never be persisted.
|
|
||||||
|
|
||||||
3. **Fixtures never reach production responses.**
|
|
||||||
`example_external_webhook_plugin()` (`crates/targets/src/catalog/mod.rs`)
|
|
||||||
is a test/demo fixture for control-plane planning tests. Production
|
|
||||||
catalog/extension handlers must not include it; regression tests
|
|
||||||
(`plugin_catalog_never_exposes_example_or_external_fixtures`,
|
|
||||||
`extension_catalog_never_exposes_example_or_external_fixtures`) enforce it.
|
|
||||||
|
|
||||||
4. **External plugin flow is planning-only and deny-by-default.**
|
|
||||||
`plan_external_target_plugin_action` returns decisions, it executes
|
|
||||||
nothing. `TargetPluginExternalFlowGate::default()` is fully closed and
|
|
||||||
`TargetPluginInstallPolicy::default().allowed_download_hosts` is empty —
|
|
||||||
keep it that way; tests opt in via explicit policies. Install validation
|
|
||||||
requires https, an allowlisted host, a full 64-hex-char sha256 digest,
|
|
||||||
signature and provenance URIs, and an artifact matching the host
|
|
||||||
`target_triple`.
|
|
||||||
|
|
||||||
5. **Custom target types must not collide.** Unknown target types get an
|
|
||||||
interned unique `custom:<type>` plugin id (`custom_plugin_id` in
|
|
||||||
`manifest.rs`). Custom plugins with secrets must register via
|
|
||||||
`TargetPluginDescriptor::with_manifest` and declare `secret_fields`;
|
|
||||||
`::new` derives a manifest with no secrets.
|
|
||||||
|
|
||||||
## Changing the admin JSON contract
|
|
||||||
|
|
||||||
- Shapes are locked twice in `plugin_contract.rs` tests: insta snapshots
|
|
||||||
(`rustfs/src/admin/snapshots/`) plus literal `json!` assertions. Update
|
|
||||||
both deliberately; a shape change is a console-facing API change.
|
|
||||||
- Field naming is `snake_case`, except discovery blocks
|
|
||||||
(`runtimeCapabilities`, `clusterSnapshot`, `extensionsCatalog`) which are
|
|
||||||
camelCase **by cross-endpoint convention** (same shape in `system.rs`,
|
|
||||||
`console.rs`, `pools.rs`). Do not "fix" that inconsistency locally.
|
|
||||||
- Contract types deliberately duplicate `rustfs_targets` types
|
|
||||||
(anti-corruption layer). Add a `From` impl; do not serialize internal
|
|
||||||
types directly.
|
|
||||||
|
|
||||||
## Handler conventions
|
|
||||||
|
|
||||||
- Every new admin plugin/extension route needs authorization at the top of
|
|
||||||
`call` and an `include_str!` guard test asserting it (repo-wide pattern —
|
|
||||||
see `plugin_instance_handlers_require_admin_authorization_contract`).
|
|
||||||
- Reads use `GetBucketTargetAction` (instances) or `ServerInfoAdminAction`
|
|
||||||
(catalogs); writes use `SetBucketTargetAction`.
|
|
||||||
- Refresh persisted module switches once per request
|
|
||||||
(`refresh_persisted_module_switches`), then evaluate the sync
|
|
||||||
`module_disabled_block_reason` per domain — do not re-read the store per
|
|
||||||
domain or per instance.
|
|
||||||
|
|
||||||
## Generic bounds
|
|
||||||
|
|
||||||
Event-payload generics use the `PluginEvent` blanket trait
|
|
||||||
(`crates/targets/src/plugin.rs`). Do not respell
|
|
||||||
`Send + Sync + 'static + Clone + Serialize + DeserializeOwned`.
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
---
|
|
||||||
name: pr-creation-checker
|
|
||||||
description: Perform the final RustFS PR preflight and draft compliant English title/body metadata immediately before creating or updating a PR. Do not use during implementation or as a second general code review.
|
|
||||||
---
|
|
||||||
|
|
||||||
# PR Creation Checker
|
|
||||||
|
|
||||||
Use this skill only at the PR boundary. Reuse completed diff review and
|
|
||||||
verification evidence; do not reread the repository or rerun equivalent checks.
|
|
||||||
|
|
||||||
## Preflight
|
|
||||||
|
|
||||||
1. Confirm the branch is based on current `origin/main` and contains only the
|
|
||||||
intended task diff.
|
|
||||||
2. Inspect `git diff --stat`, `git diff --check`, and changed file names for
|
|
||||||
secrets, logs, generated artifacts, or unrelated edits.
|
|
||||||
3. Confirm the checks selected by root `AGENTS.md` passed on the final diff.
|
|
||||||
Do not replace focused behavioral tests with a generic gate or rerun checks
|
|
||||||
already covered by an unchanged umbrella run.
|
|
||||||
4. Read `.github/pull_request_template.md`. Consult `Makefile`, `.config/make/`,
|
|
||||||
or CI only when the required command/current gate is uncertain.
|
|
||||||
5. Return `BLOCKED` for an unclean scope, missing required evidence, failed
|
|
||||||
required checks, or non-compliant metadata.
|
|
||||||
|
|
||||||
## Metadata
|
|
||||||
|
|
||||||
- Title: English Conventional Commit, at most 72 characters, with no tool
|
|
||||||
prefix.
|
|
||||||
- Body: English, exact template headings, `N/A` where needed, concise rationale,
|
|
||||||
actual verification commands, and material risks/rollback notes.
|
|
||||||
- Use repository-relative paths; never include local absolute paths.
|
|
||||||
- Keep prose paragraphs on one logical line and never include the literal
|
|
||||||
sequence `\n`.
|
|
||||||
- Use a temporary body file with `gh pr create --body-file` or
|
|
||||||
`gh pr edit --body-file`; never pass multiline Markdown inline.
|
|
||||||
|
|
||||||
## Output
|
|
||||||
|
|
||||||
- Status: `READY` or `BLOCKED`.
|
|
||||||
- Title.
|
|
||||||
- Complete PR body.
|
|
||||||
- Verification commands and results.
|
|
||||||
- Risks or `N/A`.
|
|
||||||
|
|
||||||
Immediately before the GitHub write, repeat only the five preflight checks above
|
|
||||||
against the final head.
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
interface:
|
|
||||||
display_name: "PR Creation Checker"
|
|
||||||
short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
|
|
||||||
default_prompt: "Use $pr-creation-checker for final PR preflight and compliant English title/body metadata."
|
|
||||||
@@ -1,123 +0,0 @@
|
|||||||
---
|
|
||||||
name: rust-code-quality
|
|
||||||
description: Run a focused Rust quality review when the user requests one, when reviewing a Rust PR/commit, or when another selected review workflow delegates Rust-specific checks. Do not auto-load for every implementation edit.
|
|
||||||
---
|
|
||||||
|
|
||||||
# Rust Code Quality Gate
|
|
||||||
|
|
||||||
Use this skill for a dedicated Rust review to cover rules that `cargo clippy`
|
|
||||||
does not catch.
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
1. Identify changed `.rs` files.
|
|
||||||
2. Run automated checks on changed files.
|
|
||||||
3. Run manual review checklist on the diff.
|
|
||||||
4. Resolve or rebut every finding with evidence; P0/P1 findings cannot be deferred.
|
|
||||||
|
|
||||||
## Automated Checks
|
|
||||||
|
|
||||||
Use these searches to find candidates in changed `.rs` files. Inspect syntax,
|
|
||||||
`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
|
|
||||||
filters do not reliably distinguish production code from tests.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. unwrap/expect candidates
|
|
||||||
rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
|
|
||||||
|
|
||||||
# 2. Silent type truncation via `as` cast
|
|
||||||
rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
|
|
||||||
|
|
||||||
# 3. String as error type
|
|
||||||
rg -n 'Result<.*String>' <changed-files>
|
|
||||||
|
|
||||||
# 4. Box<dyn Error> in public APIs
|
|
||||||
rg -n 'Box<dyn.*Error' <changed-files>
|
|
||||||
|
|
||||||
# 5. println/eprintln in production
|
|
||||||
rg -n 'println!\|eprintln!' <changed-files>
|
|
||||||
|
|
||||||
# 6. Ordering::Relaxed usage (verify each is intentional)
|
|
||||||
rg -n 'Ordering::Relaxed' <changed-files>
|
|
||||||
|
|
||||||
# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
|
|
||||||
rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
|
|
||||||
```
|
|
||||||
|
|
||||||
## Manual Review Checklist
|
|
||||||
|
|
||||||
For the Rust diff under review, verify:
|
|
||||||
|
|
||||||
### Error Handling
|
|
||||||
- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
|
|
||||||
- [ ] No `Result<_, String>` in public API signatures
|
|
||||||
- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
|
|
||||||
- [ ] `Error::source()` is overridden when inner error is stored
|
|
||||||
- [ ] Error messages are actionable without exposing secret input
|
|
||||||
|
|
||||||
### Type Safety
|
|
||||||
- [ ] No silent `as` truncation (negative→unsigned, large→small)
|
|
||||||
- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
|
|
||||||
- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
|
|
||||||
|
|
||||||
### Concurrency
|
|
||||||
- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
|
|
||||||
- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
|
|
||||||
- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
|
|
||||||
- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
|
|
||||||
|
|
||||||
### Memory and Performance
|
|
||||||
- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
|
|
||||||
- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
|
|
||||||
|
|
||||||
### Recursion Safety
|
|
||||||
- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
|
|
||||||
- [ ] Tree/cache traversals handle corrupted/cyclic input safely
|
|
||||||
|
|
||||||
### Testing
|
|
||||||
- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
|
|
||||||
- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
|
|
||||||
- [ ] Test volume and line count are never treated as production-code growth
|
|
||||||
|
|
||||||
### Serde
|
|
||||||
- [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
|
|
||||||
- [ ] `#[serde(default)]` not used on security-critical fields without validation
|
|
||||||
|
|
||||||
### Code Hygiene
|
|
||||||
- [ ] No `#![allow(dead_code)]` at crate root
|
|
||||||
- [ ] No camelCase statics or Hungarian notation
|
|
||||||
- [ ] New string literals don't duplicate existing constants
|
|
||||||
|
|
||||||
### Reuse and Necessity
|
|
||||||
- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
|
|
||||||
- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
|
|
||||||
- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
|
|
||||||
- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
|
|
||||||
- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
|
|
||||||
|
|
||||||
## Severity Classification
|
|
||||||
|
|
||||||
- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
|
|
||||||
- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
|
|
||||||
- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
|
|
||||||
- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
|
|
||||||
|
|
||||||
## Output Template
|
|
||||||
|
|
||||||
```
|
|
||||||
## Rust Code Quality Report
|
|
||||||
|
|
||||||
### Automated Scan
|
|
||||||
- unwrap/expect candidates inspected: N
|
|
||||||
- numeric-cast candidates inspected: N
|
|
||||||
- error-type candidates inspected: N
|
|
||||||
- output-macro candidates inspected: N
|
|
||||||
|
|
||||||
### Findings
|
|
||||||
- [P1] `path:line` — description
|
|
||||||
- Fix: ...
|
|
||||||
- Validation: ...
|
|
||||||
|
|
||||||
### Verdict
|
|
||||||
PASS / BLOCKED (list blocking findings)
|
|
||||||
```
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
---
|
|
||||||
name: rustfs-logging-governance
|
|
||||||
description: Add or review RustFS `tracing` events with the repository field shape, level policy, privacy boundaries, and guardrails. Use when a change adds or edits a tracing macro/instrumentation site or the logging guardrail script.
|
|
||||||
---
|
|
||||||
|
|
||||||
# RustFS Logging Governance
|
|
||||||
|
|
||||||
Apply this skill only to changed logging sites; do not turn a local log edit into
|
|
||||||
a broad logging cleanup.
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
|
|
||||||
1. Read the changed function/module context and classify the site as lifecycle,
|
|
||||||
request/hot path, fallback, external fetch, or summary.
|
|
||||||
2. Match neighboring structured events and reuse existing `EVENT_*`,
|
|
||||||
`LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` constants.
|
|
||||||
3. Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
|
|
||||||
then context) and a short label last.
|
|
||||||
4. Select the level by operational meaning:
|
|
||||||
- `error`: behavior/security-affecting failure;
|
|
||||||
- `warn`: degraded/fallback/operator-actionable state;
|
|
||||||
- `info`: low-frequency lifecycle/mode change;
|
|
||||||
- `debug`: targeted diagnostics;
|
|
||||||
- `trace`: repetitive request/object/shard success paths.
|
|
||||||
5. Never log secrets, tokens, auth headers, credential payloads, raw
|
|
||||||
attacker-controlled bodies, or merged config dumps. Error strings and
|
|
||||||
`Debug` output are log surfaces too.
|
|
||||||
6. Prefer one aggregate summary over inventories or startup banners.
|
|
||||||
7. Run `./scripts/check_logging_guardrails.sh` and the checks selected by root
|
|
||||||
`AGENTS.md`.
|
|
||||||
|
|
||||||
Read [logging-governance.md](references/logging-governance.md) only for a broad
|
|
||||||
logging audit, event-model migration, or guardrail expansion. Ordinary single-
|
|
||||||
site edits do not require the full workspace scope map.
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
interface:
|
|
||||||
display_name: "RustFS Logging Governance"
|
|
||||||
short_description: "Standardize RustFS logs with structured events and guardrails."
|
|
||||||
default_prompt: "Use $rustfs-logging-governance to standardize or review RustFS logging, reduce noise, and update guardrails."
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
# Logging Audit and Migration Reference
|
|
||||||
|
|
||||||
Read this reference only for a broad logging audit, an event-model migration,
|
|
||||||
or a change to `scripts/check_logging_guardrails.sh`. Use `Cargo.toml` for the
|
|
||||||
current workspace/crate list instead of maintaining one here.
|
|
||||||
|
|
||||||
## Audit by Operational Role
|
|
||||||
|
|
||||||
- Server/protocol/admin: lifecycle, authorization failures, request boundaries,
|
|
||||||
and degraded subsystems; avoid normal request success at `info`.
|
|
||||||
- Storage/heal/scanner/capacity: integrity failures and aggregate lifecycle;
|
|
||||||
avoid per-object, per-shard, and folder iteration noise.
|
|
||||||
- IAM/policy/credentials/KMS/crypto: safe identifiers and enforcement results;
|
|
||||||
never emit secrets, claims, payloads, or expected authenticators.
|
|
||||||
- Notify/audit/targets: target lifecycle and batch/backpressure summaries; avoid
|
|
||||||
per-event success logs.
|
|
||||||
- Locking/concurrency/I/O foundations: contention anomalies and state changes;
|
|
||||||
prefer metrics for high-frequency worker/permit signals.
|
|
||||||
- Shared type/schema crates: log at the operational caller boundary unless the
|
|
||||||
crate itself owns the failure context.
|
|
||||||
|
|
||||||
## Event Shape
|
|
||||||
|
|
||||||
Prefer stable fields in this order when available:
|
|
||||||
|
|
||||||
1. `event`
|
|
||||||
2. `component`
|
|
||||||
3. `subsystem`
|
|
||||||
4. `state` or `result`
|
|
||||||
5. stable context such as mode, duration, reason, counts, safe identifiers, or
|
|
||||||
capacity/permit values
|
|
||||||
6. short message label
|
|
||||||
|
|
||||||
Reuse the module's constants and neighboring field names. Do not create aliases
|
|
||||||
for the same concept.
|
|
||||||
|
|
||||||
## Patterns to Retire
|
|
||||||
|
|
||||||
- sentence-style lifecycle announcements;
|
|
||||||
- startup banners and checklist lines;
|
|
||||||
- repetitive success logs at `info`/`debug`;
|
|
||||||
- raw inventories when an aggregate count is sufficient;
|
|
||||||
- fallback prose with values embedded in the message;
|
|
||||||
- `?value`/`Debug` output for credential-bearing or attacker-controlled data;
|
|
||||||
- logging a parse input when the malformed input may itself be a secret.
|
|
||||||
|
|
||||||
## Guardrail Changes
|
|
||||||
|
|
||||||
When expanding `scripts/check_logging_guardrails.sh`:
|
|
||||||
|
|
||||||
1. Add only files/patterns intentionally migrated in the same change.
|
|
||||||
2. Keep patterns concrete and grep-friendly.
|
|
||||||
3. Do not encode a style that remains valid elsewhere as a global ban.
|
|
||||||
4. Run the guardrail script and the root validation tier.
|
|
||||||
5. Treat the script as a floor; manually verify level, field shape, and privacy.
|
|
||||||
|
|
||||||
Useful search seeds for the changed surface:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
rg -n 'error!|warn!|info!|debug!|trace!|#\[instrument' <changed-paths>
|
|
||||||
rg -n '\?[^,)]|secret|token|credential|authorization|merged_config' <changed-paths>
|
|
||||||
```
|
|
||||||
@@ -1,233 +0,0 @@
|
|||||||
---
|
|
||||||
name: rustfs-release-publish
|
|
||||||
description: "Run the end-to-end RustFS console gate, version bump, preview validation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
|
|
||||||
---
|
|
||||||
# RustFS Release Publish (preview-validated pipeline)
|
|
||||||
|
|
||||||
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
|
|
||||||
|
|
||||||
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
|
|
||||||
|
|
||||||
Pipeline shape:
|
|
||||||
|
|
||||||
```
|
|
||||||
check console main against its latest Release
|
|
||||||
-> if ahead: publish console -> wait for Release asset + latest API
|
|
||||||
-> bump RustFS version files to <target> (final version, ONE commit) -> merge
|
|
||||||
-> tag <preview-tag> at that commit -> CI green
|
|
||||||
-> verify preview Release assets -> run binary locally + console checks
|
|
||||||
-> validate with latest rc client
|
|
||||||
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
|
|
||||||
```
|
|
||||||
|
|
||||||
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
|
|
||||||
|
|
||||||
## Required inputs
|
|
||||||
|
|
||||||
- Final target version, for example `1.0.0-beta.10`.
|
|
||||||
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
|
|
||||||
|
|
||||||
If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
|
|
||||||
|
|
||||||
## Semver gate — confirm the target version before touching anything
|
|
||||||
|
|
||||||
Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
|
|
||||||
|
|
||||||
```
|
|
||||||
1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
|
|
||||||
```
|
|
||||||
|
|
||||||
Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
|
|
||||||
|
|
||||||
Rules:
|
|
||||||
|
|
||||||
- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
|
|
||||||
- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
|
|
||||||
- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
|
|
||||||
|
|
||||||
## Preview tag naming
|
|
||||||
|
|
||||||
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
|
|
||||||
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
|
|
||||||
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
|
|
||||||
|
|
||||||
## Hard rules
|
|
||||||
|
|
||||||
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
|
|
||||||
- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
|
|
||||||
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
|
|
||||||
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
|
|
||||||
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
|
|
||||||
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
|
|
||||||
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
|
|
||||||
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
|
|
||||||
- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
|
|
||||||
|
|
||||||
## Phase 0 — Preflight
|
|
||||||
|
|
||||||
- `git status --short` clean; `git fetch origin main --tags`.
|
|
||||||
- `gh auth status` works; confirm you can view `gh release list -L 3`.
|
|
||||||
- Confirm the exact final target version with the user if not explicit.
|
|
||||||
|
|
||||||
### Console release gate
|
|
||||||
|
|
||||||
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
|
|
||||||
|
|
||||||
1. Read the latest published Console tag and compare it with Console `main`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
CONSOLE_REPO="rustfs/console"
|
|
||||||
CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
|
|
||||||
gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
|
|
||||||
--jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
|
|
||||||
```
|
|
||||||
|
|
||||||
- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
|
|
||||||
- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
|
|
||||||
- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
|
|
||||||
|
|
||||||
2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
CONSOLE_SCRATCH=$(mktemp -d)
|
|
||||||
gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
|
|
||||||
git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
|
|
||||||
CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
|
|
||||||
git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
|
|
||||||
gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
|
|
||||||
```
|
|
||||||
|
|
||||||
If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
|
|
||||||
git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
|
|
||||||
```
|
|
||||||
|
|
||||||
Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
|
|
||||||
|
|
||||||
3. Find the exact tag run and wait for completion:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
|
|
||||||
gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
|
|
||||||
test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
|
|
||||||
test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
|
|
||||||
--jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
|
|
||||||
```
|
|
||||||
|
|
||||||
Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
|
|
||||||
|
|
||||||
## Phase 1 — Version bump to the final target (once)
|
|
||||||
|
|
||||||
- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
|
|
||||||
- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
|
|
||||||
- Get the PR merged into main. Record the resulting main commit:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git fetch origin main
|
|
||||||
PREVIEW_HASH=$(git rev-parse origin/main) # must contain the bump PR
|
|
||||||
```
|
|
||||||
|
|
||||||
`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
|
|
||||||
|
|
||||||
## Phase 2 — Publish the preview tag
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
|
|
||||||
git push origin "<preview-tag>"
|
|
||||||
```
|
|
||||||
|
|
||||||
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
|
|
||||||
|
|
||||||
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
|
|
||||||
|
|
||||||
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
|
|
||||||
|
|
||||||
## Phase 3 — CI and preview Release verification
|
|
||||||
|
|
||||||
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
|
|
||||||
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
|
|
||||||
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
|
|
||||||
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
|
|
||||||
- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
|
|
||||||
|
|
||||||
## Phase 4 — Run the artifact locally, verify the console
|
|
||||||
|
|
||||||
Work inside the session scratchpad directory; never leave stray data dirs.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
|
|
||||||
cd "$SCRATCH" && unzip -o rustfs-*.zip
|
|
||||||
./rustfs --version # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
|
|
||||||
mkdir -p data
|
|
||||||
RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
|
|
||||||
```
|
|
||||||
|
|
||||||
Defaults: S3 endpoint `:9000`, embedded console `:9001`.
|
|
||||||
|
|
||||||
Checks (all must pass):
|
|
||||||
|
|
||||||
- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
|
|
||||||
- `curl -fsS http://localhost:9000/health/ready` returns ready.
|
|
||||||
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
|
|
||||||
- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
|
|
||||||
|
|
||||||
## Phase 5 — Validate with the latest rc client
|
|
||||||
|
|
||||||
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
|
|
||||||
|
|
||||||
- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
|
|
||||||
- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
|
|
||||||
rc ls preview/
|
|
||||||
rc mb preview/rel-check
|
|
||||||
rc cp <local-file> preview/rel-check/
|
|
||||||
rc stat preview/rel-check/<file>
|
|
||||||
rc cat preview/rel-check/<file> # matches source
|
|
||||||
rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
|
|
||||||
rc cp -r <local-dir>/ preview/rel-check/dir/
|
|
||||||
rc find preview/rel-check --name "*"
|
|
||||||
rc share download preview/rel-check/<file> --expire 1h # presigned URL fetchable via curl
|
|
||||||
rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
|
|
||||||
rc rb preview/rel-check
|
|
||||||
rc admin user list preview/
|
|
||||||
rc admin user add preview/ relcheckuser relchecksecret12
|
|
||||||
rc admin user remove preview/ relcheckuser
|
|
||||||
rc alias remove preview
|
|
||||||
```
|
|
||||||
|
|
||||||
- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
|
|
||||||
|
|
||||||
## Phase 6 — Publish the final tag on the validated commit
|
|
||||||
|
|
||||||
No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git fetch origin --tags
|
|
||||||
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
|
|
||||||
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
|
|
||||||
git push origin "<target>"
|
|
||||||
```
|
|
||||||
|
|
||||||
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
|
|
||||||
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
|
|
||||||
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
|
|
||||||
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
|
|
||||||
|
|
||||||
## Output contract
|
|
||||||
|
|
||||||
Always report:
|
|
||||||
|
|
||||||
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
|
|
||||||
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
|
|
||||||
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
|
|
||||||
- Any deviation from this pipeline and why the user approved it.
|
|
||||||
@@ -1,120 +0,0 @@
|
|||||||
---
|
|
||||||
name: rustfs-release-version-bump
|
|
||||||
description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
|
|
||||||
---
|
|
||||||
# RustFS Release Version Bump
|
|
||||||
|
|
||||||
Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).
|
|
||||||
|
|
||||||
Validated baseline: release pattern used in PR `#2957`.
|
|
||||||
|
|
||||||
## Required inputs
|
|
||||||
|
|
||||||
- Exact target version, for example `1.0.0-beta.4`.
|
|
||||||
- Delivery scope:
|
|
||||||
- Local only (`edit/verify`).
|
|
||||||
- Local + git (`commit/push`).
|
|
||||||
- Full GitHub flow (`commit/push/PR`).
|
|
||||||
|
|
||||||
If target version is missing or ambiguous, stop and ask before editing.
|
|
||||||
|
|
||||||
Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
|
|
||||||
|
|
||||||
## Read before editing
|
|
||||||
|
|
||||||
- `AGENTS.md` (root and nearest path-specific files).
|
|
||||||
- `.github/pull_request_template.md`.
|
|
||||||
- Current branch status and diff against `origin/main`.
|
|
||||||
|
|
||||||
## Default release file scope
|
|
||||||
|
|
||||||
Treat the following file list as the default checklist for each release bump:
|
|
||||||
|
|
||||||
- `Cargo.toml`
|
|
||||||
- `Cargo.lock`
|
|
||||||
- `README.md`
|
|
||||||
- `README_ZH.md`
|
|
||||||
- `flake.nix`
|
|
||||||
- `helm/rustfs/Chart.yaml`
|
|
||||||
- `rustfs.spec`
|
|
||||||
|
|
||||||
Only drop a file when the current repository release process clearly no longer requires it.
|
|
||||||
|
|
||||||
## Hard release policy
|
|
||||||
|
|
||||||
- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
|
|
||||||
- Helm chart version mapping follows `beta.N -> 0.N.0`.
|
|
||||||
- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
|
|
||||||
- Do not change these rules without explicit confirmation.
|
|
||||||
|
|
||||||
## Step-by-step workflow
|
|
||||||
|
|
||||||
1. Confirm intent and isolate scope
|
|
||||||
- Confirm target version string exactly.
|
|
||||||
- Confirm whether user requested local-only or full GitHub flow.
|
|
||||||
- Inspect current branch and ensure only release-related files are touched for this task.
|
|
||||||
|
|
||||||
2. Update workspace versions
|
|
||||||
- Bump `[workspace.package].version` in `Cargo.toml`.
|
|
||||||
- Bump internal workspace crate dependency versions in `Cargo.toml`.
|
|
||||||
- Update `Cargo.lock` so workspace package versions match target version.
|
|
||||||
- Re-scan for partial leftovers.
|
|
||||||
|
|
||||||
3. Update release assets
|
|
||||||
- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
|
|
||||||
- `flake.nix`: update package version to target version.
|
|
||||||
- `helm/rustfs/Chart.yaml`:
|
|
||||||
- `appVersion` = target version.
|
|
||||||
- `version` follows chart mapping rule, for example:
|
|
||||||
- `1.0.0-beta.3` -> `0.3.0`
|
|
||||||
- `1.0.0-beta.4` -> `0.4.0`
|
|
||||||
- `rustfs.spec`:
|
|
||||||
- Set `Release` to prerelease suffix (example `beta.4`).
|
|
||||||
- Add/update top changelog entry with exact format:
|
|
||||||
- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
|
|
||||||
- `- Update RPM package to RustFS 1.0.0-beta.4`
|
|
||||||
- Changelog identity and time must come from current environment:
|
|
||||||
- `git config --get user.name`
|
|
||||||
- `git config --get user.email`
|
|
||||||
- `date '+%a %b %d %Y'`
|
|
||||||
- Changelog version text must match target release version exactly.
|
|
||||||
|
|
||||||
4. Verify before shipping
|
|
||||||
- Run:
|
|
||||||
- `make pre-commit`
|
|
||||||
- If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
|
|
||||||
|
|
||||||
5. Commit strategy
|
|
||||||
- Preferred split when both parts changed:
|
|
||||||
- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
|
|
||||||
- `chore(release): align release assets for <version>` for docs and packaging files.
|
|
||||||
- If user asks for one commit, use one commit.
|
|
||||||
- Stage only intended release files; do not include unrelated working tree changes.
|
|
||||||
|
|
||||||
6. Push and PR
|
|
||||||
- Push branch:
|
|
||||||
- `git push -u origin <branch>` (first push), or `git push` (tracking already exists).
|
|
||||||
- Create PR with template headings unchanged:
|
|
||||||
- `gh pr create --base main --head <branch> --title ... --body-file ...`
|
|
||||||
- PR title/body must be English.
|
|
||||||
- Use `N/A` for non-applicable template sections.
|
|
||||||
- Include verification commands and any `BLOCKED` reason clearly.
|
|
||||||
|
|
||||||
## Recommended check commands
|
|
||||||
|
|
||||||
- `git status --short --branch`
|
|
||||||
- `git diff --name-only origin/main...HEAD`
|
|
||||||
- `git diff --stat origin/main...HEAD`
|
|
||||||
- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
|
|
||||||
- `make pre-commit`
|
|
||||||
|
|
||||||
## Output contract
|
|
||||||
|
|
||||||
When using this skill, always report:
|
|
||||||
|
|
||||||
- Target version.
|
|
||||||
- Files changed.
|
|
||||||
- Any assumptions or uncertainties requiring confirmation.
|
|
||||||
- Verification result (`PASSED` or `BLOCKED`) with key evidence.
|
|
||||||
- Commit message(s) used.
|
|
||||||
- Push status and PR URL when GitHub flow is requested.
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
interface:
|
|
||||||
display_name: "RustFS Release Bump"
|
|
||||||
short_description: "Prepare RustFS release branches like PR #2957."
|
|
||||||
default_prompt: "Use $rustfs-release-version-bump to prepare and verify an exact RustFS release-version bump."
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
---
|
|
||||||
name: security-advisory-lessons
|
|
||||||
description: Perform a dedicated RustFS security/advisory review for authn/authz, IAM, RPC trust, paths, secrets, browser isolation, encryption, Object Lock, or other security boundaries. Use only when the user requests a security/advisory review or an adversarial review explicitly escalates to the full advisory map; do not auto-load solely because code touches a sensitive path.
|
|
||||||
---
|
|
||||||
|
|
||||||
# RustFS Security Advisory Lessons
|
|
||||||
|
|
||||||
Use this skill as the deep security lens. For a normal adversarial review with a
|
|
||||||
matched security surface, the concise security reference under
|
|
||||||
`adversarial-validation` is sufficient.
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
|
|
||||||
1. Freeze the exact diff/head and identify the changed trust boundaries.
|
|
||||||
2. Read [advisory-patterns.md](references/advisory-patterns.md), then apply only
|
|
||||||
the matching sections. Useful headings are
|
|
||||||
auth/admin, IAM/STS/OIDC, policy/plugins, S3/copy/multipart, protocols, paths,
|
|
||||||
secrets/logging/RPC, browser/CORS/proxy, SSE, Object Lock, and serde.
|
|
||||||
3. Trace unauthenticated, low-privilege, wrong-action/owner/bucket, malformed,
|
|
||||||
and default-config cases. Security decisions must fail closed.
|
|
||||||
4. Require a focused negative regression test for the bypass/exploit form, not
|
|
||||||
only the intended success path. State residual risk when a test is impractical.
|
|
||||||
5. Report proven vulnerabilities separately from defense-in-depth hardening.
|
|
||||||
|
|
||||||
When advisory currentness matters, fetch the live inventory instead of treating
|
|
||||||
the reference as a status mirror:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
gh api repos/rustfs/rustfs/security-advisories --paginate \
|
|
||||||
--jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
|
|
||||||
```
|
|
||||||
|
|
||||||
Fetch an individual advisory only when the live summary indicates a new or
|
|
||||||
changed lesson.
|
|
||||||
|
|
||||||
## Finding Standard
|
|
||||||
|
|
||||||
Each finding includes severity, `file:line`, attacker prerequisites, concrete
|
|
||||||
input/path, impact, smallest safe fix, and a regression check. Do not exaggerate
|
|
||||||
unauthenticated impact when the actual issue requires authenticated low privilege.
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
interface:
|
|
||||||
display_name: "Security Advisory Lessons"
|
|
||||||
short_description: "Apply advisory lessons in reviews."
|
|
||||||
default_prompt: "Use $security-advisory-lessons for a dedicated RustFS security review grounded in past advisories."
|
|
||||||
@@ -1,147 +0,0 @@
|
|||||||
# RustFS Advisory Pattern Map
|
|
||||||
|
|
||||||
This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
|
|
||||||
|
|
||||||
When current advisory state, severity, URLs, or full text matters, fetch it live:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
gh api repos/rustfs/rustfs/security-advisories --paginate \
|
|
||||||
--jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
|
|
||||||
gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
|
|
||||||
```
|
|
||||||
|
|
||||||
Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
|
|
||||||
|
|
||||||
## Pattern Index
|
|
||||||
|
|
||||||
### Admin authorization and route exposure
|
|
||||||
|
|
||||||
- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
|
|
||||||
- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
|
|
||||||
- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
|
|
||||||
- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
|
|
||||||
- `GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
|
|
||||||
- `GHSA-796f-j7xp-hwf4`: `/rustfs/admin/v3/list-remote-targets` checked only that credentials existed and leaked replication target credentials. Lesson: replication target reads are privileged admin operations, and stored remote credentials require strict authz plus response redaction review.
|
|
||||||
- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
|
|
||||||
|
|
||||||
### IAM import, service accounts, and privilege boundaries
|
|
||||||
|
|
||||||
- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
|
|
||||||
- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
|
|
||||||
- `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
|
|
||||||
- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
|
|
||||||
- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
|
|
||||||
|
|
||||||
### STS, OIDC, and federation flows
|
|
||||||
|
|
||||||
- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
|
|
||||||
- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
|
|
||||||
- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
|
|
||||||
- `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
|
|
||||||
- `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
|
|
||||||
|
|
||||||
### IAM policy conditions and external policy plugins
|
|
||||||
|
|
||||||
- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
|
|
||||||
- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
|
|
||||||
- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
|
|
||||||
|
|
||||||
### S3 object actions, copy, multipart, and upload policy validation
|
|
||||||
|
|
||||||
- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
|
|
||||||
- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
|
|
||||||
- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
|
|
||||||
- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
|
|
||||||
- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
|
|
||||||
|
|
||||||
### Protocol frontends and IAM parity
|
|
||||||
|
|
||||||
- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
|
|
||||||
- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
|
|
||||||
- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
|
|
||||||
|
|
||||||
### Filesystem paths and object key traversal
|
|
||||||
|
|
||||||
- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
|
|
||||||
- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
|
|
||||||
- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
|
|
||||||
|
|
||||||
### Secrets, defaults, and cryptographic misuse
|
|
||||||
|
|
||||||
- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
|
|
||||||
- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
|
|
||||||
- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
|
|
||||||
- `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
|
|
||||||
- `GHSA-m77q-r63m-pj89`: STS JWTs used the root secret key as the shared token signing key, allowing token forgery when the root secret was known. Lesson: STS signing keys need key separation, rotation, and key IDs; do not reuse root credentials for JWT/HMAC signing.
|
|
||||||
- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
|
|
||||||
|
|
||||||
### Sensitive logging and debug output
|
|
||||||
|
|
||||||
- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
|
|
||||||
- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
|
|
||||||
- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
|
|
||||||
|
|
||||||
### RPC input validation and panic safety
|
|
||||||
|
|
||||||
- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
|
|
||||||
- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
|
|
||||||
- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
|
|
||||||
|
|
||||||
### Browser, CORS, and console isolation
|
|
||||||
|
|
||||||
- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
|
|
||||||
- `GHSA-7gcx-wg4x-q9x6`: an incomplete preview fix reintroduced extension-based PDF detection and bypassed the sandboxed fallback for attacker-controlled content. Lesson: browser-surface fixes need regression tests for alternate viewers and file-type branches, and preview trust must come from validated content type plus sandboxing rather than object names.
|
|
||||||
- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
|
|
||||||
|
|
||||||
### Trusted proxy and source IP conditions
|
|
||||||
|
|
||||||
- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
|
|
||||||
|
|
||||||
### SSE and on-disk storage invariants
|
|
||||||
|
|
||||||
- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
|
|
||||||
|
|
||||||
### Object Lock and retention invariants
|
|
||||||
|
|
||||||
- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
|
|
||||||
|
|
||||||
### Serde deserialization and input validation
|
|
||||||
|
|
||||||
- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
|
|
||||||
- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
|
|
||||||
- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` or clamp.
|
|
||||||
- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
|
|
||||||
|
|
||||||
## Useful Search Seeds
|
|
||||||
|
|
||||||
Use these targeted searches when a diff touches security-sensitive code:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
|
|
||||||
rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
|
|
||||||
rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
|
|
||||||
rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
|
|
||||||
rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
|
|
||||||
rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
|
|
||||||
rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
|
|
||||||
rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
|
|
||||||
rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
|
|
||||||
rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
|
|
||||||
rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
|
|
||||||
```
|
|
||||||
|
|
||||||
## Minimum Regression Test Expectations
|
|
||||||
|
|
||||||
- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
|
|
||||||
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
|
|
||||||
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
|
|
||||||
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
|
|
||||||
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
|
|
||||||
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
|
|
||||||
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
|
|
||||||
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
|
|
||||||
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
|
|
||||||
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
|
|
||||||
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
|
|
||||||
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
|
|
||||||
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
|
|
||||||
@@ -1,66 +0,0 @@
|
|||||||
---
|
|
||||||
name: test-coverage-improver
|
|
||||||
description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
|
|
||||||
---
|
|
||||||
|
|
||||||
# Test Coverage Improver
|
|
||||||
|
|
||||||
Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
|
|
||||||
|
|
||||||
## Usage assumptions
|
|
||||||
- Focus scope is either changed lines/files, a module, or the whole repository.
|
|
||||||
- Coverage artifact must be generated or provided in a supported format.
|
|
||||||
- If required context is missing, call out assumptions explicitly before proposing work.
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
|
|
||||||
1. Define scope and baseline
|
|
||||||
- Confirm target language, framework, and branch.
|
|
||||||
- Confirm whether the scope is changed files only or full-repo.
|
|
||||||
|
|
||||||
2. Produce coverage snapshot
|
|
||||||
- Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
|
|
||||||
- JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
|
|
||||||
- Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
|
|
||||||
- Collect total, per-file, and changed-line coverage.
|
|
||||||
|
|
||||||
3. Rank highest-risk gaps
|
|
||||||
- Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
|
|
||||||
- Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
|
|
||||||
- Keep shortlist to 5–8 gaps.
|
|
||||||
- For each gap, capture: file, lines, uncovered branches, and estimated risk score.
|
|
||||||
|
|
||||||
4. Propose high-impact tests
|
|
||||||
- For each shortlisted gap, output:
|
|
||||||
- Intent and expected behavior.
|
|
||||||
- Normal, edge, and failure scenarios.
|
|
||||||
- Assertions and side effects to verify.
|
|
||||||
- Setup needs (fixtures, mocks, integration dependencies).
|
|
||||||
- Estimated effort (`S/M/L`).
|
|
||||||
|
|
||||||
5. Close with validation plan
|
|
||||||
- State which gaps remain after proposals.
|
|
||||||
- Provide concrete verification command and acceptance threshold.
|
|
||||||
- List assumptions or blockers (environment, fixtures, flaky dependencies).
|
|
||||||
|
|
||||||
## Output template
|
|
||||||
|
|
||||||
### Coverage Snapshot
|
|
||||||
- total / branch coverage
|
|
||||||
- changed-file coverage
|
|
||||||
- top missing regions by size
|
|
||||||
|
|
||||||
### Top Gaps (ranked)
|
|
||||||
- `path:line-range` | risk score | why critical
|
|
||||||
|
|
||||||
### Test Proposals
|
|
||||||
- `path:line-range`
|
|
||||||
- Test name
|
|
||||||
- scenarios
|
|
||||||
- assertions
|
|
||||||
- effort
|
|
||||||
|
|
||||||
### Validation Plan
|
|
||||||
- command
|
|
||||||
- pass criteria
|
|
||||||
- remaining risk
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
interface:
|
|
||||||
display_name: "Test Coverage Improver"
|
|
||||||
short_description: "Find top uncovered risk areas and propose high-impact tests."
|
|
||||||
default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."
|
|
||||||
@@ -1,25 +0,0 @@
|
|||||||
# Coverage Gap Prioritization Guide
|
|
||||||
|
|
||||||
Use this rubric for each uncovered area.
|
|
||||||
|
|
||||||
Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
|
|
||||||
|
|
||||||
- Criticality:
|
|
||||||
- 5: authz/authn, data-loss, payment/consistency path
|
|
||||||
- 4: state mutation, cache invalidation, scheduling
|
|
||||||
- 3: error handling + fallbacks in user-visible flows
|
|
||||||
- 2: parsing/format conversion paths
|
|
||||||
- 1: logging-only or low-impact utilities
|
|
||||||
|
|
||||||
- CoverageDebt:
|
|
||||||
- 0: 0–5 uncovered lines
|
|
||||||
- 1: 6–20 uncovered lines
|
|
||||||
- 2: 21–40 uncovered lines
|
|
||||||
- 3: 41+ uncovered lines
|
|
||||||
|
|
||||||
- Volatility:
|
|
||||||
- 1: stable legacy code with few recent edits
|
|
||||||
- 2: changed in last 2 releases
|
|
||||||
- 3: touched in last 30 days or currently in active PR
|
|
||||||
|
|
||||||
Sort by score descending, then by business impact.
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
---
|
|
||||||
name: tier-debug
|
|
||||||
description: Debug ILM tiering / lifecycle transition issues — NoSuchVersion on tier GET, restore failures, xl.meta inspection, remote-tier versionId tracing. Use when investigating tiered/transitioned objects, warm backends, or transition metadata.
|
|
||||||
---
|
|
||||||
|
|
||||||
# Tier / ILM Debugging
|
|
||||||
|
|
||||||
Full playbook: [docs/operations/tier-ilm-debugging.md](../../../docs/operations/tier-ilm-debugging.md)
|
|
||||||
— read it before changing tier code.
|
|
||||||
|
|
||||||
Quick moves:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Inspect transition metadata on disk (one xl.meta per erasure shard disk)
|
|
||||||
cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/{bucket}/{object}/xl.meta"
|
|
||||||
|
|
||||||
# Trace what versionId is sent to the remote tier
|
|
||||||
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./target/debug/rustfs …
|
|
||||||
```
|
|
||||||
|
|
||||||
Interpretation:
|
|
||||||
|
|
||||||
- `transition_ver_id: <none>` → correct for an unversioned tier bucket; no
|
|
||||||
`versionId` must be sent on tier GET/DELETE.
|
|
||||||
- `transition_ver_id: 00000000-…` (nil) → corrupt legacy write-back; readers
|
|
||||||
must filter it out, never send it.
|
|
||||||
- Empty-string `transitioned-versionID` metadata under both
|
|
||||||
`x-rustfs-internal-*` and `x-minio-internal-*` keys → object went to an
|
|
||||||
unversioned tier bucket.
|
|
||||||
|
|
||||||
Code entry points: `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs`
|
|
||||||
(ILM actions), `crates/ecstore/src/services/tier/` (warm backends),
|
|
||||||
`crates/filemeta/src/filemeta/version.rs` (metadata read/write + regression
|
|
||||||
tests).
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
# RustFS Cargo configuration
|
|
||||||
|
|
||||||
# NOTE: `--cfg tokio_unstable` is deliberately NOT set here.
|
|
||||||
#
|
|
||||||
# It used to be a global `[build] rustflags` entry so that the (default-off)
|
|
||||||
# `dial9` telemetry feature could compile. That made every build depend on
|
|
||||||
# Tokio's unstable, non-semver API, and it broke silently whenever a caller
|
|
||||||
# exported their own RUSTFLAGS — an environment RUSTFLAGS replaces the value
|
|
||||||
# from this file rather than appending to it.
|
|
||||||
#
|
|
||||||
# The flag is now scoped to telemetry builds, which opt in explicitly:
|
|
||||||
#
|
|
||||||
# make build-profiling
|
|
||||||
# RUSTFLAGS="--cfg tokio_unstable" cargo build --features dial9
|
|
||||||
#
|
|
||||||
# `crates/obs/build.rs` fails the compile if the `dial9` feature is on without
|
|
||||||
# the flag, so the two can no longer drift apart unnoticed.
|
|
||||||
#
|
|
||||||
# For CPU profiling, add `-C force-frame-pointers=yes` to that RUSTFLAGS value.
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
../.agents/skills
|
|
||||||
@@ -1,2 +0,0 @@
|
|||||||
sha256-darwin=b4ae71aa894e5c7795ae3eb8116f1777a7601d0f5db3898be2e48faf3329bd9b
|
|
||||||
sha256-linux=433debd9d9defa832986269abdf0f1d131597b2d7a417ce930e17c1fd47d85ba
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
sha256=9b9bc336b43b70d0e06e0adb5455bf035bb18945d85d60936eb6fe4d48e0e680
|
|
||||||
@@ -1,2 +0,0 @@
|
|||||||
sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315
|
|
||||||
sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
sha256=655a3f3c1d042e694339d15caba7580518320322d1bac0f09450b37e6c09e2e7
|
|
||||||
@@ -1 +0,0 @@
|
|||||||
sha256=ec27cde6ce6400723c4b372bfbd2ac61709c744294e4810af765e8a808d8e31d
|
|
||||||
@@ -1,64 +0,0 @@
|
|||||||
## —— Development/Source builds using direct buildx commands ---------------------------------------
|
|
||||||
|
|
||||||
.PHONY: docker-dev
|
|
||||||
docker-dev: ## Build dev multi-arch image (cannot load locally)
|
|
||||||
@echo "🏗️ Building multi-architecture development Docker images with buildx..."
|
|
||||||
@echo "💡 This builds from source code and is intended for local development and testing"
|
|
||||||
@echo "⚠️ Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
|
|
||||||
$(DOCKER_CLI) buildx build \
|
|
||||||
--platform linux/amd64,linux/arm64 \
|
|
||||||
--file $(DOCKERFILE_SOURCE) \
|
|
||||||
--tag rustfs:source-latest \
|
|
||||||
--tag rustfs:dev-latest \
|
|
||||||
.
|
|
||||||
|
|
||||||
.PHONY: docker-dev-local
|
|
||||||
docker-dev-local: ## Build dev single-arch image (local load)
|
|
||||||
@echo "🏗️ Building single-architecture development Docker image for local use..."
|
|
||||||
@echo "💡 This builds from source code for the current platform and loads locally"
|
|
||||||
$(DOCKER_CLI) buildx build \
|
|
||||||
--file $(DOCKERFILE_SOURCE) \
|
|
||||||
--tag rustfs:source-latest \
|
|
||||||
--tag rustfs:dev-latest \
|
|
||||||
--load \
|
|
||||||
.
|
|
||||||
|
|
||||||
.PHONY: docker-dev-push
|
|
||||||
docker-dev-push: ## Build and push multi-arch development image # e.g (make docker-dev-push REGISTRY=xxx)
|
|
||||||
@if [ -z "$(REGISTRY)" ]; then \
|
|
||||||
echo "❌ Error: Please specify registry, example: make docker-dev-push REGISTRY=ghcr.io/username"; \
|
|
||||||
exit 1; \
|
|
||||||
fi
|
|
||||||
@echo "🚀 Building and pushing multi-architecture development Docker images..."
|
|
||||||
@echo "💡 Pushing to registry: $(REGISTRY)"
|
|
||||||
$(DOCKER_CLI) buildx build \
|
|
||||||
--platform linux/amd64,linux/arm64 \
|
|
||||||
--file $(DOCKERFILE_SOURCE) \
|
|
||||||
--tag $(REGISTRY)/rustfs:source-latest \
|
|
||||||
--tag $(REGISTRY)/rustfs:dev-latest \
|
|
||||||
--push \
|
|
||||||
.
|
|
||||||
|
|
||||||
.PHONY: dev-env-start
|
|
||||||
dev-env-start: ## Start development container environment
|
|
||||||
@echo "🚀 Starting development environment..."
|
|
||||||
$(DOCKER_CLI) buildx build \
|
|
||||||
--file $(DOCKERFILE_SOURCE) \
|
|
||||||
--tag rustfs:dev \
|
|
||||||
--load \
|
|
||||||
.
|
|
||||||
$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
|
|
||||||
$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
|
|
||||||
$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) \
|
|
||||||
-p 9010:9010 -p 9000:9000 \
|
|
||||||
-v $(shell pwd):/workspace \
|
|
||||||
-it rustfs:dev
|
|
||||||
|
|
||||||
.PHONY: dev-env-stop
|
|
||||||
dev-env-stop: ## Stop development container environment
|
|
||||||
@echo "🛑 Stopping development environment..."
|
|
||||||
$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
|
|
||||||
$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
|
|
||||||
|
|
||||||
.PHONY: dev-env-restart
|
|
||||||
dev-env-restart: dev-env-stop dev-env-start ## Restart development container environment
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
## —— Production builds using docker buildx (for CI/CD and production) -----------------------------
|
|
||||||
|
|
||||||
.PHONY: docker-buildx
|
|
||||||
docker-buildx: ## Build production multi-arch image (no push)
|
|
||||||
@echo "🏗️ Building multi-architecture production Docker images with buildx..."
|
|
||||||
./docker-buildx.sh
|
|
||||||
|
|
||||||
.PHONY: docker-buildx-push
|
|
||||||
docker-buildx-push: ## Build and push production multi-arch image
|
|
||||||
@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
|
|
||||||
./docker-buildx.sh --push
|
|
||||||
|
|
||||||
.PHONY: docker-buildx-version
|
|
||||||
docker-buildx-version: ## Build and version production multi-arch image # e.g (make docker-buildx-version VERSION=v1.0.0)
|
|
||||||
@if [ -z "$(VERSION)" ]; then \
|
|
||||||
echo "❌ Error: Please specify version, example: make docker-buildx-version VERSION=v1.0.0"; \
|
|
||||||
exit 1; \
|
|
||||||
fi
|
|
||||||
@echo "🏗️ Building multi-architecture production Docker images (version: $(VERSION))..."
|
|
||||||
./docker-buildx.sh --release $(VERSION)
|
|
||||||
|
|
||||||
.PHONY: docker-buildx-push-version
|
|
||||||
docker-buildx-push-version: ## Build and version and push production multi-arch image # e.g (make docker-buildx-push-version VERSION=v1.0.0)
|
|
||||||
@if [ -z "$(VERSION)" ]; then \
|
|
||||||
echo "❌ Error: Please specify version, example: make docker-buildx-push-version VERSION=v1.0.0"; \
|
|
||||||
exit 1; \
|
|
||||||
fi
|
|
||||||
@echo "🚀 Building and pushing multi-architecture production Docker images (version: $(VERSION))..."
|
|
||||||
./docker-buildx.sh --release $(VERSION) --push
|
|
||||||
|
|
||||||
.PHONY: docker-buildx-production-local
|
|
||||||
docker-buildx-production-local: ## Build production single-arch image locally
|
|
||||||
@echo "🏗️ Building single-architecture production Docker image locally..."
|
|
||||||
@echo "💡 Alternative to docker-buildx.sh for local testing"
|
|
||||||
$(DOCKER_CLI) buildx build \
|
|
||||||
--file $(DOCKERFILE_PRODUCTION) \
|
|
||||||
--tag rustfs:production-latest \
|
|
||||||
--tag rustfs:latest \
|
|
||||||
--load \
|
|
||||||
--build-arg RELEASE=latest \
|
|
||||||
.
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
## —— Single Architecture Docker Builds (Traditional) ----------------------------------------------
|
|
||||||
|
|
||||||
.PHONY: docker-build-production
|
|
||||||
docker-build-production: ## Build single-arch production image
|
|
||||||
@echo "🏗️ Building single-architecture production Docker image..."
|
|
||||||
@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
|
|
||||||
$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
|
|
||||||
|
|
||||||
.PHONY: docker-build-source
|
|
||||||
docker-build-source: ## Build single-arch source image
|
|
||||||
@echo "🏗️ Building single-architecture source Docker image..."
|
|
||||||
@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
|
|
||||||
DOCKER_BUILDKIT=1 $(DOCKER_CLI) build \
|
|
||||||
--build-arg BUILDKIT_INLINE_CACHE=1 \
|
|
||||||
-f $(DOCKERFILE_SOURCE) -t rustfs:source .
|
|
||||||
|
|
||||||
@@ -1,22 +0,0 @@
|
|||||||
## —— Docker-based build (alternative approach) ----------------------------------------------------
|
|
||||||
|
|
||||||
# Usage: make BUILD_OS=ubuntu22.04 build-docker
|
|
||||||
# Output: target/ubuntu22.04/release/rustfs
|
|
||||||
|
|
||||||
.PHONY: build-docker
|
|
||||||
build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
|
|
||||||
build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
|
|
||||||
build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
|
|
||||||
build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
|
|
||||||
@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
|
|
||||||
$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) --load .
|
|
||||||
$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
|
|
||||||
|
|
||||||
.PHONY: docker-inspect-multiarch
|
|
||||||
docker-inspect-multiarch: ## Check image architecture support
|
|
||||||
@if [ -z "$(IMAGE)" ]; then \
|
|
||||||
echo "❌ Error: Please specify image, example: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
|
|
||||||
exit 1; \
|
|
||||||
fi
|
|
||||||
@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
|
|
||||||
docker buildx imagetools inspect $(IMAGE)
|
|
||||||
@@ -1,74 +0,0 @@
|
|||||||
## —— Local Native Build using build-rustfs.sh script (Recommended) --------------------------------
|
|
||||||
|
|
||||||
.PHONY: build
|
|
||||||
build: ## Build RustFS binary (includes console by default)
|
|
||||||
@echo "🔨 Building RustFS using build-rustfs.sh script..."
|
|
||||||
./build-rustfs.sh
|
|
||||||
|
|
||||||
.PHONY: build-dev
|
|
||||||
build-dev: ## Build RustFS in Development mode
|
|
||||||
@echo "🔨 Building RustFS in development mode..."
|
|
||||||
./build-rustfs.sh --dev
|
|
||||||
|
|
||||||
.PHONY: build-musl
|
|
||||||
build-musl: ## Build x86_64 musl version
|
|
||||||
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
|
|
||||||
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
|
|
||||||
./build-rustfs.sh --platform x86_64-unknown-linux-musl
|
|
||||||
|
|
||||||
.PHONY: build-gnu
|
|
||||||
build-gnu: ## Build x86_64 GNU version
|
|
||||||
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
|
|
||||||
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
|
|
||||||
./build-rustfs.sh --platform x86_64-unknown-linux-gnu
|
|
||||||
|
|
||||||
.PHONY: build-musl-arm64
|
|
||||||
build-musl-arm64: ## Build aarch64 musl version
|
|
||||||
@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
|
|
||||||
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
|
|
||||||
./build-rustfs.sh --platform aarch64-unknown-linux-musl
|
|
||||||
|
|
||||||
.PHONY: build-gnu-arm64
|
|
||||||
build-gnu-arm64: ## Build aarch64 GNU version
|
|
||||||
@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
|
|
||||||
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
|
|
||||||
./build-rustfs.sh --platform aarch64-unknown-linux-gnu
|
|
||||||
|
|
||||||
|
|
||||||
## —— Profiling build (dial9 Tokio runtime telemetry) ------------------------------------------
|
|
||||||
|
|
||||||
# dial9 hooks Tokio's unstable runtime instrumentation, so it needs
|
|
||||||
# `--cfg tokio_unstable`. That flag is deliberately absent from
|
|
||||||
# .cargo/config.toml: it is not free, and release binaries do not carry it.
|
|
||||||
# Setting RUSTFLAGS here replaces (never appends to) the config-file value, and
|
|
||||||
# crates/obs/build.rs fails the build if the feature and the flag disagree.
|
|
||||||
#
|
|
||||||
# There are no task-dump or S3-upload features — see the notes in
|
|
||||||
# crates/obs/Cargo.toml for why.
|
|
||||||
DIAL9_FEATURES ?= dial9
|
|
||||||
DIAL9_RUSTFLAGS ?= --cfg tokio_unstable
|
|
||||||
|
|
||||||
.PHONY: build-profiling
|
|
||||||
build-profiling: ## Build RustFS with dial9 Tokio runtime telemetry (diagnostic builds only)
|
|
||||||
@echo "🔬 Building RustFS with dial9 telemetry (features: $(DIAL9_FEATURES))..."
|
|
||||||
@echo "⚠️ Diagnostic build: telemetry writes trace segments to disk continuously."
|
|
||||||
RUSTFLAGS="$(DIAL9_RUSTFLAGS)" cargo build --release --bin rustfs --features $(DIAL9_FEATURES)
|
|
||||||
|
|
||||||
.PHONY: build-cross-all
|
|
||||||
build-cross-all: core-deps ## Build binaries for all architectures
|
|
||||||
@echo "🔧 Building all target architectures..."
|
|
||||||
@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
|
|
||||||
@echo "🔨 Generating protobuf code..."
|
|
||||||
cargo run --bin gproto || true
|
|
||||||
|
|
||||||
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
|
|
||||||
./build-rustfs.sh --platform x86_64-unknown-linux-musl
|
|
||||||
|
|
||||||
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
|
|
||||||
./build-rustfs.sh --platform x86_64-unknown-linux-gnu
|
|
||||||
|
|
||||||
@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
|
|
||||||
./build-rustfs.sh --platform aarch64-unknown-linux-musl
|
|
||||||
|
|
||||||
@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
|
|
||||||
./build-rustfs.sh --platform aarch64-unknown-linux-gnu
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
## —— Check and Inform Dependencies ----------------------------------------------------------------
|
|
||||||
|
|
||||||
# Fatal check
|
|
||||||
# Checks all required dependencies and exits with error if not found
|
|
||||||
# (e.g., cargo, rustfmt)
|
|
||||||
check-%:
|
|
||||||
@command -v $* >/dev/null 2>&1 || { \
|
|
||||||
echo >&2 "❌ '$*' is not installed."; \
|
|
||||||
exit 1; \
|
|
||||||
}
|
|
||||||
|
|
||||||
# Warning-only check
|
|
||||||
# Checks for optional dependencies and issues a warning if not found
|
|
||||||
warn-%:
|
|
||||||
@command -v $* >/dev/null 2>&1 || { \
|
|
||||||
echo >&2 "⚠️ '$*' is not installed."; \
|
|
||||||
}
|
|
||||||
|
|
||||||
# For checking dependencies use check-<dep-name> or warn-<dep-name>
|
|
||||||
#
|
|
||||||
# NOTE: cargo-nextest is a HARD dependency of `make test`, gated inside the
|
|
||||||
# test recipe itself (with a RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 escape hatch)
|
|
||||||
# rather than a warn-only prerequisite here — see .config/make/tests.mak.
|
|
||||||
.PHONY: core-deps fmt-deps
|
|
||||||
core-deps: check-cargo ## Check core dependencies
|
|
||||||
fmt-deps: check-rustfmt ## Check lint and formatting dependencies
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
## —— Coverage --------------------------------------------------------------------------------------
|
|
||||||
|
|
||||||
# Local equivalent of the weekly coverage workflow (.github/workflows/coverage.yml,
|
|
||||||
# backlog#1153 infra-5): same measurement scope (--workspace --exclude e2e_test,
|
|
||||||
# nextest `ci` profile) and the same per-crate table. Slow — the instrumented
|
|
||||||
# build cannot reuse your normal target cache and then runs the whole suite.
|
|
||||||
# Doctests are not measured (needs nightly). Outputs land in target/llvm-cov/.
|
|
||||||
.PHONY: coverage
|
|
||||||
coverage: core-deps ## Workspace line coverage (cargo-llvm-cov + nextest; slow, writes target/llvm-cov/)
|
|
||||||
@if ! command -v cargo-llvm-cov >/dev/null 2>&1; then \
|
|
||||||
echo >&2 "❌ cargo-llvm-cov is required for 'make coverage' but was not found."; \
|
|
||||||
echo >&2 " Install it with:"; \
|
|
||||||
echo >&2 " cargo install cargo-llvm-cov --locked"; \
|
|
||||||
echo >&2 " rustup component add llvm-tools-preview"; \
|
|
||||||
exit 1; \
|
|
||||||
fi
|
|
||||||
@if ! command -v cargo-nextest >/dev/null 2>&1; then \
|
|
||||||
echo >&2 "❌ cargo-nextest is required for 'make coverage' (see 'make test')."; \
|
|
||||||
echo >&2 " Install it with: cargo install cargo-nextest --locked"; \
|
|
||||||
exit 1; \
|
|
||||||
fi
|
|
||||||
NEXTEST_PROFILE=ci cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
|
|
||||||
@mkdir -p target/llvm-cov
|
|
||||||
cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
|
|
||||||
cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
|
|
||||||
python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json
|
|
||||||
@@ -1,6 +0,0 @@
|
|||||||
## —— Deploy using dev_deploy.sh script ------------------------------------------------------------
|
|
||||||
|
|
||||||
.PHONY: deploy-dev
|
|
||||||
deploy-dev: build-musl ## Deploy to dev server
|
|
||||||
@echo "🚀 Deploying to dev server: $${IP}"
|
|
||||||
./scripts/dev_deploy.sh $${IP}
|
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
## —— Help, Help Build and Help Docker -------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
.PHONY: help
|
|
||||||
help: ## Shows This Help Menu
|
|
||||||
echo -e "$$HEADER"
|
|
||||||
grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} /^## / {printf "\n${green}%s${reset}\n", $$0; next} {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}'
|
|
||||||
|
|
||||||
.PHONY: help-build
|
|
||||||
help-build: ## Shows RustFS build help
|
|
||||||
@echo ""
|
|
||||||
@echo "💡 build-rustfs.sh script provides more options, smart detection and binary verification"
|
|
||||||
@echo ""
|
|
||||||
@echo "🔧 Direct usage of build-rustfs.sh script:"
|
|
||||||
@echo ""
|
|
||||||
@echo " ./build-rustfs.sh --help # View script help"
|
|
||||||
@echo " ./build-rustfs.sh --no-console # Build without console resources"
|
|
||||||
@echo " ./build-rustfs.sh --force-console-update # Force update console resources"
|
|
||||||
@echo " ./build-rustfs.sh --dev # Development mode build"
|
|
||||||
@echo " ./build-rustfs.sh --sign # Sign binary files"
|
|
||||||
@echo " ./build-rustfs.sh --platform x86_64-unknown-linux-gnu # Specify target platform"
|
|
||||||
@echo " ./build-rustfs.sh --skip-verification # Skip binary verification"
|
|
||||||
@echo ""
|
|
||||||
|
|
||||||
.PHONY: help-docker
|
|
||||||
help-docker: ## Shows docker environment and suggestion help
|
|
||||||
@echo ""
|
|
||||||
@echo "📋 Environment Variables:"
|
|
||||||
@echo " REGISTRY Image registry address (required for push)"
|
|
||||||
@echo " DOCKERHUB_USERNAME Docker Hub username"
|
|
||||||
@echo " DOCKERHUB_TOKEN Docker Hub access token"
|
|
||||||
@echo " GITHUB_TOKEN GitHub access token"
|
|
||||||
@echo ""
|
|
||||||
@echo "💡 Suggestions:"
|
|
||||||
@echo " Production use: Use docker-buildx* commands (based on precompiled binaries)"
|
|
||||||
@echo " Local development: Use docker-dev* commands (build from source)"
|
|
||||||
@echo " Development environment: Use dev-env-* commands to manage dev containers"
|
|
||||||
@echo ""
|
|
||||||
@@ -1,91 +0,0 @@
|
|||||||
## —— Code quality and Formatting ------------------------------------------------------------------
|
|
||||||
|
|
||||||
.NOTPARALLEL: fix
|
|
||||||
|
|
||||||
.PHONY: fmt
|
|
||||||
fmt: core-deps fmt-deps ## Format code
|
|
||||||
@echo "🔧 Formatting code..."
|
|
||||||
cargo fmt --all
|
|
||||||
|
|
||||||
.PHONY: fmt-check
|
|
||||||
fmt-check: core-deps fmt-deps ## Check code formatting
|
|
||||||
@echo "📝 Checking code formatting..."
|
|
||||||
cargo fmt --all --check
|
|
||||||
|
|
||||||
.PHONY: clippy-check
|
|
||||||
clippy-check: core-deps ## Run clippy checks
|
|
||||||
@echo "🔍 Running clippy checks..."
|
|
||||||
cargo clippy --all-targets -- -D warnings
|
|
||||||
|
|
||||||
.PHONY: clippy-fix
|
|
||||||
clippy-fix: core-deps ## Apply clippy fixes
|
|
||||||
@echo "🔧 Applying clippy fixes..."
|
|
||||||
cargo clippy --fix --allow-dirty
|
|
||||||
|
|
||||||
.PHONY: fix
|
|
||||||
fix: fmt clippy-fix ## Format code and apply clippy fixes
|
|
||||||
|
|
||||||
.PHONY: quick-check
|
|
||||||
quick-check: core-deps ## Run fast workspace compilation check
|
|
||||||
@echo "🔨 Running fast compilation check..."
|
|
||||||
cargo check --workspace --exclude e2e_test
|
|
||||||
|
|
||||||
.PHONY: unsafe-code-check
|
|
||||||
unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
|
|
||||||
@echo "🔒 Checking unsafe_code allowances..."
|
|
||||||
./scripts/check_unsafe_code_allowances.sh
|
|
||||||
|
|
||||||
.PHONY: architecture-migration-check
|
|
||||||
architecture-migration-check: ## Check architecture migration guardrails
|
|
||||||
@echo "🏗️ Checking architecture migration guardrails..."
|
|
||||||
./scripts/check_architecture_migration_rules.sh
|
|
||||||
|
|
||||||
.PHONY: logging-guardrails-check
|
|
||||||
logging-guardrails-check: ## Check logging guardrails for redaction and noise regressions
|
|
||||||
@echo "🪵 Checking logging guardrails..."
|
|
||||||
./scripts/check_logging_guardrails.sh
|
|
||||||
|
|
||||||
.PHONY: tokio-io-uring-check
|
|
||||||
tokio-io-uring-check: ## Check tokio io-uring runtime feature stays removed
|
|
||||||
@echo "🚫 Checking tokio io-uring feature guard..."
|
|
||||||
./scripts/check_no_tokio_io_uring.sh
|
|
||||||
|
|
||||||
.PHONY: extension-schema-check
|
|
||||||
extension-schema-check: ## Check extension-schema stays a lightweight contract crate
|
|
||||||
@echo "🧩 Checking extension schema boundaries..."
|
|
||||||
./scripts/check_extension_schema_boundaries.sh
|
|
||||||
|
|
||||||
.PHONY: body-cache-whitelist-check
|
|
||||||
body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fail-closed allow-list
|
|
||||||
@echo "🧱 Checking body-cache whitelist guard..."
|
|
||||||
./scripts/check_body_cache_whitelist.sh
|
|
||||||
|
|
||||||
.PHONY: s3s-footprint-check
|
|
||||||
s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
|
|
||||||
@echo "📦 Checking s3s footprint ratchet..."
|
|
||||||
./scripts/check_s3s_footprint.sh
|
|
||||||
|
|
||||||
.PHONY: fips-wording-check
|
|
||||||
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
|
|
||||||
@echo "📣 Checking cryptographic capability wording guard..."
|
|
||||||
./scripts/check_fips_wording.sh
|
|
||||||
|
|
||||||
.PHONY: embedded-secrets-check
|
|
||||||
embedded-secrets-check: ## Check no private key material or credential literal is committed
|
|
||||||
@echo "🔑 Checking embedded secret material guard..."
|
|
||||||
./scripts/check_embedded_secrets.sh
|
|
||||||
|
|
||||||
.PHONY: test-wiring-check
|
|
||||||
test-wiring-check: ## Check tests stay registered and selected by their intended runners
|
|
||||||
@echo "🧪 Checking test wiring..."
|
|
||||||
python3 ./scripts/check_test_wiring.py
|
|
||||||
|
|
||||||
.PHONY: log-analyzer-rules-check
|
|
||||||
log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
|
|
||||||
@echo "🩺 Checking log-analyzer rule anchors..."
|
|
||||||
./scripts/check_log_analyzer_rules.sh
|
|
||||||
|
|
||||||
.PHONY: compilation-check
|
|
||||||
compilation-check: core-deps ## Run compilation check
|
|
||||||
@echo "🔨 Running compilation check..."
|
|
||||||
cargo check --all-targets
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
## —— Pre Commit Checks ----------------------------------------------------------------------------
|
|
||||||
|
|
||||||
.NOTPARALLEL: pre-commit pre-pr dev-check
|
|
||||||
|
|
||||||
.PHONY: setup-hooks
|
|
||||||
setup-hooks: ## Set up git hooks
|
|
||||||
@echo "🔧 Setting up git hooks..."
|
|
||||||
chmod +x .git/hooks/pre-commit
|
|
||||||
@echo "✅ Git hooks setup complete!"
|
|
||||||
|
|
||||||
.PHONY: doc-paths-check
|
|
||||||
doc-paths-check: ## Check that instruction/architecture docs reference existing file paths
|
|
||||||
@echo "📄 Checking doc path references..."
|
|
||||||
./scripts/check_doc_paths.sh
|
|
||||||
|
|
||||||
.PHONY: planning-docs-check
|
|
||||||
planning-docs-check: ## Check that no planning-type documents are committed
|
|
||||||
@echo "📄 Checking for committed planning docs..."
|
|
||||||
./scripts/check_no_planning_docs.sh
|
|
||||||
|
|
||||||
.PHONY: pre-commit
|
|
||||||
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
|
|
||||||
@echo "✅ All pre-commit checks passed!"
|
|
||||||
|
|
||||||
.PHONY: pre-pr
|
|
||||||
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
|
|
||||||
@echo "✅ All pre-PR checks passed!"
|
|
||||||
|
|
||||||
.PHONY: dev-check
|
|
||||||
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
|
|
||||||
@echo "✅ Fast development checks passed!"
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
## —— Tests and e2e test ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
TEST_THREADS ?= 1
|
|
||||||
|
|
||||||
# cargo-nextest is a HARD dependency of `make test`.
|
|
||||||
#
|
|
||||||
# nextest changes test semantics vs plain `cargo test`: it runs every test in
|
|
||||||
# its own process (so serial_test's #[serial] mutex does not serialize across
|
|
||||||
# tests) and it is the only runner that honours .config/nextest.toml
|
|
||||||
# [test-groups] (e.g. the ecstore-serial-flaky serialization guard). CI runs
|
|
||||||
# nextest (.github/actions/setup/action.yml installs it), so a silent fallback
|
|
||||||
# to `cargo test` would run with different serialization behaviour than CI and
|
|
||||||
# mask (or invent) flakes.
|
|
||||||
#
|
|
||||||
# Installing cargo-nextest:
|
|
||||||
# cargo install cargo-nextest --locked # from source
|
|
||||||
# # or a prebuilt binary (faster) via the taiki-e installer / get.nexte.st:
|
|
||||||
# # https://nexte.st/docs/installation/
|
|
||||||
#
|
|
||||||
# Escape hatch: set RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to run the plain
|
|
||||||
# `cargo test` fallback anyway. Results are NOT authoritative — semantics
|
|
||||||
# differ from CI and .config/nextest.toml test-groups will NOT apply.
|
|
||||||
|
|
||||||
.PHONY: script-tests
|
|
||||||
script-tests: ## Run shell script tests
|
|
||||||
@echo "Running script tests..."
|
|
||||||
./scripts/test_build_rustfs_options.sh
|
|
||||||
./scripts/test_docker_runtime_timezone.sh
|
|
||||||
./scripts/test_entrypoint_credentials.sh
|
|
||||||
./scripts/test_internode_grpc_ab_bench.sh
|
|
||||||
./scripts/test_object_batch_bench_enhanced.sh
|
|
||||||
./scripts/test_hotpath_warp_ab_gate.sh
|
|
||||||
./scripts/test_hotpath_warp_abba.sh
|
|
||||||
./scripts/test_exact_1mib_handoff_abba.sh
|
|
||||||
./scripts/test_pinned_paired_abba_bench.sh
|
|
||||||
./scripts/test_manual_transition_runbooks.sh
|
|
||||||
./scripts/check_embedded_secrets.sh --self-test
|
|
||||||
python3 ./scripts/check_test_wiring.py --self-test
|
|
||||||
python3 ./scripts/s3-tests/test_report_compat.py
|
|
||||||
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
|
|
||||||
python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
|
|
||||||
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
|
|
||||||
|
|
||||||
.PHONY: test
|
|
||||||
test: core-deps script-tests ## Run all tests (needs cargo-nextest; RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to override)
|
|
||||||
@echo "🧪 Running tests..."
|
|
||||||
@if command -v cargo-nextest >/dev/null 2>&1; then \
|
|
||||||
cargo nextest run --all --exclude e2e_test; \
|
|
||||||
elif [ "$${RUSTFS_ALLOW_CARGO_TEST_FALLBACK:-0}" = "1" ]; then \
|
|
||||||
echo >&2 "⚠️ ============================================================================"; \
|
|
||||||
echo >&2 "⚠️ cargo-nextest NOT found — running the 'cargo test' fallback (opt-in)."; \
|
|
||||||
echo >&2 "⚠️ TEST SEMANTICS DIFFER FROM CI; results are NOT authoritative:"; \
|
|
||||||
echo >&2 "⚠️ * nextest runs each test in its own process; 'cargo test' does not,"; \
|
|
||||||
echo >&2 "⚠️ so serial_test #[serial] serialization behaves differently."; \
|
|
||||||
echo >&2 "⚠️ * .config/nextest.toml [test-groups] will NOT apply (e.g. the"; \
|
|
||||||
echo >&2 "⚠️ ecstore-serial-flaky group), so load-sensitive tests may flake here"; \
|
|
||||||
echo >&2 "⚠️ but pass on CI (or vice versa)."; \
|
|
||||||
echo >&2 "⚠️ Install cargo-nextest and re-run before trusting these results."; \
|
|
||||||
echo >&2 "⚠️ ============================================================================"; \
|
|
||||||
cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
|
|
||||||
else \
|
|
||||||
echo >&2 "❌ cargo-nextest is required for 'make test' but was not found."; \
|
|
||||||
echo >&2 ""; \
|
|
||||||
echo >&2 " RustFS tests run under cargo-nextest (process-per-test isolation)."; \
|
|
||||||
echo >&2 " CI runs nextest and .config/nextest.toml [test-groups] only take effect"; \
|
|
||||||
echo >&2 " under nextest. Plain 'cargo test' has different serialization semantics"; \
|
|
||||||
echo >&2 " and is NOT a faithful substitute."; \
|
|
||||||
echo >&2 ""; \
|
|
||||||
echo >&2 " Install it with either:"; \
|
|
||||||
echo >&2 " cargo install cargo-nextest --locked"; \
|
|
||||||
echo >&2 " or a prebuilt binary (faster) — see https://nexte.st/docs/installation/"; \
|
|
||||||
echo >&2 ""; \
|
|
||||||
echo >&2 " To run the plain 'cargo test' fallback anyway (results NOT authoritative;"; \
|
|
||||||
echo >&2 " serialization semantics differ from CI), re-run with:"; \
|
|
||||||
echo >&2 " RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test"; \
|
|
||||||
exit 1; \
|
|
||||||
fi
|
|
||||||
cargo test --all --doc
|
|
||||||
|
|
||||||
.PHONY: e2e-server
|
|
||||||
e2e-server: ## Run e2e-server tests
|
|
||||||
sh $(shell pwd)/scripts/run.sh
|
|
||||||
|
|
||||||
.PHONY: probe-e2e
|
|
||||||
probe-e2e: ## Probe e2e tests
|
|
||||||
sh $(shell pwd)/scripts/probe.sh
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
# Committed floor for the number of tests selected by the migration-critical
|
|
||||||
# CI gate (see scripts/check_migration_gate_count.sh, backlog#1153 infra-12).
|
|
||||||
#
|
|
||||||
# The floor equals the exact count of rustfs-ecstore --lib tests matching the
|
|
||||||
# gate filter (name substrings: data_movement, rebalance, decommission,
|
|
||||||
# source_cleanup, delete_marker) at the time this file was last updated.
|
|
||||||
# CI fails if the selected count drops below this number, so renames or
|
|
||||||
# removals that thin the gate must update this file in the same PR.
|
|
||||||
# Adding tests does not require a bump, but bumping keeps the guard tight.
|
|
||||||
571
|
|
||||||
@@ -1,447 +0,0 @@
|
|||||||
# nextest configuration for RustFS.
|
|
||||||
#
|
|
||||||
# Serialize the ecstore tests that share the process-wide disk registry or
|
|
||||||
# exercise a multi-disk commit handoff across nextest process boundaries.
|
|
||||||
#
|
|
||||||
# * store::bucket::tests::bucket_delete_* share process/global state (disk
|
|
||||||
# registry, lock client) and race make_bucket into InsufficientWriteQuorum
|
|
||||||
# when run concurrently with other ecstore tests.
|
|
||||||
# * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
|
|
||||||
# uses the shared multipart fixture and a deterministic uploadId-lock
|
|
||||||
# handoff, so it must not overlap another process mutating that fixture.
|
|
||||||
# * bucket::metadata_sys::tests::concurrent_config_writes_from_separate_nodes_do_not_lose_writes
|
|
||||||
# uses the shared transaction lock and must not overlap other ecstore tests.
|
|
||||||
#
|
|
||||||
# serial_test's #[serial] attribute does NOT serialize these across runs:
|
|
||||||
# nextest executes each test in its own process, where the in-process
|
|
||||||
# serial_test mutex has no effect. A nextest test-group with max-threads = 1 is
|
|
||||||
# the mechanism that actually serializes across nextest's process boundary.
|
|
||||||
#
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Profiles
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# The `default` profile is what local `cargo nextest run` uses. It NEVER
|
|
||||||
# retries: a red test locally means a real failure to investigate, not noise to
|
|
||||||
# paper over. The `ci` profile (below) is the strict CI gate: global
|
|
||||||
# retries = 0 so a new race's first occurrence is never masked, plus a
|
|
||||||
# narrowly-scoped quarantine list (retries = 2) for tests with a tracked OPEN
|
|
||||||
# flake issue. Flake policy lives in docs/testing/README.md.
|
|
||||||
|
|
||||||
[test-groups]
|
|
||||||
ecstore-serial-flaky = { max-threads = 1 }
|
|
||||||
embedded-test-ports = { max-threads = 1 }
|
|
||||||
e2e-vault = { max-threads = 1 }
|
|
||||||
|
|
||||||
# Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
|
|
||||||
# server and manipulate its disk directories at runtime (crates/e2e_test:
|
|
||||||
# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13, and
|
|
||||||
# replacement_privileged_e2e_test when explicitly run as root on Linux). They
|
|
||||||
# are correct in isolation but resource-heavy; serialize them under nextest's
|
|
||||||
# process boundary (serial_test's #[serial] does not cross it) so several 4-disk
|
|
||||||
# servers never run at once. The e2e-full merge/main lane picks these up;
|
|
||||||
# they are deliberately NOT in the fast PR `e2e-smoke` filter.
|
|
||||||
e2e-reliability = { max-threads = 1 }
|
|
||||||
e2e-inline-boundaries = { max-threads = 1 }
|
|
||||||
e2e-cluster-nightly = { max-threads = 1 }
|
|
||||||
|
|
||||||
# --- default profile (local): serialize the flaky groups, never retry --------
|
|
||||||
[[profile.default.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
|
|
||||||
# each spawns a 4-disk hermetic erasure set and drives full staged-upload +
|
|
||||||
# commit + GET cycles — the same cross-disk-commit IO shape that made
|
|
||||||
# concurrent_resend load-sensitive. Preventive serialization only, no retries.
|
|
||||||
# The matching ci-profile override is after [profile.ci].
|
|
||||||
[[profile.default.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
# The production-handler relocation regression builds an isolated 8-disk,
|
|
||||||
# 2-pool store and commits a 72 MiB multipart object. Keep that cross-disk IO
|
|
||||||
# from overlapping the ecstore commit fixtures above.
|
|
||||||
[[profile.default.overrides]]
|
|
||||||
filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
# Embedded integration-test binaries discover an ephemeral port and release
|
|
||||||
# the probe listener before RustFS binds it. Serialize that cross-process
|
|
||||||
# TOCTOU window; retries would only hide real startup failures.
|
|
||||||
[[profile.default.overrides]]
|
|
||||||
filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
|
|
||||||
test-group = 'embedded-test-ports'
|
|
||||||
|
|
||||||
# Serialize the durable manual-transition checkpoint test across nextest's
|
|
||||||
# process boundary; it mutates bucket lifecycle metadata and is not quarantined.
|
|
||||||
[[profile.default.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
# Serialize the bucket-incarnation / lifecycle-fence tests. They drive
|
|
||||||
# init_bucket_metadata_sys and bucket_metadata_sys_of, i.e. process-global
|
|
||||||
# OnceLock state that serial_test's #[serial] cannot protect across nextest's
|
|
||||||
# process boundary, and they delete+recreate buckets — the same shape that
|
|
||||||
# raced into InsufficientWriteQuorum in backlog#937. Preventive only, no
|
|
||||||
# retries. The matching ci-profile override is after [profile.ci].
|
|
||||||
[[profile.default.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
# Serialize the default-off dst-dir fsync group-commit tests. They use
|
|
||||||
# process-global test hooks/registry to deterministically freeze fsync batches;
|
|
||||||
# no retries, just one at a time under nextest too.
|
|
||||||
[[profile.default.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
# Serialize the 4-disk reliability / degraded-read e2e tests (see the
|
|
||||||
# e2e-reliability test-group note above). The matching ci-profile override is at
|
|
||||||
# the end of the file, after [profile.ci] is declared.
|
|
||||||
[[profile.default.overrides]]
|
|
||||||
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
|
|
||||||
test-group = 'e2e-reliability'
|
|
||||||
|
|
||||||
[[profile.default.overrides]]
|
|
||||||
filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
|
|
||||||
test-group = 'e2e-inline-boundaries'
|
|
||||||
|
|
||||||
# Vault KMS tests share the fixed dev-server port 8200. serial_test's #[serial]
|
|
||||||
# does not cross nextest process boundaries, so keep these tests in one group.
|
|
||||||
[[profile.default.overrides]]
|
|
||||||
filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
|
|
||||||
test-group = 'e2e-vault'
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
[profile.ci]
|
|
||||||
# Strict: a new race must fail on its first occurrence, never be retried away.
|
|
||||||
retries = 0
|
|
||||||
# Report every failure in one run instead of bailing on the first.
|
|
||||||
fail-fast = false
|
|
||||||
|
|
||||||
[profile.ci.junit]
|
|
||||||
# Emitted to target/nextest/ci/junit.xml; uploaded as a CI artifact.
|
|
||||||
# Tests that pass only after a quarantine retry are marked `flaky` here — that
|
|
||||||
# marker is the observable signal the flake policy is built around.
|
|
||||||
path = "junit.xml"
|
|
||||||
|
|
||||||
# ===========================================================================
|
|
||||||
# QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
|
|
||||||
#
|
|
||||||
# RULES (enforced by review, see docs/testing/README.md):
|
|
||||||
# * Every entry MUST link exactly one OPEN issue tracking the flake.
|
|
||||||
# * An entry stays until the issue is fixed (test made robust) or the test is
|
|
||||||
# deleted — 30-day policy. No entry may exist without a live issue link.
|
|
||||||
#
|
|
||||||
# Each entry also re-declares the `ecstore-serial-flaky` test-group so the
|
|
||||||
# serialization holds under the ci profile (nextest evaluates a named
|
|
||||||
# profile's own overrides list, not the default profile's).
|
|
||||||
# ===========================================================================
|
|
||||||
|
|
||||||
# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
|
|
||||||
# make_bucket into InsufficientWriteQuorum via shared global state under load.
|
|
||||||
[[profile.ci.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
retries = 2
|
|
||||||
|
|
||||||
# Keep deterministic ECStore write handoffs isolated across nextest processes.
|
|
||||||
[[profile.ci.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
|
|
||||||
# on producer/consumer timing windows that stretch past the budget on loaded
|
|
||||||
# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
|
|
||||||
[[profile.ci.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
|
|
||||||
retries = 2
|
|
||||||
|
|
||||||
# Serialize the 4-disk reliability / degraded-read e2e tests under the ci
|
|
||||||
# profile too (see the e2e-reliability test-group note near the top). Not a
|
|
||||||
# quarantine: no retries, just single-threaded so several 4-disk servers never
|
|
||||||
# run concurrently when e2e-full runs the suite.
|
|
||||||
[[profile.ci.overrides]]
|
|
||||||
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
|
|
||||||
test-group = 'e2e-reliability'
|
|
||||||
|
|
||||||
# Serialize the multipart crash-consistency scenarios under the ci profile too
|
|
||||||
# (see the matching default-profile override near the top). Not a quarantine:
|
|
||||||
# no retries, just serialized 4-disk cross-disk-commit IO.
|
|
||||||
[[profile.ci.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
[[profile.ci.overrides]]
|
|
||||||
filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
# Match the default-profile embedded test isolation without quarantining or
|
|
||||||
# retrying failures in CI.
|
|
||||||
[[profile.ci.overrides]]
|
|
||||||
filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
|
|
||||||
test-group = 'embedded-test-ports'
|
|
||||||
|
|
||||||
# Serialize the durable manual-transition checkpoint test under the ci profile
|
|
||||||
# too. No retries: failures stay visible.
|
|
||||||
[[profile.ci.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
# Serialize the bucket-incarnation / lifecycle-fence tests under the ci profile
|
|
||||||
# too (see the matching default-profile override near the top). No retries.
|
|
||||||
[[profile.ci.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
[[profile.ci.overrides]]
|
|
||||||
filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
|
|
||||||
test-group = 'ecstore-serial-flaky'
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# PR smoke subset of the e2e_test crate (backlog#1149 ci-4). This profile is
|
|
||||||
# the single wiring mechanism for e2e tests in CI: other suites join by
|
|
||||||
# extending this filter (or a sibling profile), never by adding ad-hoc e2e
|
|
||||||
# jobs to ci.yml. Admission criteria (see crates/e2e_test/README.md): fast,
|
|
||||||
# single-node topology, no external dependencies (no awscurl / Vault / fixed
|
|
||||||
# ports / pre-started server), no #[ignore].
|
|
||||||
#
|
|
||||||
# Each e2e test spawns its own rustfs server on a random port with an isolated
|
|
||||||
# temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
|
|
||||||
#
|
|
||||||
# Replication failure harness (backlog#1147 repl-8): the first clause admits
|
|
||||||
# its four in-process fake-target self-tests. They bind random loopback ports,
|
|
||||||
# use no external service, and finish in under a second.
|
|
||||||
#
|
|
||||||
# Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
|
|
||||||
# FAST bucket-replication tests from replication_extension_test — the
|
|
||||||
# target-registration / replication-check / list / remove / delete admin paths
|
|
||||||
# that validate config synchronously and never wait for asynchronous
|
|
||||||
# replication convergence. Each spawns its own single-node rustfs server(s) on
|
|
||||||
# random ports (source, plus an independent single-node target for the pair
|
|
||||||
# checks — NOT a cluster), so the subset stays parallel-safe and single-digit
|
|
||||||
# seconds. The data-plane tests that poll for convergence and all
|
|
||||||
# `_real_dual_node` / `_real_single_node`
|
|
||||||
# site-replication tests run in the [profile.e2e-repl-nightly] lane below, NOT
|
|
||||||
# here. This allowlist is the single source of truth for the PR/nightly split:
|
|
||||||
# the nightly profile derives its set as "the replication module MINUS this
|
|
||||||
# allowlist", so any new replication test lands in nightly by default (never
|
|
||||||
# silently unrun) until it is explicitly blessed as fast here. Keep the two
|
|
||||||
# regexes byte-identical. The committed profile selection digests make changes
|
|
||||||
# visible in CI; current counts live in docs/testing/e2e-suite-inventory.md.
|
|
||||||
# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
|
|
||||||
# (#4724) because they set a loopback (127.0.0.1) replication target that the
|
|
||||||
# SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed —
|
|
||||||
# the guard now honours an off-by-default opt-in and this suite's source servers
|
|
||||||
# set it (RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) — so the allowlist below is
|
|
||||||
# restored.
|
|
||||||
#
|
|
||||||
# Security negative-auth subset (backlog#1151 sec-5): the three attacker-facing
|
|
||||||
# S3 auth-rejection suites join the first clause above by module name —
|
|
||||||
# presigned_negative (sec-2), negative_sigv4 (sec-1, header SigV4), and
|
|
||||||
# admin_auth (sec-4, admin gate + root-credential lifecycle). All three use
|
|
||||||
# RustFSTestEnvironment on a random port and are parallel-safe, so they meet the
|
|
||||||
# smoke admission criteria unchanged. This is the wiring step that makes those
|
|
||||||
# merged suites actually execute on every PR (they were dead until listed here).
|
|
||||||
# A rename that drops any of them out of this filter would silently thin the
|
|
||||||
# security gate with no CI signal, so scripts/check_security_smoke_count.sh owns
|
|
||||||
# a count-floor guard over exactly this subset (infra-12 mechanism, floor in
|
|
||||||
# .config/security-smoke-floor.txt), invoked from the e2e-tests job in ci.yml.
|
|
||||||
# NOT here by topology: the GHSA-3p3x FTPS/WebDAV constant-time e2e
|
|
||||||
# (protocols::test_protocol_core_suite) binds fixed ports and needs the
|
|
||||||
# ftps,webdav features, so it cannot join this random-port, default-feature
|
|
||||||
# profile; its GHSA-r5qv sibling is a unit test that already runs in the
|
|
||||||
# test-and-lint `--all --exclude e2e_test` pass. See
|
|
||||||
# docs/testing/security-regressions.md for the full CI-execution map.
|
|
||||||
#
|
|
||||||
# ILM tiering main path (backlog#1148 ilm-7): the `reliant::tiering::` clause
|
|
||||||
# admits the hermetic transition e2e. Like the fast replication pair checks it
|
|
||||||
# spawns a second independent single-node server (the cold RustFS tier), not a
|
|
||||||
# cluster, so it keeps the lane's parallel-safe / no-external-dependency
|
|
||||||
# properties. The RustFS warm backend has no loopback guard (that guard is
|
|
||||||
# replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
|
|
||||||
#
|
|
||||||
# Disk compression (backlog#1848): the `compression` module joins the smoke
|
|
||||||
# lane so the multipart disk-compression roundtrips (restored after
|
|
||||||
# rustfs/rustfs#5169 disabled them) have PR-lane signal, not just merge-gate.
|
|
||||||
# Single-node servers on random ports with isolated temp dirs — meets the
|
|
||||||
# admission criteria unchanged.
|
|
||||||
[profile.e2e-smoke]
|
|
||||||
default-filter = """
|
|
||||||
package(e2e_test) & (
|
|
||||||
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
|
|
||||||
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
|
|
||||||
| test(/^reliant::lifecycle::/)
|
|
||||||
| test(/^reliant::tiering::/)
|
|
||||||
)
|
|
||||||
"""
|
|
||||||
fail-fast = false
|
|
||||||
|
|
||||||
[profile.e2e-smoke.junit]
|
|
||||||
path = "junit.xml"
|
|
||||||
|
|
||||||
# The pagination boundary cases can stall when a server/listing regression
|
|
||||||
# prevents the continuation request from completing. Keep the timeout scoped
|
|
||||||
# to those known failure modes so legitimate lifecycle/tiering waits retain
|
|
||||||
# their test-level timing budget.
|
|
||||||
[[profile.e2e-smoke.overrides]]
|
|
||||||
filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
|
|
||||||
slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# backlog#1147 repl-1 (deps: ci-4). Runs the SLOW / cross-process replication
|
|
||||||
# tests that are unfit for the per-PR e2e-smoke gate:
|
|
||||||
#
|
|
||||||
# * 2 remote-target TLS validation tests.
|
|
||||||
# * 15 bucket-replication data-plane/helper tests — they PUT/delete objects
|
|
||||||
# and poll until source and target converge; two replicate over HTTPS,
|
|
||||||
# six pin SSE replication contracts (managed SSE-S3/SSE-KMS re-encrypt on
|
|
||||||
# the target incl. multipart and the resync path, SSE-C and
|
|
||||||
# target-without-KMS stay fail-closed), and one guards event/history
|
|
||||||
# observers.
|
|
||||||
# * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
|
|
||||||
# servers and drives the cross-process site-replication control plane.
|
|
||||||
# * 1 `_real_three_node` site-replication test.
|
|
||||||
# * 1 `_real_single_node` service-account round-trip test.
|
|
||||||
#
|
|
||||||
# The set is defined as "everything in replication_extension_test that is NOT
|
|
||||||
# in the e2e-smoke PR allowlist above" (the negated clause is byte-identical to
|
|
||||||
# the allowlist), so a newly added replication test automatically runs here
|
|
||||||
# until it is explicitly promoted to the fast PR subset — no replication test
|
|
||||||
# is ever silently left out of CI.
|
|
||||||
#
|
|
||||||
# #[serial] does NOT serialize under nextest (process-per-test; see the file
|
|
||||||
# header). These tests need no cross-test serialization: each spawns its own
|
|
||||||
# server(s) on random ports with isolated temp dirs, so they are parallel-safe
|
|
||||||
# by construction — the same property the e2e-smoke subset relies on. If load
|
|
||||||
# on the runner surfaces a real flake, quarantine the specific test with an
|
|
||||||
# OPEN issue link (ci-10 / backlog#937 policy), never blanket-retry or exclude.
|
|
||||||
#
|
|
||||||
# Wired by .github/workflows/e2e-replication-nightly.yml (schedule +
|
|
||||||
# workflow_dispatch), which builds the rustfs binary once, installs awscurl so
|
|
||||||
# the STS dual-node test actually exercises its path (it skips gracefully with
|
|
||||||
# a visible log line when awscurl is absent), and routes scheduled failures
|
|
||||||
# through .github/actions/schedule-failure-issue (ci-8). Explicit division of
|
|
||||||
# labor with e2e-full: these tests run only in the consolidated nightly
|
|
||||||
# workflow, not in the merge/main lane.
|
|
||||||
[profile.e2e-repl-nightly]
|
|
||||||
default-filter = """
|
|
||||||
package(e2e_test)
|
|
||||||
& test(/^replication_extension_test::/)
|
|
||||||
& !test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
|
|
||||||
"""
|
|
||||||
fail-fast = false
|
|
||||||
|
|
||||||
[profile.e2e-repl-nightly.junit]
|
|
||||||
# Emitted to target/nextest/e2e-repl-nightly/junit.xml; uploaded by the nightly
|
|
||||||
# workflow as the failure-triage artifact.
|
|
||||||
path = "junit.xml"
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# e2e-nightly profile — destructive multi-process cluster fault domains
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# These seven modules are deliberately outside e2e-full's merge budget. Each
|
|
||||||
# starts a real multi-process or multi-disk topology and exercises node/disk
|
|
||||||
# loss, quorum, cleanup, notification fan-in, or admin-timeout behavior. The
|
|
||||||
# consolidated nightly workflow runs them serially to avoid resource
|
|
||||||
# starvation; failures are never retried.
|
|
||||||
[profile.e2e-nightly]
|
|
||||||
default-filter = """
|
|
||||||
package(e2e_test)
|
|
||||||
& test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
|
|
||||||
"""
|
|
||||||
fail-fast = false
|
|
||||||
|
|
||||||
[profile.e2e-nightly.junit]
|
|
||||||
path = "junit.xml"
|
|
||||||
|
|
||||||
[[profile.e2e-nightly.overrides]]
|
|
||||||
filter = 'package(e2e_test)'
|
|
||||||
test-group = 'e2e-cluster-nightly'
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# e2e-protocols profile — serial protocol lane
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# The suite owns fixed ports, so the nightly workflow runs this exact profile
|
|
||||||
# with one nextest worker.
|
|
||||||
[profile.e2e-protocols]
|
|
||||||
default-filter = 'package(e2e_test) & test(/^protocols::/)'
|
|
||||||
fail-fast = false
|
|
||||||
|
|
||||||
[profile.e2e-protocols.junit]
|
|
||||||
path = "junit.xml"
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# The merge gate (ci.yml `e2e-full` job: push main + merge_group +
|
|
||||||
# workflow_dispatch). Runs the user-visible KMS, object-lock, multipart-auth,
|
|
||||||
# quota, checksum, encryption,
|
|
||||||
# security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
|
|
||||||
# skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
|
|
||||||
# --profile e2e-full` (see docs/testing/e2e-suite-inventory.md).
|
|
||||||
#
|
|
||||||
# The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
|
|
||||||
# * protocols:: — FTPS/SFTP/WebDAV, run from the dedicated protocol profile
|
|
||||||
# with one worker because the suite owns fixed ports.
|
|
||||||
# * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
|
|
||||||
# (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
|
|
||||||
# namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
|
|
||||||
# object_lambda) — too heavy for the merge budget; they run in the
|
|
||||||
# e2e-nightly serial cluster-fault lane.
|
|
||||||
# * replication_extension_test — repl-1 already splits it into the PR
|
|
||||||
# `e2e-smoke` (20 fast) and `e2e-repl-nightly` (55 slow) lanes and reserves
|
|
||||||
# it for those, so e2e-full does not double-run it.
|
|
||||||
# * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
|
|
||||||
# manual-localhost:9000 reliant/policy tests are ci-13's migration.
|
|
||||||
#
|
|
||||||
# Each e2e test spawns its own single-node rustfs server on a random port with
|
|
||||||
# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
|
|
||||||
# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
|
|
||||||
# 4-disk reliability / degraded-read fault-injection tests and the fixed-port
|
|
||||||
# Vault tests, both serialized below.
|
|
||||||
# KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
|
|
||||||
# 341 ran / 32 failed on the suites' first automated run ever). Deterministic
|
|
||||||
# product failures cannot be quarantined away with retries, so each family is
|
|
||||||
# excluded here with its tracking issue, under the same discipline as the
|
|
||||||
# ci-profile quarantine (docs/testing/README.md): every entry MUST cite one
|
|
||||||
# OPEN issue, and the fixing PR MUST delete the exclusion. The passing
|
|
||||||
# negative-path siblings of each family stay in as regression guards.
|
|
||||||
# * rustfs#4843 — over-limit archive entry paths hard-reject the whole
|
|
||||||
# archive even under ignore-errors semantics.
|
|
||||||
[profile.e2e-full]
|
|
||||||
default-filter = """
|
|
||||||
package(e2e_test)
|
|
||||||
& !test(/^protocols::/)
|
|
||||||
& !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
|
|
||||||
& !test(/^replication_extension_test::/)
|
|
||||||
& !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
|
|
||||||
& !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
|
|
||||||
"""
|
|
||||||
fail-fast = false
|
|
||||||
|
|
||||||
[profile.e2e-full.junit]
|
|
||||||
# Emitted to target/nextest/e2e-full/junit.xml; uploaded by the e2e-full job.
|
|
||||||
path = "junit.xml"
|
|
||||||
|
|
||||||
# Serialize the 4-disk reliability / degraded-read e2e tests under e2e-full too
|
|
||||||
# (see the e2e-reliability test-group note near the top of this file). Not a
|
|
||||||
# quarantine: no retries, just single-threaded so several 4-disk servers never
|
|
||||||
# run concurrently.
|
|
||||||
[[profile.e2e-full.overrides]]
|
|
||||||
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
|
|
||||||
test-group = 'e2e-reliability'
|
|
||||||
|
|
||||||
[[profile.e2e-full.overrides]]
|
|
||||||
filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
|
|
||||||
test-group = 'e2e-inline-boundaries'
|
|
||||||
|
|
||||||
[[profile.e2e-full.overrides]]
|
|
||||||
filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
|
|
||||||
test-group = 'e2e-vault'
|
|
||||||
@@ -1,12 +0,0 @@
|
|||||||
# Committed floor for the number of security negative-auth tests selected by the
|
|
||||||
# e2e-smoke PR profile (see scripts/check_security_smoke_count.sh, backlog#1151
|
|
||||||
# sec-5).
|
|
||||||
#
|
|
||||||
# The floor equals the exact count of e2e_test cases whose name starts with a
|
|
||||||
# security module prefix (negative_sigv4_test, presigned_negative_test,
|
|
||||||
# admin_auth_test) that the [profile.e2e-smoke] default-filter in
|
|
||||||
# .config/nextest.toml selects, at the time this file was last updated. CI fails
|
|
||||||
# if the selected count drops below this number, so a rename or removal that
|
|
||||||
# thins the security smoke gate must update this file in the same PR.
|
|
||||||
# Adding tests does not require a bump, but bumping keeps the guard tight.
|
|
||||||
16
|
|
||||||
+229
-99
@@ -1,131 +1,261 @@
|
|||||||
# RustFS Docker Infrastructure
|
# RustFS Docker Images
|
||||||
|
|
||||||
This directory contains the complete Docker infrastructure for building, deploying, and monitoring RustFS. It provides ready-to-use configurations for development, testing, and production-grade observability.
|
This directory contains Docker configuration files and supporting infrastructure for building and running RustFS container images.
|
||||||
|
|
||||||
## 📂 Directory Structure
|
## 📁 Directory Structure
|
||||||
|
|
||||||
| Directory | Description | Status |
|
```
|
||||||
| :--- | :--- | :--- |
|
rustfs/
|
||||||
| **[`observability/`](observability/README.md)** | **[RECOMMENDED]** Full-stack observability (Prometheus, Grafana, Tempo, Loki). | ✅ Production-Ready |
|
├── Dockerfile # Production image (Alpine + pre-built binaries)
|
||||||
| **[`compose/`](compose/README.md)** | Specialized setups (e.g., 4-node distributed cluster testing). | ⚠️ Testing Only |
|
├── Dockerfile.source # Development image (Debian + source build)
|
||||||
| **[`mqtt/`](mqtt/README.md)** | EMQX Broker configuration for MQTT integration testing. | 🧪 Development |
|
├── docker-buildx.sh # Multi-architecture build script
|
||||||
| **[`openobserve-otel/`](openobserve-otel/README.md)** | Alternative lightweight observability stack using OpenObserve. | 🔄 Alternative |
|
├── Makefile # Build automation with simplified commands
|
||||||
|
└── .docker/ # Supporting infrastructure
|
||||||
---
|
├── observability/ # Monitoring and observability configs
|
||||||
|
├── compose/ # Docker Compose configurations
|
||||||
## 📄 Root Directory Files
|
├── mqtt/ # MQTT broker configs
|
||||||
|
└── openobserve-otel/ # OpenObserve + OpenTelemetry configs
|
||||||
The following files in the project root are essential for Docker operations:
|
|
||||||
|
|
||||||
### Build Scripts & Dockerfiles
|
|
||||||
|
|
||||||
| File | Description | Usage |
|
|
||||||
| :--- | :--- | :--- |
|
|
||||||
| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
|
|
||||||
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
|
|
||||||
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
|
|
||||||
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
|
|
||||||
|
|
||||||
### Docker Compose Configurations
|
|
||||||
|
|
||||||
| File | Description | Usage |
|
|
||||||
| :--- | :--- | :--- |
|
|
||||||
| **`docker-compose.yml`** | **Main Development Setup**<br>Comprehensive setup with profiles for development, observability, and proxying. | `docker compose up -d`<br>`docker compose --profile observability up -d` |
|
|
||||||
| **`docker-compose-simple.yml`** | **Quick Start Setup**<br>Minimal configuration running a single RustFS instance with 4 volumes. Perfect for first-time users. | `docker compose -f docker-compose-simple.yml up -d` |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🌟 Observability Stack (Recommended)
|
|
||||||
|
|
||||||
Located in: [`.docker/observability/`](observability/README.md)
|
|
||||||
|
|
||||||
We provide a comprehensive, industry-standard observability stack designed for deep insights into RustFS performance. This is the recommended setup for both development and production monitoring.
|
|
||||||
|
|
||||||
### Components
|
|
||||||
- **Metrics**: Prometheus (Collection) + Grafana (Visualization)
|
|
||||||
- **Traces**: Tempo (Storage) + Jaeger (UI)
|
|
||||||
- **Logs**: Loki
|
|
||||||
- **Ingestion**: OpenTelemetry Collector
|
|
||||||
|
|
||||||
### Key Features
|
|
||||||
- **Full Persistence**: All metrics, logs, and traces are saved to Docker volumes, ensuring no data loss on restarts.
|
|
||||||
- **Correlation**: Seamlessly jump between Logs, Traces, and Metrics in Grafana.
|
|
||||||
- **High Performance**: Optimized configurations for batching, compression, and memory management.
|
|
||||||
|
|
||||||
### Quick Start
|
|
||||||
```bash
|
|
||||||
cd .docker/observability
|
|
||||||
docker compose up -d
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
## 🎯 Image Variants
|
||||||
|
|
||||||
## 🧪 Specialized Environments
|
### Core Images
|
||||||
|
|
||||||
Located in: [`.docker/compose/`](compose/README.md)
|
| Image | Base OS | Build Method | Size | Use Case |
|
||||||
|
|-------|---------|--------------|------|----------|
|
||||||
|
| `production` (default) | Alpine 3.18 | GitHub Releases | Smallest | Production deployment |
|
||||||
|
| `source` | Debian Bookworm | Source build | Medium | Custom builds with cross-compilation |
|
||||||
|
| `dev` | Debian Bookworm | Development tools | Large | Interactive development |
|
||||||
|
|
||||||
These configurations are tailored for specific testing scenarios that require complex topologies.
|
## 🚀 Usage Examples
|
||||||
|
|
||||||
|
### Quick Start (Production)
|
||||||
|
|
||||||
### Distributed Cluster (4-Nodes)
|
|
||||||
Simulates a real-world distributed environment with 4 RustFS nodes running locally.
|
|
||||||
```bash
|
```bash
|
||||||
docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
|
# Default production image (Alpine + GitHub Releases)
|
||||||
|
docker run -p 9000:9000 rustfs/rustfs:latest
|
||||||
|
|
||||||
|
# Specific version
|
||||||
|
docker run -p 9000:9000 rustfs/rustfs:1.2.3
|
||||||
```
|
```
|
||||||
|
|
||||||
### Integrated Observability Test
|
### Complete Tag Strategy Examples
|
||||||
A self-contained environment running 4 RustFS nodes alongside the full observability stack. Useful for end-to-end testing of telemetry.
|
|
||||||
```bash
|
```bash
|
||||||
docker compose -f .docker/compose/docker-compose.observability.yaml up -d
|
# Stable Releases
|
||||||
|
docker run rustfs/rustfs:1.2.3 # Main version (production)
|
||||||
|
docker run rustfs/rustfs:1.2.3-production # Explicit production variant
|
||||||
|
docker run rustfs/rustfs:1.2.3-source # Source build variant
|
||||||
|
docker run rustfs/rustfs:latest # Latest stable
|
||||||
|
|
||||||
|
# Prerelease Versions
|
||||||
|
docker run rustfs/rustfs:1.3.0-alpha.2 # Specific alpha version
|
||||||
|
docker run rustfs/rustfs:alpha # Latest alpha
|
||||||
|
docker run rustfs/rustfs:beta # Latest beta
|
||||||
|
docker run rustfs/rustfs:rc # Latest release candidate
|
||||||
|
|
||||||
|
# Development Versions
|
||||||
|
docker run rustfs/rustfs:dev # Latest main branch development
|
||||||
|
docker run rustfs/rustfs:dev-13e4a0b # Specific commit
|
||||||
|
docker run rustfs/rustfs:dev-latest # Latest development
|
||||||
|
docker run rustfs/rustfs:main-latest # Main branch latest
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
### Development Environment
|
||||||
|
|
||||||
## 📡 MQTT Integration
|
|
||||||
|
|
||||||
Located in: [`.docker/mqtt/`](mqtt/README.md)
|
|
||||||
|
|
||||||
Provides an EMQX broker for testing RustFS MQTT features.
|
|
||||||
|
|
||||||
### Quick Start
|
|
||||||
```bash
|
```bash
|
||||||
cd .docker/mqtt
|
# Quick setup using Makefile (recommended)
|
||||||
docker compose up -d
|
make docker-dev-local # Build development image locally
|
||||||
```
|
make dev-env-start # Start development container
|
||||||
- **Dashboard**: [http://localhost:18083](http://localhost:18083) (Default: `admin` / `public`)
|
|
||||||
- **MQTT Port**: `1883`
|
|
||||||
|
|
||||||
---
|
# Manual Docker commands
|
||||||
|
docker run -it -v $(pwd):/workspace -p 9000:9000 rustfs/rustfs:latest-dev
|
||||||
|
|
||||||
## 👁️ Alternative: OpenObserve
|
# Build from source locally
|
||||||
|
docker build -f Dockerfile.source -t rustfs:custom .
|
||||||
|
|
||||||
Located in: [`.docker/openobserve-otel/`](openobserve-otel/README.md)
|
# Development with hot reload
|
||||||
|
docker-compose up rustfs-dev
|
||||||
For users preferring a lightweight, all-in-one solution, we support OpenObserve. It combines logs, metrics, and traces into a single binary and UI.
|
|
||||||
|
|
||||||
### Quick Start
|
|
||||||
```bash
|
|
||||||
cd .docker/openobserve-otel
|
|
||||||
docker compose up -d
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
## 🏗️ Build Arguments and Scripts
|
||||||
|
|
||||||
## 🔧 Common Operations
|
### Using Makefile Commands (Recommended)
|
||||||
|
|
||||||
|
The easiest way to build images using simplified commands:
|
||||||
|
|
||||||
### Cleaning Up
|
|
||||||
To stop all containers and remove volumes (**WARNING**: deletes all persisted data):
|
|
||||||
```bash
|
```bash
|
||||||
docker compose down -v
|
# Development images (build from source)
|
||||||
|
make docker-dev-local # Build for local use (single arch)
|
||||||
|
make docker-dev # Build multi-arch (for CI/CD)
|
||||||
|
make docker-dev-push REGISTRY=xxx # Build and push to registry
|
||||||
|
|
||||||
|
# Production images (using pre-built binaries)
|
||||||
|
make docker-buildx # Build multi-arch production images
|
||||||
|
make docker-buildx-push # Build and push production images
|
||||||
|
make docker-buildx-version VERSION=v1.0.0 # Build specific version
|
||||||
|
|
||||||
|
# Development environment
|
||||||
|
make dev-env-start # Start development container
|
||||||
|
make dev-env-stop # Stop development container
|
||||||
|
make dev-env-restart # Restart development container
|
||||||
|
|
||||||
|
# Help
|
||||||
|
make help-docker # Show all Docker-related commands
|
||||||
```
|
```
|
||||||
|
|
||||||
### Viewing Logs
|
### Using docker-buildx.sh (Advanced)
|
||||||
To follow logs for a specific service:
|
|
||||||
|
For direct script usage and advanced scenarios:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose logs -f [service_name]
|
# Build latest version for all architectures
|
||||||
|
./docker-buildx.sh
|
||||||
|
|
||||||
|
# Build and push to registry
|
||||||
|
./docker-buildx.sh --push
|
||||||
|
|
||||||
|
# Build specific version
|
||||||
|
./docker-buildx.sh --release v1.2.3
|
||||||
|
|
||||||
|
# Build and push specific version
|
||||||
|
./docker-buildx.sh --release v1.2.3 --push
|
||||||
```
|
```
|
||||||
|
|
||||||
### Checking Status
|
### Manual Docker Builds
|
||||||
To see the status of all running containers:
|
|
||||||
|
All images support dynamic version selection:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose ps
|
# Build production image with latest release
|
||||||
|
docker build --build-arg RELEASE="latest" -t rustfs:latest .
|
||||||
|
|
||||||
|
# Build from source with specific target
|
||||||
|
docker build -f Dockerfile.source \
|
||||||
|
--build-arg TARGETPLATFORM="linux/amd64" \
|
||||||
|
-t rustfs:source .
|
||||||
|
|
||||||
|
# Development build
|
||||||
|
docker build -f Dockerfile.source -t rustfs:dev .
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## 🔧 Binary Download Sources
|
||||||
|
|
||||||
|
### Unified GitHub Releases
|
||||||
|
|
||||||
|
The production image downloads from GitHub Releases for reliability and transparency:
|
||||||
|
|
||||||
|
- ✅ **production** → GitHub Releases API with automatic latest detection
|
||||||
|
- ✅ **Checksum verification** → SHA256SUMS validation when available
|
||||||
|
- ✅ **Multi-architecture** → Supports amd64 and arm64
|
||||||
|
|
||||||
|
### Source Build
|
||||||
|
|
||||||
|
The source variant compiles from source code with advanced features:
|
||||||
|
|
||||||
|
- 🔧 **Cross-compilation** → Supports multiple target platforms via `TARGETPLATFORM`
|
||||||
|
- ⚡ **Build caching** → sccache for faster compilation
|
||||||
|
- 🎯 **Optimized builds** → Release optimizations with LTO and symbol stripping
|
||||||
|
|
||||||
|
## 📋 Architecture Support
|
||||||
|
|
||||||
|
All variants support multi-architecture builds:
|
||||||
|
|
||||||
|
- **linux/amd64** (x86_64)
|
||||||
|
- **linux/arm64** (aarch64)
|
||||||
|
|
||||||
|
Architecture is automatically detected during build using Docker's `TARGETARCH` build argument.
|
||||||
|
|
||||||
|
## 🔐 Security Features
|
||||||
|
|
||||||
|
- **Checksum Verification**: Production image verifies SHA256SUMS when available
|
||||||
|
- **Non-root User**: All images run as user `rustfs` (UID 1000)
|
||||||
|
- **Minimal Runtime**: Production image only includes necessary dependencies
|
||||||
|
- **Secure Defaults**: No hardcoded credentials or keys
|
||||||
|
|
||||||
|
## 🛠️ Development Workflow
|
||||||
|
|
||||||
|
### Quick Start with Makefile (Recommended)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Start development environment
|
||||||
|
make dev-env-start
|
||||||
|
|
||||||
|
# 2. Your development container is now running with:
|
||||||
|
# - Port 9000 exposed for RustFS
|
||||||
|
# - Port 9010 exposed for admin console
|
||||||
|
# - Current directory mounted as /workspace
|
||||||
|
|
||||||
|
# 3. Stop when done
|
||||||
|
make dev-env-stop
|
||||||
|
```
|
||||||
|
|
||||||
|
### Manual Development Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build development image from source
|
||||||
|
make docker-dev-local
|
||||||
|
|
||||||
|
# Or use traditional Docker commands
|
||||||
|
docker build -f Dockerfile.source -t rustfs:dev .
|
||||||
|
|
||||||
|
# Run with development tools
|
||||||
|
docker run -it -v $(pwd):/workspace -p 9000:9000 rustfs:dev bash
|
||||||
|
|
||||||
|
# Or use docker-compose for complex setups
|
||||||
|
docker-compose up rustfs-dev
|
||||||
|
```
|
||||||
|
|
||||||
|
### Common Development Tasks
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build and test locally
|
||||||
|
make build # Build binary natively
|
||||||
|
make docker-dev-local # Build development Docker image
|
||||||
|
make test # Run tests
|
||||||
|
make fmt # Format code
|
||||||
|
make clippy # Run linter
|
||||||
|
|
||||||
|
# Get help
|
||||||
|
make help # General help
|
||||||
|
make help-docker # Docker-specific help
|
||||||
|
make help-build # Build-specific help
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🚀 CI/CD Integration
|
||||||
|
|
||||||
|
The project uses GitHub Actions for automated multi-architecture Docker builds:
|
||||||
|
|
||||||
|
### Automated Builds
|
||||||
|
|
||||||
|
- **Tags**: Automatic builds triggered on version tags (e.g., `v1.2.3`)
|
||||||
|
- **Main Branch**: Development builds with `dev-latest` and `main-latest` tags
|
||||||
|
- **Pull Requests**: Test builds without registry push
|
||||||
|
|
||||||
|
### Build Variants
|
||||||
|
|
||||||
|
Each build creates three image variants:
|
||||||
|
|
||||||
|
- `rustfs/rustfs:v1.2.3` (production - Alpine-based)
|
||||||
|
- `rustfs/rustfs:v1.2.3-source` (source build - Debian-based)
|
||||||
|
- `rustfs/rustfs:v1.2.3-dev` (development - Debian-based with tools)
|
||||||
|
|
||||||
|
### Manual Builds
|
||||||
|
|
||||||
|
Trigger custom builds via GitHub Actions:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Use workflow_dispatch to build specific versions
|
||||||
|
# Available options: latest, main-latest, dev-latest, v1.2.3, dev-abc123
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📦 Supporting Infrastructure
|
||||||
|
|
||||||
|
The `.docker/` directory contains supporting configuration files:
|
||||||
|
|
||||||
|
- **observability/** - Prometheus, Grafana, OpenTelemetry configs
|
||||||
|
- **compose/** - Multi-service Docker Compose setups
|
||||||
|
- **mqtt/** - MQTT broker configurations
|
||||||
|
- **openobserve-otel/** - Log aggregation and tracing setup
|
||||||
|
|
||||||
|
See individual README files in each subdirectory for specific usage instructions.
|
||||||
|
|||||||
+58
-80
@@ -1,102 +1,80 @@
|
|||||||
# Specialized Docker Compose Configurations
|
# Docker Compose Configurations
|
||||||
|
|
||||||
This directory contains specialized Docker Compose configurations for specific testing scenarios.
|
This directory contains specialized Docker Compose configurations for different use cases.
|
||||||
|
|
||||||
## ⚠️ Important Note
|
|
||||||
|
|
||||||
**For Observability:**
|
|
||||||
We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
|
|
||||||
|
|
||||||
The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.
|
|
||||||
|
|
||||||
## 📁 Configuration Files
|
## 📁 Configuration Files
|
||||||
|
|
||||||
### Cluster Testing
|
This directory contains specialized Docker Compose configurations and their associated Dockerfiles, keeping related files organized together.
|
||||||
|
|
||||||
- **`docker-compose.cluster.yaml`**
|
### Main Configuration (Root Directory)
|
||||||
- **Purpose**: Simulates a 4-node RustFS distributed cluster.
|
|
||||||
- **Use Case**: Testing distributed storage logic, consensus, and failover.
|
|
||||||
- **Nodes**: 4 RustFS instances.
|
|
||||||
- **Storage**: Uses local HTTP endpoints.
|
|
||||||
|
|
||||||
### Legacy / Minimal Observability
|
- **`../../docker-compose.yml`** - **Default Production Setup**
|
||||||
|
- Complete production-ready configuration
|
||||||
|
- Includes RustFS server + full observability stack
|
||||||
|
- Supports multiple profiles: `dev`, `observability`, `cache`, `proxy`
|
||||||
|
- Recommended for most users
|
||||||
|
|
||||||
- **`docker-compose.observability.yaml`**
|
### Specialized Configurations
|
||||||
- **Purpose**: A minimal observability setup.
|
|
||||||
- **Status**: **Deprecated**. Please use `../observability/docker-compose.yml` instead.
|
- **`docker-compose.cluster.yaml`** - **Distributed Testing**
|
||||||
|
- 4-node cluster setup for testing distributed storage
|
||||||
|
- Uses local compiled binaries
|
||||||
|
- Simulates multi-node environment
|
||||||
|
- Ideal for development and cluster testing
|
||||||
|
|
||||||
|
- **`docker-compose.observability.yaml`** - **Observability Focus**
|
||||||
|
- Specialized setup for testing observability features
|
||||||
|
- Includes OpenTelemetry, Jaeger, Prometheus, Loki, Grafana
|
||||||
|
- Uses `../../Dockerfile.source` for builds
|
||||||
|
- Perfect for observability development
|
||||||
|
|
||||||
## 🚀 Usage Examples
|
## 🚀 Usage Examples
|
||||||
|
|
||||||
|
### Production Setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start main service
|
||||||
|
docker-compose up -d
|
||||||
|
|
||||||
|
# Start with development profile
|
||||||
|
docker-compose --profile dev up -d
|
||||||
|
|
||||||
|
# Start with full observability
|
||||||
|
docker-compose --profile observability up -d
|
||||||
|
```
|
||||||
|
|
||||||
### Cluster Testing
|
### Cluster Testing
|
||||||
|
|
||||||
To start a 4-node cluster for distributed testing:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# From project root
|
# Build and start 4-node cluster (run from project root)
|
||||||
docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
|
cd .docker/compose
|
||||||
|
docker-compose -f docker-compose.cluster.yaml up -d
|
||||||
|
|
||||||
|
# Or run directly from project root
|
||||||
|
docker-compose -f .docker/compose/docker-compose.cluster.yaml up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
### Script-Based 4-Node Validation (Recommended)
|
### Observability Testing
|
||||||
|
|
||||||
Use the local validation script when you need local-source image build, failover checks,
|
|
||||||
and benchmark workflow in one command:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Default mode: WAIT_PROBE_MODE=service
|
# Start observability-focused environment (run from project root)
|
||||||
# This avoids false negatives where /health/ready remains 503 locally
|
cd .docker/compose
|
||||||
# while the service path is already available.
|
docker-compose -f docker-compose.observability.yaml up -d
|
||||||
./scripts/run_four_node_cluster_failover_bench.sh
|
|
||||||
|
# Or run directly from project root
|
||||||
|
docker-compose -f .docker/compose/docker-compose.observability.yaml up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
Strict mode is available when you explicitly want `/health/ready == 200` as the gate:
|
## 🔧 Configuration Overview
|
||||||
|
|
||||||
```bash
|
| Configuration | Nodes | Storage | Observability | Use Case |
|
||||||
WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
|
|---------------|-------|---------|---------------|----------|
|
||||||
```
|
| **Main** | 1 | Volume mounts | Full stack | Production |
|
||||||
|
| **Cluster** | 4 | HTTP endpoints | Basic | Testing |
|
||||||
|
| **Observability** | 4 | Local data | Advanced | Development |
|
||||||
|
|
||||||
### Profiling + Trace Validation
|
## 📝 Notes
|
||||||
|
|
||||||
The profiling-focused 4-node compose keeps profiling enabled and points RustFS
|
- Always ensure you have built the required binaries before starting cluster tests
|
||||||
to an OTLP/HTTP collector endpoint:
|
- The main configuration is sufficient for most use cases
|
||||||
|
- Specialized configurations are for specific testing scenarios
|
||||||
```bash
|
|
||||||
docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
Important behavior notes:
|
|
||||||
|
|
||||||
- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
|
|
||||||
traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
|
|
||||||
- Startup usually produces logs and metrics first. That does not guarantee
|
|
||||||
visible traces yet.
|
|
||||||
- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
|
|
||||||
- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
|
|
||||||
many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
|
|
||||||
`RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.
|
|
||||||
|
|
||||||
Minimal trace verification flow:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. Start the profiling compose with richer span visibility.
|
|
||||||
RUSTFS_OBS_LOGGER_LEVEL=debug \
|
|
||||||
docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
|
|
||||||
|
|
||||||
# 2. Generate real request traffic after startup.
|
|
||||||
curl -I http://127.0.0.1:9000/health
|
|
||||||
curl -I http://127.0.0.1:9000/health/ready
|
|
||||||
|
|
||||||
# 3. Then inspect Tempo or Jaeger.
|
|
||||||
# Grafana: http://localhost:3000
|
|
||||||
# Jaeger: http://localhost:16686
|
|
||||||
```
|
|
||||||
|
|
||||||
If logs and metrics are present but traces are sparse, the most common cause is
|
|
||||||
"no real request traffic yet" or "`info` level filtered nested spans", not an
|
|
||||||
OTLP routing failure.
|
|
||||||
|
|
||||||
### (Deprecated) Minimal Observability
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# From project root
|
|
||||||
docker compose -f .docker/compose/docker-compose.observability.yaml up -d
|
|
||||||
```
|
|
||||||
|
|||||||
@@ -1,236 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
# Profiling-first 4-node local-build compose.
|
|
||||||
#
|
|
||||||
# Goals:
|
|
||||||
# - force linux/amd64 runtime/build on Apple Silicon hosts;
|
|
||||||
# - enable RustFS built-in CPU profiling;
|
|
||||||
# - keep all tuning knobs host-overridable via env.
|
|
||||||
#
|
|
||||||
# Observability notes:
|
|
||||||
# - `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS appends
|
|
||||||
# `/v1/traces`, `/v1/metrics`, and `/v1/logs` automatically.
|
|
||||||
# - Logs and metrics usually appear during startup. Traces mainly appear after
|
|
||||||
# real HTTP/S3/gRPC requests create spans.
|
|
||||||
# - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request trace span but
|
|
||||||
# filters many `debug`-level nested spans. Use `debug` when validating trace
|
|
||||||
# richness rather than collector reachability.
|
|
||||||
|
|
||||||
services:
|
|
||||||
node1:
|
|
||||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
|
|
||||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
|
||||||
build:
|
|
||||||
context: ../..
|
|
||||||
dockerfile: Dockerfile.source
|
|
||||||
hostname: node1
|
|
||||||
environment:
|
|
||||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
|
|
||||||
- RUSTFS_ADDRESS=:9000
|
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
|
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
|
|
||||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
|
||||||
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
|
|
||||||
# should show richer nested spans during request-path verification.
|
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
|
||||||
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
|
|
||||||
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
|
|
||||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
|
||||||
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
|
|
||||||
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
|
|
||||||
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
|
|
||||||
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
|
|
||||||
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
|
|
||||||
extra_hosts:
|
|
||||||
- "host.docker.internal:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- node1_data_0:/data/rustfs0
|
|
||||||
- node1_data_1:/data/rustfs1
|
|
||||||
- node1_data_2:/data/rustfs2
|
|
||||||
- node1_data_3:/data/rustfs3
|
|
||||||
ports:
|
|
||||||
- "9000:9000"
|
|
||||||
networks:
|
|
||||||
- rustfs-cluster-net
|
|
||||||
|
|
||||||
node2:
|
|
||||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
|
|
||||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
|
||||||
build:
|
|
||||||
context: ../..
|
|
||||||
dockerfile: Dockerfile.source
|
|
||||||
hostname: node2
|
|
||||||
environment:
|
|
||||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
|
|
||||||
- RUSTFS_ADDRESS=:9000
|
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
|
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
|
|
||||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
|
||||||
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
|
|
||||||
# should show richer nested spans during request-path verification.
|
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
|
||||||
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
|
|
||||||
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
|
|
||||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
|
||||||
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
|
|
||||||
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
|
|
||||||
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
|
|
||||||
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
|
|
||||||
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
|
|
||||||
extra_hosts:
|
|
||||||
- "host.docker.internal:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- node2_data_0:/data/rustfs0
|
|
||||||
- node2_data_1:/data/rustfs1
|
|
||||||
- node2_data_2:/data/rustfs2
|
|
||||||
- node2_data_3:/data/rustfs3
|
|
||||||
ports:
|
|
||||||
- "9001:9000"
|
|
||||||
networks:
|
|
||||||
- rustfs-cluster-net
|
|
||||||
|
|
||||||
node3:
|
|
||||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
|
|
||||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
|
||||||
build:
|
|
||||||
context: ../..
|
|
||||||
dockerfile: Dockerfile.source
|
|
||||||
hostname: node3
|
|
||||||
environment:
|
|
||||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
|
|
||||||
- RUSTFS_ADDRESS=:9000
|
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
|
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
|
|
||||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
|
||||||
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
|
|
||||||
# should show richer nested spans during request-path verification.
|
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
|
||||||
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
|
|
||||||
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
|
|
||||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
|
||||||
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
|
|
||||||
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
|
|
||||||
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
|
|
||||||
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
|
|
||||||
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
|
|
||||||
extra_hosts:
|
|
||||||
- "host.docker.internal:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- node3_data_0:/data/rustfs0
|
|
||||||
- node3_data_1:/data/rustfs1
|
|
||||||
- node3_data_2:/data/rustfs2
|
|
||||||
- node3_data_3:/data/rustfs3
|
|
||||||
ports:
|
|
||||||
- "9002:9000"
|
|
||||||
networks:
|
|
||||||
- rustfs-cluster-net
|
|
||||||
|
|
||||||
node4:
|
|
||||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
|
|
||||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
|
||||||
build:
|
|
||||||
context: ../..
|
|
||||||
dockerfile: Dockerfile.source
|
|
||||||
hostname: node4
|
|
||||||
environment:
|
|
||||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
|
|
||||||
- RUSTFS_ADDRESS=:9000
|
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
|
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
|
|
||||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
|
||||||
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
|
|
||||||
# should show richer nested spans during request-path verification.
|
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
|
||||||
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
|
|
||||||
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
|
|
||||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
|
||||||
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
|
|
||||||
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
|
|
||||||
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
|
|
||||||
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
|
|
||||||
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
|
|
||||||
extra_hosts:
|
|
||||||
- "host.docker.internal:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- node4_data_0:/data/rustfs0
|
|
||||||
- node4_data_1:/data/rustfs1
|
|
||||||
- node4_data_2:/data/rustfs2
|
|
||||||
- node4_data_3:/data/rustfs3
|
|
||||||
ports:
|
|
||||||
- "9003:9000"
|
|
||||||
networks:
|
|
||||||
- rustfs-cluster-net
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
node1_data_0:
|
|
||||||
node1_data_1:
|
|
||||||
node1_data_2:
|
|
||||||
node1_data_3:
|
|
||||||
node2_data_0:
|
|
||||||
node2_data_1:
|
|
||||||
node2_data_2:
|
|
||||||
node2_data_3:
|
|
||||||
node3_data_0:
|
|
||||||
node3_data_1:
|
|
||||||
node3_data_2:
|
|
||||||
node3_data_3:
|
|
||||||
node4_data_0:
|
|
||||||
node4_data_1:
|
|
||||||
node4_data_2:
|
|
||||||
node4_data_3:
|
|
||||||
|
|
||||||
networks:
|
|
||||||
rustfs-cluster-net:
|
|
||||||
driver: bridge
|
|
||||||
@@ -1,220 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
services:
|
|
||||||
node1:
|
|
||||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
|
||||||
build:
|
|
||||||
context: ../..
|
|
||||||
dockerfile: Dockerfile.source
|
|
||||||
hostname: node1
|
|
||||||
environment:
|
|
||||||
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
|
|
||||||
- RUSTFS_ADDRESS=:9000
|
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
|
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
|
|
||||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
|
||||||
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
|
|
||||||
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
|
|
||||||
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
|
|
||||||
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
|
|
||||||
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
|
|
||||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
|
||||||
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
|
|
||||||
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
|
|
||||||
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
|
|
||||||
- RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
|
|
||||||
- RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
|
|
||||||
- RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
|
|
||||||
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
|
|
||||||
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
|
|
||||||
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
|
|
||||||
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
|
|
||||||
- RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
|
|
||||||
extra_hosts:
|
|
||||||
- "host.docker.internal:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- node1_data_0:/data/rustfs0
|
|
||||||
- node1_data_1:/data/rustfs1
|
|
||||||
- node1_data_2:/data/rustfs2
|
|
||||||
- node1_data_3:/data/rustfs3
|
|
||||||
ports:
|
|
||||||
- "9000:9000"
|
|
||||||
networks:
|
|
||||||
- rustfs-cluster-net
|
|
||||||
|
|
||||||
node2:
|
|
||||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
|
||||||
build:
|
|
||||||
context: ../..
|
|
||||||
dockerfile: Dockerfile.source
|
|
||||||
hostname: node2
|
|
||||||
environment:
|
|
||||||
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
|
|
||||||
- RUSTFS_ADDRESS=:9000
|
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
|
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
|
|
||||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
|
||||||
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
|
|
||||||
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
|
|
||||||
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
|
|
||||||
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
|
|
||||||
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
|
|
||||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
|
||||||
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
|
|
||||||
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
|
|
||||||
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
|
|
||||||
- RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
|
|
||||||
- RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
|
|
||||||
- RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
|
|
||||||
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
|
|
||||||
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
|
|
||||||
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
|
|
||||||
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
|
|
||||||
- RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
|
|
||||||
extra_hosts:
|
|
||||||
- "host.docker.internal:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- node2_data_0:/data/rustfs0
|
|
||||||
- node2_data_1:/data/rustfs1
|
|
||||||
- node2_data_2:/data/rustfs2
|
|
||||||
- node2_data_3:/data/rustfs3
|
|
||||||
ports:
|
|
||||||
- "9001:9000"
|
|
||||||
networks:
|
|
||||||
- rustfs-cluster-net
|
|
||||||
|
|
||||||
node3:
|
|
||||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
|
||||||
build:
|
|
||||||
context: ../..
|
|
||||||
dockerfile: Dockerfile.source
|
|
||||||
hostname: node3
|
|
||||||
environment:
|
|
||||||
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
|
|
||||||
- RUSTFS_ADDRESS=:9000
|
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
|
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
|
|
||||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
|
||||||
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
|
|
||||||
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
|
|
||||||
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
|
|
||||||
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
|
|
||||||
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
|
|
||||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
|
||||||
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
|
|
||||||
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
|
|
||||||
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
|
|
||||||
- RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
|
|
||||||
- RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
|
|
||||||
- RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
|
|
||||||
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
|
|
||||||
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
|
|
||||||
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
|
|
||||||
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
|
|
||||||
- RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
|
|
||||||
extra_hosts:
|
|
||||||
- "host.docker.internal:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- node3_data_0:/data/rustfs0
|
|
||||||
- node3_data_1:/data/rustfs1
|
|
||||||
- node3_data_2:/data/rustfs2
|
|
||||||
- node3_data_3:/data/rustfs3
|
|
||||||
ports:
|
|
||||||
- "9002:9000"
|
|
||||||
networks:
|
|
||||||
- rustfs-cluster-net
|
|
||||||
|
|
||||||
node4:
|
|
||||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
|
||||||
build:
|
|
||||||
context: ../..
|
|
||||||
dockerfile: Dockerfile.source
|
|
||||||
hostname: node4
|
|
||||||
environment:
|
|
||||||
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
|
|
||||||
- RUSTFS_ADDRESS=:9000
|
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
|
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
|
|
||||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
|
||||||
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
|
|
||||||
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
|
|
||||||
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
|
|
||||||
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
|
|
||||||
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
|
|
||||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
|
||||||
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
|
|
||||||
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
|
|
||||||
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
|
|
||||||
- RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
|
|
||||||
- RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
|
|
||||||
- RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
|
|
||||||
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
|
|
||||||
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
|
|
||||||
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
|
|
||||||
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
|
|
||||||
- RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
|
|
||||||
- RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
|
|
||||||
extra_hosts:
|
|
||||||
- "host.docker.internal:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- node4_data_0:/data/rustfs0
|
|
||||||
- node4_data_1:/data/rustfs1
|
|
||||||
- node4_data_2:/data/rustfs2
|
|
||||||
- node4_data_3:/data/rustfs3
|
|
||||||
ports:
|
|
||||||
- "9003:9000"
|
|
||||||
networks:
|
|
||||||
- rustfs-cluster-net
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
node1_data_0:
|
|
||||||
node1_data_1:
|
|
||||||
node1_data_2:
|
|
||||||
node1_data_3:
|
|
||||||
node2_data_0:
|
|
||||||
node2_data_1:
|
|
||||||
node2_data_2:
|
|
||||||
node2_data_3:
|
|
||||||
node3_data_0:
|
|
||||||
node3_data_1:
|
|
||||||
node3_data_2:
|
|
||||||
node3_data_3:
|
|
||||||
node4_data_0:
|
|
||||||
node4_data_1:
|
|
||||||
node4_data_2:
|
|
||||||
node4_data_3:
|
|
||||||
|
|
||||||
networks:
|
|
||||||
rustfs-cluster-net:
|
|
||||||
driver: bridge
|
|
||||||
-56
@@ -1,56 +0,0 @@
|
|||||||
services:
|
|
||||||
node1:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=16
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
|
|
||||||
|
|
||||||
node2:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=16
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
|
|
||||||
|
|
||||||
node3:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=16
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
|
|
||||||
|
|
||||||
node4:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=16
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
|
|
||||||
-56
@@ -1,56 +0,0 @@
|
|||||||
services:
|
|
||||||
node1:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=24
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
|
|
||||||
|
|
||||||
node2:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=24
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
|
|
||||||
|
|
||||||
node3:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=24
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
|
|
||||||
|
|
||||||
node4:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=24
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
|
|
||||||
@@ -1,56 +0,0 @@
|
|||||||
services:
|
|
||||||
node1:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=20
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
|
|
||||||
|
|
||||||
node2:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=20
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
|
|
||||||
|
|
||||||
node3:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=20
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
|
|
||||||
|
|
||||||
node4:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=20
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
|
|
||||||
@@ -1,56 +0,0 @@
|
|||||||
services:
|
|
||||||
node1:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=12
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
|
|
||||||
|
|
||||||
node2:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=12
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
|
|
||||||
|
|
||||||
node3:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=12
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
|
|
||||||
|
|
||||||
node4:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=12
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
|
|
||||||
@@ -1,56 +0,0 @@
|
|||||||
services:
|
|
||||||
node1:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=6
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
|
|
||||||
|
|
||||||
node2:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=6
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
|
|
||||||
|
|
||||||
node3:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=6
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
|
|
||||||
|
|
||||||
node4:
|
|
||||||
environment:
|
|
||||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
|
|
||||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
|
||||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
|
||||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
|
||||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
|
|
||||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
|
||||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
|
|
||||||
- RUSTFS_RUNTIME_WORKER_THREADS=6
|
|
||||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
|
|
||||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
|
||||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
|
|
||||||
@@ -21,8 +21,8 @@ services:
|
|||||||
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
|
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
|
||||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
- RUSTFS_CONSOLE_ENABLE=true
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
|
- RUSTFS_ACCESS_KEY=rustfsadmin
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
|
- RUSTFS_SECRET_KEY=rustfsadmin
|
||||||
platform: linux/amd64
|
platform: linux/amd64
|
||||||
ports:
|
ports:
|
||||||
- "9000:9000" # Map port 9001 of the host to port 9000 of the container
|
- "9000:9000" # Map port 9001 of the host to port 9000 of the container
|
||||||
@@ -38,8 +38,8 @@ services:
|
|||||||
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
|
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
|
||||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
- RUSTFS_CONSOLE_ENABLE=true
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
|
- RUSTFS_ACCESS_KEY=rustfsadmin
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
|
- RUSTFS_SECRET_KEY=rustfsadmin
|
||||||
platform: linux/amd64
|
platform: linux/amd64
|
||||||
ports:
|
ports:
|
||||||
- "9001:9000" # Map port 9002 of the host to port 9000 of the container
|
- "9001:9000" # Map port 9002 of the host to port 9000 of the container
|
||||||
@@ -55,8 +55,8 @@ services:
|
|||||||
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
|
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
|
||||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
- RUSTFS_CONSOLE_ENABLE=true
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
|
- RUSTFS_ACCESS_KEY=rustfsadmin
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
|
- RUSTFS_SECRET_KEY=rustfsadmin
|
||||||
platform: linux/amd64
|
platform: linux/amd64
|
||||||
ports:
|
ports:
|
||||||
- "9002:9000" # Map port 9003 of the host to port 9000 of the container
|
- "9002:9000" # Map port 9003 of the host to port 9000 of the container
|
||||||
@@ -72,8 +72,8 @@ services:
|
|||||||
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
|
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
|
||||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
- RUSTFS_CONSOLE_ENABLE=true
|
||||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
|
- RUSTFS_ACCESS_KEY=rustfsadmin
|
||||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
|
- RUSTFS_SECRET_KEY=rustfsadmin
|
||||||
platform: linux/amd64
|
platform: linux/amd64
|
||||||
ports:
|
ports:
|
||||||
- "9003:9000" # Map port 9004 of the host to port 9000 of the container
|
- "9003:9000" # Map port 9004 of the host to port 9000 of the container
|
||||||
|
|||||||
@@ -13,183 +13,62 @@
|
|||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
services:
|
services:
|
||||||
# --- Observability Stack ---
|
|
||||||
|
|
||||||
tempo-init:
|
|
||||||
image: busybox:latest
|
|
||||||
command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
|
|
||||||
volumes:
|
|
||||||
- tempo-data:/var/tempo
|
|
||||||
user: root
|
|
||||||
networks:
|
|
||||||
- rustfs-network
|
|
||||||
restart: "no"
|
|
||||||
|
|
||||||
tempo:
|
|
||||||
image: grafana/tempo:2.10.5
|
|
||||||
user: "10001"
|
|
||||||
command: [ "-config.file=/etc/tempo.yaml" ]
|
|
||||||
volumes:
|
|
||||||
- ../../.docker/observability/tempo.yaml:/etc/tempo.yaml:ro
|
|
||||||
- tempo-data:/var/tempo
|
|
||||||
ports:
|
|
||||||
- "3200:3200" # tempo
|
|
||||||
- "4317" # otlp grpc
|
|
||||||
- "4318" # otlp http
|
|
||||||
- "7946" # memberlist
|
|
||||||
restart: unless-stopped
|
|
||||||
networks:
|
|
||||||
- rustfs-network
|
|
||||||
depends_on:
|
|
||||||
tempo-init:
|
|
||||||
condition: service_completed_successfully
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "/tempo", "-version" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
start_period: 15s
|
|
||||||
|
|
||||||
otel-collector:
|
otel-collector:
|
||||||
image: otel/opentelemetry-collector-contrib:latest
|
image: otel/opentelemetry-collector-contrib:0.129.1
|
||||||
environment:
|
environment:
|
||||||
- TZ=Asia/Shanghai
|
- TZ=Asia/Shanghai
|
||||||
volumes:
|
volumes:
|
||||||
- ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
|
- ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
|
||||||
ports:
|
ports:
|
||||||
- "1888:1888" # pprof
|
- 1888:1888
|
||||||
- "8888:8888" # Prometheus metrics for Collector
|
- 8888:8888
|
||||||
- "8889:8889" # Prometheus metrics for application indicators
|
- 8889:8889
|
||||||
- "13133:13133" # health check
|
- 13133:13133
|
||||||
- "4317:4317" # OTLP gRPC
|
- 4317:4317
|
||||||
- "4318:4318" # OTLP HTTP
|
- 4318:4318
|
||||||
- "55679:55679" # zpages
|
- 55679:55679
|
||||||
networks:
|
networks:
|
||||||
- rustfs-network
|
- rustfs-network
|
||||||
depends_on:
|
|
||||||
- tempo
|
|
||||||
- jaeger
|
|
||||||
- prometheus
|
|
||||||
- loki
|
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "/otelcol-contrib", "--version" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
|
|
||||||
jaeger:
|
jaeger:
|
||||||
image: jaegertracing/jaeger:latest
|
image: jaegertracing/jaeger:2.8.0
|
||||||
environment:
|
environment:
|
||||||
- TZ=Asia/Shanghai
|
- TZ=Asia/Shanghai
|
||||||
- SPAN_STORAGE_TYPE=badger
|
|
||||||
- BADGER_EPHEMERAL=false
|
|
||||||
- BADGER_DIRECTORY_VALUE=/badger/data
|
|
||||||
- BADGER_DIRECTORY_KEY=/badger/key
|
|
||||||
- COLLECTOR_OTLP_ENABLED=true
|
|
||||||
volumes:
|
|
||||||
- ../../.docker/observability/jaeger.yaml:/etc/jaeger/config.yml:ro
|
|
||||||
- jaeger-data:/badger
|
|
||||||
ports:
|
ports:
|
||||||
- "16686:16686" # Web UI
|
- "16686:16686"
|
||||||
- "14269:14269" # Admin/Metrics
|
- "14317:4317"
|
||||||
- "4317" # otlp grpc
|
- "14318:4318"
|
||||||
- "4318" # otlp http
|
|
||||||
command: [ "--config", "/etc/jaeger/config.yml" ]
|
|
||||||
networks:
|
networks:
|
||||||
- rustfs-network
|
- rustfs-network
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
start_period: 15s
|
|
||||||
|
|
||||||
prometheus:
|
prometheus:
|
||||||
image: prom/prometheus:latest
|
image: prom/prometheus:v3.4.2
|
||||||
environment:
|
environment:
|
||||||
- TZ=Asia/Shanghai
|
- TZ=Asia/Shanghai
|
||||||
volumes:
|
volumes:
|
||||||
- ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
- ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml
|
||||||
- ../../.docker/observability/prometheus-rules:/etc/prometheus/rules:ro
|
|
||||||
- prometheus-data:/prometheus
|
|
||||||
ports:
|
ports:
|
||||||
- "9090:9090"
|
- "9090:9090"
|
||||||
command:
|
|
||||||
- '--config.file=/etc/prometheus/prometheus.yml'
|
|
||||||
- '--web.enable-otlp-receiver'
|
|
||||||
- '--web.enable-remote-write-receiver'
|
|
||||||
- '--enable-feature=promql-experimental-functions'
|
|
||||||
- '--storage.tsdb.path=/prometheus'
|
|
||||||
- '--storage.tsdb.retention.time=30d'
|
|
||||||
networks:
|
networks:
|
||||||
- rustfs-network
|
- rustfs-network
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
|
|
||||||
loki:
|
loki:
|
||||||
image: grafana/loki:latest
|
image: grafana/loki:3.5.1
|
||||||
environment:
|
environment:
|
||||||
- TZ=Asia/Shanghai
|
- TZ=Asia/Shanghai
|
||||||
volumes:
|
volumes:
|
||||||
- ../../.docker/observability/loki.yaml:/etc/loki/loki.yaml:ro
|
- ../../.docker/observability/loki-config.yaml:/etc/loki/local-config.yaml
|
||||||
- loki-data:/loki
|
|
||||||
ports:
|
ports:
|
||||||
- "3100:3100"
|
- "3100:3100"
|
||||||
command: -config.file=/etc/loki/loki.yaml
|
command: -config.file=/etc/loki/local-config.yaml
|
||||||
networks:
|
networks:
|
||||||
- rustfs-network
|
- rustfs-network
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
|
|
||||||
interval: 15s
|
|
||||||
timeout: 10s
|
|
||||||
retries: 5
|
|
||||||
start_period: 60s
|
|
||||||
|
|
||||||
pyroscope:
|
|
||||||
image: grafana/pyroscope:latest
|
|
||||||
ports:
|
|
||||||
- "4040:4040"
|
|
||||||
command:
|
|
||||||
- -self-profiling.disable-push=true
|
|
||||||
networks:
|
|
||||||
- rustfs-network
|
|
||||||
restart: unless-stopped
|
|
||||||
|
|
||||||
grafana:
|
grafana:
|
||||||
image: grafana/grafana:latest
|
image: grafana/grafana:12.0.2
|
||||||
ports:
|
ports:
|
||||||
- "3000:3000" # Web UI
|
- "3000:3000" # Web UI
|
||||||
environment:
|
environment:
|
||||||
- GF_SECURITY_ADMIN_PASSWORD=admin
|
- GF_SECURITY_ADMIN_PASSWORD=admin
|
||||||
- GF_SECURITY_ADMIN_USER=admin
|
|
||||||
- TZ=Asia/Shanghai
|
- TZ=Asia/Shanghai
|
||||||
- GF_INSTALL_PLUGINS=grafana-pyroscope-datasource
|
|
||||||
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
|
|
||||||
networks:
|
networks:
|
||||||
- rustfs-network
|
- rustfs-network
|
||||||
volumes:
|
|
||||||
- ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
|
|
||||||
- ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
|
|
||||||
- grafana-data:/var/lib/grafana
|
|
||||||
depends_on:
|
|
||||||
- prometheus
|
|
||||||
- tempo
|
|
||||||
- loki
|
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
|
|
||||||
# --- RustFS Cluster ---
|
|
||||||
|
|
||||||
node1:
|
node1:
|
||||||
build:
|
build:
|
||||||
@@ -200,15 +79,13 @@ services:
|
|||||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
||||||
- RUSTFS_ADDRESS=:9000
|
- RUSTFS_ADDRESS=:9000
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
- RUSTFS_CONSOLE_ENABLE=true
|
||||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
||||||
platform: linux/amd64
|
platform: linux/amd64
|
||||||
ports:
|
ports:
|
||||||
- "9001:9000"
|
- "9001:9000" # Map port 9001 of the host to port 9000 of the container
|
||||||
networks:
|
networks:
|
||||||
- rustfs-network
|
- rustfs-network
|
||||||
depends_on:
|
|
||||||
- otel-collector
|
|
||||||
|
|
||||||
node2:
|
node2:
|
||||||
build:
|
build:
|
||||||
@@ -219,15 +96,13 @@ services:
|
|||||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
||||||
- RUSTFS_ADDRESS=:9000
|
- RUSTFS_ADDRESS=:9000
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
- RUSTFS_CONSOLE_ENABLE=true
|
||||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
||||||
platform: linux/amd64
|
platform: linux/amd64
|
||||||
ports:
|
ports:
|
||||||
- "9002:9000"
|
- "9002:9000" # Map port 9002 of the host to port 9000 of the container
|
||||||
networks:
|
networks:
|
||||||
- rustfs-network
|
- rustfs-network
|
||||||
depends_on:
|
|
||||||
- otel-collector
|
|
||||||
|
|
||||||
node3:
|
node3:
|
||||||
build:
|
build:
|
||||||
@@ -238,15 +113,13 @@ services:
|
|||||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
||||||
- RUSTFS_ADDRESS=:9000
|
- RUSTFS_ADDRESS=:9000
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
- RUSTFS_CONSOLE_ENABLE=true
|
||||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
||||||
platform: linux/amd64
|
platform: linux/amd64
|
||||||
ports:
|
ports:
|
||||||
- "9003:9000"
|
- "9003:9000" # Map port 9003 of the host to port 9000 of the container
|
||||||
networks:
|
networks:
|
||||||
- rustfs-network
|
- rustfs-network
|
||||||
depends_on:
|
|
||||||
- otel-collector
|
|
||||||
|
|
||||||
node4:
|
node4:
|
||||||
build:
|
build:
|
||||||
@@ -257,22 +130,13 @@ services:
|
|||||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
||||||
- RUSTFS_ADDRESS=:9000
|
- RUSTFS_ADDRESS=:9000
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
- RUSTFS_CONSOLE_ENABLE=true
|
||||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
||||||
platform: linux/amd64
|
platform: linux/amd64
|
||||||
ports:
|
ports:
|
||||||
- "9004:9000"
|
- "9004:9000" # Map port 9004 of the host to port 9000 of the container
|
||||||
networks:
|
networks:
|
||||||
- rustfs-network
|
- rustfs-network
|
||||||
depends_on:
|
|
||||||
- otel-collector
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
prometheus-data:
|
|
||||||
tempo-data:
|
|
||||||
loki-data:
|
|
||||||
jaeger-data:
|
|
||||||
grafana-data:
|
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
rustfs-network:
|
rustfs-network:
|
||||||
|
|||||||
@@ -1,30 +0,0 @@
|
|||||||
# MQTT Broker (EMQX)
|
|
||||||
|
|
||||||
This directory contains the configuration for running an EMQX MQTT broker, which can be used for testing RustFS's MQTT integration.
|
|
||||||
|
|
||||||
## 🚀 Quick Start
|
|
||||||
|
|
||||||
To start the EMQX broker:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📊 Access
|
|
||||||
|
|
||||||
- **Dashboard**: [http://localhost:18083](http://localhost:18083)
|
|
||||||
- **Default Credentials**: `admin` / `public`
|
|
||||||
- **MQTT Port**: `1883`
|
|
||||||
- **WebSocket Port**: `8083`
|
|
||||||
|
|
||||||
## 🛠️ Configuration
|
|
||||||
|
|
||||||
The `docker-compose.yml` file sets up a single-node EMQX instance.
|
|
||||||
|
|
||||||
- **Persistence**: Data is not persisted by default (for testing).
|
|
||||||
- **Network**: Uses the default bridge network.
|
|
||||||
|
|
||||||
## 📝 Notes
|
|
||||||
|
|
||||||
- This setup is intended for development and testing purposes.
|
|
||||||
- For production deployments, please refer to the official [EMQX Documentation](https://www.emqx.io/docs/en/latest/).
|
|
||||||
@@ -1,93 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
worker_processes auto;
|
|
||||||
pid /var/run/nginx.pid;
|
|
||||||
|
|
||||||
events {
|
|
||||||
worker_connections 1024;
|
|
||||||
}
|
|
||||||
|
|
||||||
http {
|
|
||||||
include /etc/nginx/mime.types;
|
|
||||||
default_type application/octet-stream;
|
|
||||||
|
|
||||||
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
|
|
||||||
'$status $body_bytes_sent "$http_referer" '
|
|
||||||
'"$http_user_agent" "$http_x_forwarded_for"';
|
|
||||||
|
|
||||||
access_log /var/log/nginx/access.log main;
|
|
||||||
error_log /var/log/nginx/error.log warn;
|
|
||||||
|
|
||||||
sendfile on;
|
|
||||||
keepalive_timeout 65;
|
|
||||||
|
|
||||||
# RustFS Server Block
|
|
||||||
server {
|
|
||||||
listen 80;
|
|
||||||
server_name localhost;
|
|
||||||
|
|
||||||
# Redirect HTTP to HTTPS (optional, uncomment if SSL is configured)
|
|
||||||
# return 301 https://$host$request_uri;
|
|
||||||
|
|
||||||
location / {
|
|
||||||
proxy_pass http://rustfs:9000;
|
|
||||||
proxy_set_header Host $host;
|
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
||||||
proxy_set_header X-Forwarded-Proto $scheme;
|
|
||||||
|
|
||||||
# S3 specific headers
|
|
||||||
proxy_set_header X-Amz-Date $http_x_amz_date;
|
|
||||||
proxy_set_header Authorization $http_authorization;
|
|
||||||
|
|
||||||
# Disable buffering for large uploads
|
|
||||||
proxy_request_buffering off;
|
|
||||||
client_max_body_size 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
location /rustfs/console {
|
|
||||||
proxy_pass http://rustfs:9001;
|
|
||||||
proxy_set_header Host $host;
|
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
||||||
proxy_set_header X-Forwarded-Proto $scheme;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# SSL Configuration (Example)
|
|
||||||
# server {
|
|
||||||
# listen 443 ssl;
|
|
||||||
# server_name localhost;
|
|
||||||
#
|
|
||||||
# ssl_certificate /etc/nginx/ssl/server.crt;
|
|
||||||
# ssl_certificate_key /etc/nginx/ssl/server.key;
|
|
||||||
#
|
|
||||||
# # Restrict to modern TLS versions and ciphers. Operators copying this
|
|
||||||
# # example must keep at least these directives — without them, nginx
|
|
||||||
# # may negotiate older protocol versions that have known weaknesses.
|
|
||||||
# ssl_protocols TLSv1.2 TLSv1.3;
|
|
||||||
# ssl_prefer_server_ciphers on;
|
|
||||||
# ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
|
|
||||||
# ssl_session_timeout 1d;
|
|
||||||
# ssl_session_cache shared:SSL:10m;
|
|
||||||
# ssl_session_tickets off;
|
|
||||||
# # add_header Strict-Transport-Security "max-age=63072000" always;
|
|
||||||
#
|
|
||||||
# location / {
|
|
||||||
# proxy_pass http://rustfs:9000;
|
|
||||||
# ...
|
|
||||||
# }
|
|
||||||
# }
|
|
||||||
}
|
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
jaeger-data/*
|
|
||||||
loki-data/*
|
|
||||||
prometheus-data/*
|
|
||||||
tempo-data/*
|
|
||||||
grafana-data/*
|
|
||||||
+81
-193
@@ -1,221 +1,109 @@
|
|||||||
# RustFS Observability Stack
|
# Observability
|
||||||
|
|
||||||
This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.
|
This directory contains the observability stack for the application. The stack is composed of the following components:
|
||||||
|
|
||||||
## Components
|
- Prometheus v3.2.1
|
||||||
|
- Grafana 11.6.0
|
||||||
|
- Loki 3.4.2
|
||||||
|
- Jaeger 2.4.0
|
||||||
|
- Otel Collector 0.120.0 # 0.121.0 remove loki
|
||||||
|
|
||||||
The stack is composed of the following best-in-class open-source components:
|
## Prometheus
|
||||||
|
|
||||||
- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
|
Prometheus is a monitoring and alerting toolkit. It scrapes metrics from instrumented jobs, either directly or via an
|
||||||
- **Grafana** (v11.1.0): The leading platform for observability visualization.
|
intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to
|
||||||
- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
|
either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be
|
||||||
- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
|
used to visualize the collected data.
|
||||||
- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
|
|
||||||
- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
|
|
||||||
|
|
||||||
By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
|
## Grafana
|
||||||
If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
|
|
||||||
|
|
||||||
## Architecture
|
Grafana is a multi-platform open-source analytics and interactive visualization web application. It provides charts,
|
||||||
|
graphs, and alerts for the web when connected to supported data sources.
|
||||||
|
|
||||||
1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
|
## Loki
|
||||||
2. **Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
|
|
||||||
- **Traces** -> **Tempo** (Primary) & **Jaeger** (Secondary/Optional)
|
|
||||||
- **Metrics** -> **Prometheus** (via scraping the Collector's exporter)
|
|
||||||
- **Logs** -> **Loki**
|
|
||||||
3. **Visualization**: **Grafana** connects to all backends (Prometheus, Tempo, Loki, Jaeger) to provide a unified dashboard experience.
|
|
||||||
|
|
||||||
## Features
|
Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is
|
||||||
|
designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of
|
||||||
|
labels for each log stream.
|
||||||
|
|
||||||
- **Full Persistence**: All data (Metrics, Logs, Traces) is persisted to Docker volumes, ensuring no data loss on restart.
|
## Jaeger
|
||||||
- **Correlation**: Seamless navigation between Metrics, Logs, and Traces in Grafana.
|
|
||||||
- Jump from a Metric spike to relevant Traces.
|
|
||||||
- Jump from a Trace to relevant Logs.
|
|
||||||
- **High Performance**: Optimized configurations for batching, compression, and memory management.
|
|
||||||
- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
|
|
||||||
|
|
||||||
## GET Performance Optimization Dashboards
|
Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and
|
||||||
|
troubleshooting microservices-based distributed systems, including:
|
||||||
|
|
||||||
Three pre-built Grafana dashboards are included for monitoring RustFS GET performance optimization rollout:
|
- Distributed context propagation
|
||||||
|
- Distributed transaction monitoring
|
||||||
|
- Root cause analysis
|
||||||
|
- Service dependency analysis
|
||||||
|
- Performance / latency optimization
|
||||||
|
|
||||||
### Available Dashboards
|
## Otel Collector
|
||||||
|
|
||||||
| Dashboard | File | Description |
|
The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry
|
||||||
|-----------|------|-------------|
|
data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source
|
||||||
| **GET Rollout Health** | `grafana-get-rollout-health.json` | Monitors optimization rollout: latency by reader path, early-stop hit rate, codec streaming usage, pipeline failures |
|
observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends.
|
||||||
| **GET Data Integrity** | `grafana-get-data-integrity.json` | Monitors data safety: bitrot verify failures, decode errors, short reads, shard read outcomes |
|
|
||||||
| **GET Resource Impact** | `grafana-get-resource-impact.json` | Monitors resource usage: concurrent requests, IO queue utilization, disk permit wait, RSS trend |
|
|
||||||
| **Object Data Cache** | `grafana-object-data-cache.json` | Monitors the GET body cache (`rustfs_object_data_cache_*`): hit ratio, lookup/plan/fill outcomes, fill duration quantiles, hit vs fill throughput, entries/weighted bytes, inflight fills, memory-pressure skips, invalidations, and size-class breakdowns |
|
|
||||||
|
|
||||||
### Prometheus Alert Rules
|
## How to use
|
||||||
|
|
||||||
The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-configured alerting rules:
|
To deploy the observability stack, run the following command:
|
||||||
|
|
||||||
| Alert | Severity | Condition |
|
- docker latest version
|
||||||
|-------|----------|-----------|
|
|
||||||
| `GetP99Regression` | Critical | GET p99 latency > 2x baseline for 10m |
|
|
||||||
| `PipelineFailureSpike` | Critical | Pipeline failure rate > 5x baseline for 5m |
|
|
||||||
| `BitrotMismatchSpike` | Critical | Bitrot mismatch rate > 3x baseline for 5m |
|
|
||||||
| `EarlyStopInsufficientQuorum` | Warning | Early-stop insufficient quorum rate > 0.1/s for 5m |
|
|
||||||
| `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
|
|
||||||
| `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
|
|
||||||
|
|
||||||
The file `prometheus-rules/rustfs-kms-alerts.yml` contains alerting rules for the KMS backend operation metrics. Thresholds are conservative defaults pending staging baseline calibration; response procedures live in `docs/operations/kms-observability-runbook.md`, and the matching dashboard is `deploy/observability/grafana/rustfs-kms-observability.json`.
|
```bash
|
||||||
|
docker compose -f docker-compose.yml -f docker-compose.override.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
| Alert | Severity | Condition |
|
- docker compose v2.0.0 or before
|
||||||
|-------|----------|-----------|
|
|
||||||
| `KmsBackendFatalErrors` | Critical | Fatal (non-retryable) attempt failures > 0 for 5m |
|
|
||||||
| `KmsBackendHighErrorRate` | Critical | Non-success operation ratio > 5% for 10m (with traffic guard) |
|
|
||||||
| `KmsBackendP99LatencyHigh` | Warning | Operation p99 duration (incl. retries) > 2s for 10m |
|
|
||||||
| `KmsBackendAttemptFailureSpike` | Warning | Attempt failure rate > 0.5/s for 10m |
|
|
||||||
| `KmsBackendRetryBudgetExhausted` | Warning | budget_exhausted / deadline_exceeded outcomes > 0.05/s for 10m |
|
|
||||||
|
|
||||||
### Enabling Alert Rules
|
```bash
|
||||||
|
docke-compose -f docker-compose.yml -f docker-compose.override.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
Add the alert rules file to your Prometheus configuration:
|
To access the Grafana dashboard, navigate to `http://localhost:3000` in your browser. The default username and password
|
||||||
|
are `admin` and `admin`, respectively.
|
||||||
|
|
||||||
|
To access the Jaeger dashboard, navigate to `http://localhost:16686` in your browser.
|
||||||
|
|
||||||
|
To access the Prometheus dashboard, navigate to `http://localhost:9090` in your browser.
|
||||||
|
|
||||||
|
## How to stop
|
||||||
|
|
||||||
|
To stop the observability stack, run the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.yml -f docker-compose.override.yml down
|
||||||
|
```
|
||||||
|
|
||||||
|
## How to remove data
|
||||||
|
|
||||||
|
To remove the data generated by the observability stack, run the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.yml -f docker-compose.override.yml down -v
|
||||||
|
```
|
||||||
|
|
||||||
|
## How to configure
|
||||||
|
|
||||||
|
To configure the observability stack, modify the `docker-compose.override.yml` file. The file contains the following
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
# prometheus.yml
|
services:
|
||||||
rule_files:
|
prometheus:
|
||||||
- "/etc/prometheus/rules/*.yml"
|
environment:
|
||||||
|
- PROMETHEUS_CONFIG_FILE=/etc/prometheus/prometheus.yml
|
||||||
|
volumes:
|
||||||
|
- ./prometheus.yml:/etc/prometheus/prometheus.yml
|
||||||
|
|
||||||
# Or mount the file in docker-compose.yml:
|
grafana:
|
||||||
# volumes:
|
environment:
|
||||||
# - ./prometheus-rules:/etc/prometheus/rules
|
- GF_SECURITY_ADMIN_PASSWORD=admin
|
||||||
|
volumes:
|
||||||
|
- ./grafana/provisioning:/etc/grafana/provisioning
|
||||||
```
|
```
|
||||||
|
|
||||||
### Dashboard Usage
|
The `prometheus` service mounts the `prometheus.yml` file to `/etc/prometheus/prometheus.yml`. The `grafana` service
|
||||||
|
mounts the `grafana/provisioning` directory to `/etc/grafana/provisioning`. You can modify these files to configure the
|
||||||
|
observability stack.
|
||||||
|
|
||||||
The dashboards are automatically provisioned when Grafana starts. They use the `${DS_PROMETHEUS}` datasource variable, so you need a Prometheus datasource configured in Grafana.
|
|
||||||
|
|
||||||
Key panels to monitor during optimization rollout:
|
|
||||||
|
|
||||||
1. **GET Latency by Reader Path** - Compare `codec_streaming` vs `legacy_duplex` latency
|
|
||||||
2. **Early-Stop Hit Rate** - Verify early-stop is triggering effectively
|
|
||||||
3. **Pipeline Failure Rate** - Detect any new failure modes introduced by optimizations
|
|
||||||
4. **Bitrot Verify Failures** - Ensure data integrity is maintained
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
### Prerequisites
|
|
||||||
|
|
||||||
- Docker
|
|
||||||
- Docker Compose
|
|
||||||
|
|
||||||
### Deploy
|
|
||||||
|
|
||||||
Run the following command to start the entire stack:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
### High Availability Tempo
|
|
||||||
|
|
||||||
The default `docker-compose.yml` is the single-node stack.
|
|
||||||
If you need the Kafka-backed HA Tempo configuration, start it with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
### Access Dashboards
|
|
||||||
|
|
||||||
| Service | URL | Credentials | Description |
|
|
||||||
| :------------- | :----------------------------------------------- | :---------------- | :----------------------------- |
|
|
||||||
| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | Main visualization hub. |
|
|
||||||
| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | Metric queries and status. |
|
|
||||||
| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | Secondary trace visualization. |
|
|
||||||
| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo status/metrics. |
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
### Data Persistence
|
|
||||||
|
|
||||||
Data is stored in the following Docker volumes:
|
|
||||||
|
|
||||||
- `prometheus-data`: Prometheus metrics
|
|
||||||
- `tempo-data`: Tempo traces (WAL and Blocks)
|
|
||||||
- `loki-data`: Loki logs (Chunks and Rules)
|
|
||||||
- `jaeger-data`: Jaeger traces (Badger DB)
|
|
||||||
|
|
||||||
To clear all data:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose down -v
|
|
||||||
```
|
|
||||||
|
|
||||||
### Customization
|
|
||||||
|
|
||||||
- **Prometheus**: Edit `prometheus.yml` to add scrape targets or alerting rules.
|
|
||||||
- **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
|
|
||||||
- **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
|
|
||||||
|
|
||||||
### Verifying RustFS Traces
|
|
||||||
|
|
||||||
When RustFS points `RUSTFS_OBS_ENDPOINT` at this stack, treat the value as the
|
|
||||||
OTLP/HTTP base URL, for example:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
|
|
||||||
```
|
|
||||||
|
|
||||||
RustFS automatically expands that base URL to:
|
|
||||||
|
|
||||||
- `/v1/traces`
|
|
||||||
- `/v1/metrics`
|
|
||||||
- `/v1/logs`
|
|
||||||
|
|
||||||
Important behavior notes:
|
|
||||||
|
|
||||||
- Logs and metrics usually appear during startup, so seeing those two signals
|
|
||||||
first is expected.
|
|
||||||
- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
|
|
||||||
those attributes as structured metadata, and the Collector also mirrors the
|
|
||||||
common troubleshooting fields into the log line so simple line filters can
|
|
||||||
find them.
|
|
||||||
- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
|
|
||||||
startup, because request-path spans are created on demand.
|
|
||||||
- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
|
|
||||||
many nested `debug` spans. If Tempo or Jaeger looks sparse, retry with
|
|
||||||
`RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting collector or Tempo issues.
|
|
||||||
|
|
||||||
Minimal validation flow:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. Start this observability stack.
|
|
||||||
docker compose up -d
|
|
||||||
|
|
||||||
# 2. Start RustFS with OTLP/HTTP export and richer span visibility.
|
|
||||||
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
|
|
||||||
export RUSTFS_OBS_LOGGER_LEVEL=debug
|
|
||||||
|
|
||||||
# 3. Generate real request traffic.
|
|
||||||
curl -I http://127.0.0.1:9000/health
|
|
||||||
curl -I http://127.0.0.1:9000/health/ready
|
|
||||||
|
|
||||||
# 4. Inspect Grafana or Jaeger.
|
|
||||||
# Grafana: http://localhost:3000
|
|
||||||
# Jaeger: http://localhost:16686
|
|
||||||
```
|
|
||||||
|
|
||||||
For a structured RustFS log such as an inter-node RPC authentication failure,
|
|
||||||
the Loki line now includes fields such as `event`, `component`, `subsystem`,
|
|
||||||
`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
|
|
||||||
LogQL checks:
|
|
||||||
|
|
||||||
```logql
|
|
||||||
{service_name="RustFS"} |= "RPC signature verification failed"
|
|
||||||
{service_name="RustFS"} |= "failure_reason="
|
|
||||||
{service_name="RustFS"} | failure_reason != ""
|
|
||||||
```
|
|
||||||
|
|
||||||
If logs and metrics are present but traces are sparse, the most common cause is
|
|
||||||
"no real request traffic yet" or "`info` level filtered nested spans", not an
|
|
||||||
OTLP routing failure.
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
- **Service Health**: Check the health of services using `docker compose ps`.
|
|
||||||
- **Logs**: View logs for a specific service using `docker compose logs -f <service_name>`.
|
|
||||||
- **Otel Collector**: Check `http://localhost:13133` for health status and `http://localhost:1888/debug/pprof/` for profiling.
|
|
||||||
|
|||||||
@@ -1,211 +1,27 @@
|
|||||||
# RustFS 可观测性技术栈
|
## 部署可观测性系统
|
||||||
|
|
||||||
本目录包含 RustFS 的全面可观测性技术栈，旨在提供对应用程序性能、日志和追踪的深入洞察。
|
OpenTelemetry Collector 提供了一个厂商中立的遥测数据处理方案，用于接收、处理和导出遥测数据。它消除了为支持多种开源可观测性数据格式（如
|
||||||
|
Jaeger、Prometheus 等）而需要运行和维护多个代理/收集器的必要性。
|
||||||
|
|
||||||
## 组件
|
### 快速部署
|
||||||
|
|
||||||
该技术栈由以下一流的开源组件组成：
|
1. 进入 `.docker/observability` 目录
|
||||||
|
2. 执行以下命令启动服务：
|
||||||
- **Prometheus** (v2.53.1): 行业标准的指标收集和告警工具。
|
|
||||||
- **Grafana** (v11.1.0): 领先的可观测性可视化平台。
|
|
||||||
- **Loki** (v3.1.0): 水平可扩展、高可用、多租户的日志聚合系统。
|
|
||||||
- **Tempo** (v2.5.0): 高吞吐量、最小依赖的分布式追踪后端。
|
|
||||||
- **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
|
|
||||||
- **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
|
|
||||||
|
|
||||||
默认情况下，这套技术栈使用 Tempo 单二进制模式，不依赖 Kafka/Redpanda。
|
|
||||||
如果需要基于 Kafka 的 HA Tempo 路径，请使用 `docker-compose-example-for-rustfs.yml` 配合 `docker-compose-tempo-ha-override.yml`。
|
|
||||||
|
|
||||||
## 架构
|
|
||||||
|
|
||||||
1. **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
|
|
||||||
2. **处理与导出**: Collector 处理数据（批处理、内存限制）并将其导出到相应的后端：
|
|
||||||
- **追踪** -> **Tempo** (主要) & **Jaeger** (辅助/可选)
|
|
||||||
- **指标** -> **Prometheus** (通过抓取 Collector 的导出器)
|
|
||||||
- **日志** -> **Loki**
|
|
||||||
3. **可视化**: **Grafana** 连接到所有后端（Prometheus, Tempo, Loki, Jaeger），提供统一的仪表盘体验。
|
|
||||||
|
|
||||||
## 特性
|
|
||||||
|
|
||||||
- **完全持久化**: 所有数据（指标、日志、追踪）都持久化到 Docker 卷，确保重启后无数据丢失。
|
|
||||||
- **关联性**: 在 Grafana 中实现指标、日志和追踪之间的无缝导航。
|
|
||||||
- 从指标峰值跳转到相关追踪。
|
|
||||||
- 从追踪跳转到相关日志。
|
|
||||||
- **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
|
|
||||||
- **标准化协议**: 完全基于 OpenTelemetry 标准构建。
|
|
||||||
|
|
||||||
## GET 性能优化仪表盘
|
|
||||||
|
|
||||||
包含三个预构建的 Grafana 仪表盘，用于监控 RustFS GET 性能优化发布：
|
|
||||||
|
|
||||||
### 可用仪表盘
|
|
||||||
|
|
||||||
| 仪表盘 | 文件 | 描述 |
|
|
||||||
|--------|------|------|
|
|
||||||
| **GET 发布健康度** | `grafana-get-rollout-health.json` | 监控优化发布：按 reader path 的延迟、early-stop 命中率、codec streaming 使用率、pipeline 失败率 |
|
|
||||||
| **GET 数据完整性** | `grafana-get-data-integrity.json` | 监控数据安全：bitrot 校验失败、decode 错误、short read、shard 读取结果 |
|
|
||||||
| **GET 资源影响** | `grafana-get-resource-impact.json` | 监控资源使用：并发请求数、IO 队列利用率、disk permit 等待、RSS 趋势 |
|
|
||||||
| **对象数据缓存** | `grafana-object-data-cache.json` | 监控 GET body 缓存（`rustfs_object_data_cache_*`）：命中率、查找/规划/填充结果、填充耗时分位、命中 vs 填充吞吐、条目数/加权字节、在途填充、内存压力拒绝、失效、按尺寸档拆分 |
|
|
||||||
|
|
||||||
### Prometheus 告警规则
|
|
||||||
|
|
||||||
文件 `prometheus-rules/rustfs-get-optimization-alerts.yaml` 包含预配置的告警规则：
|
|
||||||
|
|
||||||
| 告警 | 级别 | 条件 |
|
|
||||||
|------|------|------|
|
|
||||||
| `GetP99Regression` | 严重 | GET p99 延迟 > 2x 基线，持续 10 分钟 |
|
|
||||||
| `PipelineFailureSpike` | 严重 | Pipeline 失败率 > 5x 基线，持续 5 分钟 |
|
|
||||||
| `BitrotMismatchSpike` | 严重 | Bitrot 不匹配率 > 3x 基线，持续 5 分钟 |
|
|
||||||
| `EarlyStopInsufficientQuorum` | 警告 | Early-stop quorum 不足率 > 0.1/s，持续 5 分钟 |
|
|
||||||
| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线，持续 10 分钟 |
|
|
||||||
| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%，持续 5 分钟 |
|
|
||||||
|
|
||||||
文件 `prometheus-rules/rustfs-kms-alerts.yml` 包含 KMS 后端操作指标的告警规则。阈值为保守默认值，待 staging 基线校准；响应流程见 `docs/operations/kms-observability-runbook.md`，配套仪表盘为 `deploy/observability/grafana/rustfs-kms-observability.json`。
|
|
||||||
|
|
||||||
| 告警 | 级别 | 条件 |
|
|
||||||
|------|------|------|
|
|
||||||
| `KmsBackendFatalErrors` | 严重 | fatal（不可重试）尝试失败 > 0，持续 5 分钟 |
|
|
||||||
| `KmsBackendHighErrorRate` | 严重 | 非 success 操作占比 > 5%，持续 10 分钟（含流量下限保护） |
|
|
||||||
| `KmsBackendP99LatencyHigh` | 警告 | 操作 p99 耗时（含重试）> 2s，持续 10 分钟 |
|
|
||||||
| `KmsBackendAttemptFailureSpike` | 警告 | 尝试失败率 > 0.5/s，持续 10 分钟 |
|
|
||||||
| `KmsBackendRetryBudgetExhausted` | 警告 | budget_exhausted / deadline_exceeded 结果 > 0.05/s，持续 10 分钟 |
|
|
||||||
|
|
||||||
### 启用告警规则
|
|
||||||
|
|
||||||
在 Prometheus 配置中添加告警规则文件：
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# prometheus.yml
|
|
||||||
rule_files:
|
|
||||||
- "/etc/prometheus/rules/*.yml"
|
|
||||||
|
|
||||||
# 或在 docker-compose.yml 中挂载文件：
|
|
||||||
# volumes:
|
|
||||||
# - ./prometheus-rules:/etc/prometheus/rules
|
|
||||||
```
|
|
||||||
|
|
||||||
### 仪表盘使用
|
|
||||||
|
|
||||||
仪表盘在 Grafana 启动时自动预置。它们使用 `${DS_PROMETHEUS}` 数据源变量，因此需要在 Grafana 中配置 Prometheus 数据源。
|
|
||||||
|
|
||||||
优化发布期间需要关注的关键面板：
|
|
||||||
|
|
||||||
1. **GET 延迟按 Reader Path** - 对比 `codec_streaming` vs `legacy_duplex` 延迟
|
|
||||||
2. **Early-Stop 命中率** - 验证 early-stop 是否有效触发
|
|
||||||
3. **Pipeline 失败率** - 检测优化引入的新故障模式
|
|
||||||
4. **Bitrot 校验失败** - 确保数据完整性
|
|
||||||
|
|
||||||
## 快速开始
|
|
||||||
|
|
||||||
### 前置条件
|
|
||||||
|
|
||||||
- Docker
|
|
||||||
- Docker Compose
|
|
||||||
|
|
||||||
### 部署
|
|
||||||
|
|
||||||
运行以下命令启动整个技术栈：
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker compose up -d
|
docker compose -f docker-compose.yml up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
### Tempo 高可用模式
|
### 访问监控面板
|
||||||
|
|
||||||
默认的 `docker-compose.yml` 对应单机栈。
|
服务启动后，可通过以下地址访问各个监控面板：
|
||||||
如果需要基于 Kafka 的 HA Tempo 配置，请使用：
|
|
||||||
|
|
||||||
```bash
|
- Grafana: `http://localhost:3000` (默认账号/密码：`admin`/`admin`)
|
||||||
docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
|
- Jaeger: `http://localhost:16686`
|
||||||
|
- Prometheus: `http://localhost:9090`
|
||||||
|
|
||||||
|
## 配置可观测性
|
||||||
|
|
||||||
|
```shell
|
||||||
|
export RUSTFS_OBS_ENDPOINT="http://localhost:4317" # OpenTelemetry Collector 地址
|
||||||
```
|
```
|
||||||
|
|
||||||
### 访问仪表盘
|
|
||||||
|
|
||||||
| 服务 | URL | 凭据 | 描述 |
|
|
||||||
| :--- | :--- | :--- | :--- |
|
|
||||||
| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | 主要可视化中心。 |
|
|
||||||
| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | 指标查询和状态。 |
|
|
||||||
| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | 辅助追踪可视化。 |
|
|
||||||
| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo 状态/指标。 |
|
|
||||||
|
|
||||||
## 配置
|
|
||||||
|
|
||||||
### 数据持久化
|
|
||||||
|
|
||||||
数据存储在以下 Docker 卷中：
|
|
||||||
|
|
||||||
- `prometheus-data`: Prometheus 指标
|
|
||||||
- `tempo-data`: Tempo 追踪 (WAL 和 Blocks)
|
|
||||||
- `loki-data`: Loki 日志 (Chunks 和 Rules)
|
|
||||||
- `jaeger-data`: Jaeger 追踪 (Badger DB)
|
|
||||||
|
|
||||||
要清除所有数据：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose down -v
|
|
||||||
```
|
|
||||||
|
|
||||||
### 自定义
|
|
||||||
|
|
||||||
- **Prometheus**: 编辑 `prometheus.yml` 以添加抓取目标或告警规则。
|
|
||||||
- **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
|
|
||||||
- **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
|
|
||||||
|
|
||||||
### 验证 RustFS Trace
|
|
||||||
|
|
||||||
当 RustFS 将 `RUSTFS_OBS_ENDPOINT` 指向这套技术栈时，应将该值视为
|
|
||||||
OTLP/HTTP 的基础 URL，例如：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
|
|
||||||
```
|
|
||||||
|
|
||||||
RustFS 会自动在该基础 URL 后补全：
|
|
||||||
|
|
||||||
- `/v1/traces`
|
|
||||||
- `/v1/metrics`
|
|
||||||
- `/v1/logs`
|
|
||||||
|
|
||||||
需要注意：
|
|
||||||
|
|
||||||
- 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
|
|
||||||
- OpenTelemetry bridge 会把 `tracing` 字段作为日志 attributes 发送。Loki 会将这些 attributes 存为 structured metadata，同时 Collector 会把常用排障字段镜像进日志行，方便用简单的行内容过滤直接查到。
|
|
||||||
- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
|
|
||||||
- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
|
|
||||||
如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
|
|
||||||
|
|
||||||
最小验证流程：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 1. 启动本目录下的可观测性技术栈。
|
|
||||||
docker compose up -d
|
|
||||||
|
|
||||||
# 2. 以 OTLP/HTTP 导出方式启动 RustFS，并提高 span 可见性。
|
|
||||||
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
|
|
||||||
export RUSTFS_OBS_LOGGER_LEVEL=debug
|
|
||||||
|
|
||||||
# 3. 产生真实请求流量。
|
|
||||||
curl -I http://127.0.0.1:9000/health
|
|
||||||
curl -I http://127.0.0.1:9000/health/ready
|
|
||||||
|
|
||||||
# 4. 到 Grafana 或 Jaeger 中检查。
|
|
||||||
# Grafana: http://localhost:3000
|
|
||||||
# Jaeger: http://localhost:16686
|
|
||||||
```
|
|
||||||
|
|
||||||
对于 RustFS 结构化日志，例如节点间 RPC 鉴权失败，Loki 日志行现在会包含 `event`、`component`、`subsystem`、`failure_reason`、`rpc_service`、`rpc_method`、`expected_audience` 等字段。常用 LogQL 检查：
|
|
||||||
|
|
||||||
```logql
|
|
||||||
{service_name="RustFS"} |= "RPC signature verification failed"
|
|
||||||
{service_name="RustFS"} |= "failure_reason="
|
|
||||||
{service_name="RustFS"} | failure_reason != ""
|
|
||||||
```
|
|
||||||
|
|
||||||
如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
|
|
||||||
“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
|
|
||||||
|
|
||||||
## 故障排除
|
|
||||||
|
|
||||||
- **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
|
|
||||||
- **日志**: 使用 `docker compose logs -f <service_name>` 查看特定服务的日志。
|
|
||||||
- **Otel Collector**: 检查 `http://localhost:13133` 获取健康状态，检查 `http://localhost:1888/debug/pprof/` 进行性能分析。
|
|
||||||
|
|||||||
@@ -1,268 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
services:
|
|
||||||
rustfs:
|
|
||||||
security_opt:
|
|
||||||
- "no-new-privileges:true"
|
|
||||||
image: rustfs/rustfs:latest
|
|
||||||
container_name: rustfs-server
|
|
||||||
ports:
|
|
||||||
- "9000:9000" # S3 API port
|
|
||||||
- "9001:9001" # Console port
|
|
||||||
environment:
|
|
||||||
- RUSTFS_VOLUMES=/data/rustfs
|
|
||||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
|
||||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
|
||||||
- RUSTFS_CORS_ALLOWED_ORIGINS=*
|
|
||||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
|
||||||
- RUSTFS_ACCESS_KEY=rustfsadmin
|
|
||||||
- RUSTFS_SECRET_KEY=rustfsadmin
|
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=info
|
|
||||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
|
||||||
- RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
|
|
||||||
volumes:
|
|
||||||
- rustfs-data:/data/rustfs
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test:
|
|
||||||
[
|
|
||||||
"CMD",
|
|
||||||
"sh",
|
|
||||||
"-c",
|
|
||||||
"curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
|
|
||||||
]
|
|
||||||
interval: 30s
|
|
||||||
timeout: 10s
|
|
||||||
retries: 3
|
|
||||||
start_period: 40s
|
|
||||||
depends_on:
|
|
||||||
otel-collector:
|
|
||||||
condition: service_started
|
|
||||||
|
|
||||||
rustfs-init:
|
|
||||||
image: alpine
|
|
||||||
container_name: rustfs-init
|
|
||||||
volumes:
|
|
||||||
- rustfs-data:/data
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
command: >
|
|
||||||
sh -c "
|
|
||||||
chown -R 10001:10001 /data &&
|
|
||||||
echo 'Volume Permissions fixed' &&
|
|
||||||
exit 0
|
|
||||||
"
|
|
||||||
restart: no
|
|
||||||
|
|
||||||
# --- Tracing ---
|
|
||||||
|
|
||||||
tempo:
|
|
||||||
image: grafana/tempo:latest
|
|
||||||
container_name: tempo
|
|
||||||
command: [ "-config.file=/etc/tempo.yaml" ]
|
|
||||||
volumes:
|
|
||||||
- ./tempo.yaml:/etc/tempo.yaml:ro
|
|
||||||
- tempo-data:/var/tempo
|
|
||||||
ports:
|
|
||||||
- "3200:3200" # tempo
|
|
||||||
- "4317" # otlp grpc
|
|
||||||
- "4318" # otlp http
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "/tempo", "-version" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
start_period: 15s
|
|
||||||
|
|
||||||
redpanda:
|
|
||||||
image: redpandadata/redpanda:latest # for tempo ingest
|
|
||||||
container_name: redpanda
|
|
||||||
ports:
|
|
||||||
- "9092:9092"
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
command: >
|
|
||||||
redpanda start --overprovisioned
|
|
||||||
--mode=dev-container
|
|
||||||
--kafka-addr=PLAINTEXT://0.0.0.0:9092
|
|
||||||
--advertise-kafka-addr=PLAINTEXT://redpanda:9092
|
|
||||||
|
|
||||||
jaeger:
|
|
||||||
image: jaegertracing/jaeger:latest
|
|
||||||
container_name: jaeger
|
|
||||||
environment:
|
|
||||||
- SPAN_STORAGE_TYPE=badger
|
|
||||||
- BADGER_EPHEMERAL=false
|
|
||||||
- BADGER_DIRECTORY_VALUE=/badger/data
|
|
||||||
- BADGER_DIRECTORY_KEY=/badger/key
|
|
||||||
- COLLECTOR_OTLP_ENABLED=true
|
|
||||||
volumes:
|
|
||||||
- ./jaeger.yaml:/etc/jaeger/config.yml
|
|
||||||
- jaeger-data:/badger
|
|
||||||
ports:
|
|
||||||
- "16686:16686" # Web UI
|
|
||||||
- "14269:14269" # Admin/Metrics
|
|
||||||
- "4317" # otlp grpc
|
|
||||||
- "4318" # otlp http
|
|
||||||
command: [ "--config", "/etc/jaeger/config.yml" ]
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
start_period: 15s
|
|
||||||
|
|
||||||
# --- Metrics ---
|
|
||||||
|
|
||||||
prometheus:
|
|
||||||
image: prom/prometheus:latest
|
|
||||||
container_name: prometheus
|
|
||||||
volumes:
|
|
||||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
|
||||||
- prometheus-data:/prometheus
|
|
||||||
ports:
|
|
||||||
- "9090:9090"
|
|
||||||
command:
|
|
||||||
- "--config.file=/etc/prometheus/prometheus.yml"
|
|
||||||
- "--web.enable-otlp-receiver" # Enable OTLP
|
|
||||||
- "--web.enable-remote-write-receiver" # Enable remote write
|
|
||||||
- "--enable-feature=promql-experimental-functions" # Enable info()
|
|
||||||
- "--storage.tsdb.retention.time=30d"
|
|
||||||
restart: unless-stopped
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
|
|
||||||
# --- Logging ---
|
|
||||||
|
|
||||||
loki:
|
|
||||||
image: grafana/loki:latest
|
|
||||||
container_name: loki
|
|
||||||
volumes:
|
|
||||||
- ./loki.yaml:/etc/loki/loki.yaml:ro
|
|
||||||
- loki-data:/loki
|
|
||||||
ports:
|
|
||||||
- "3100:3100"
|
|
||||||
command: -config.file=/etc/loki/loki.yaml
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
|
|
||||||
interval: 15s
|
|
||||||
timeout: 10s
|
|
||||||
retries: 5
|
|
||||||
start_period: 60s
|
|
||||||
|
|
||||||
# --- Collection ---
|
|
||||||
|
|
||||||
otel-collector:
|
|
||||||
image: otel/opentelemetry-collector-contrib:latest
|
|
||||||
volumes:
|
|
||||||
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
|
|
||||||
ports:
|
|
||||||
- "1888:1888" # pprof
|
|
||||||
- "8888:8888" # Prometheus metrics for Collector
|
|
||||||
- "8889:8889" # Prometheus metrics for application indicators
|
|
||||||
- "13133:13133" # health check
|
|
||||||
- "4317:4317" # OTLP gRPC
|
|
||||||
- "4318:4318" # OTLP HTTP
|
|
||||||
- "55679:55679" # zpages
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
depends_on:
|
|
||||||
- tempo
|
|
||||||
- jaeger
|
|
||||||
- prometheus
|
|
||||||
- loki
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "/otelcol-contrib", "--version" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
|
|
||||||
# --- Profiles ---
|
|
||||||
|
|
||||||
pyroscope:
|
|
||||||
image: grafana/pyroscope:latest
|
|
||||||
container_name: pyroscope
|
|
||||||
ports:
|
|
||||||
- "4040:4040"
|
|
||||||
command:
|
|
||||||
- -self-profiling.disable-push=true
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
|
|
||||||
# --- Visualization ---
|
|
||||||
|
|
||||||
grafana:
|
|
||||||
image: grafana/grafana:latest
|
|
||||||
container_name: grafana
|
|
||||||
ports:
|
|
||||||
- "3000:3000"
|
|
||||||
environment:
|
|
||||||
- GF_SECURITY_ADMIN_PASSWORD=admin
|
|
||||||
- GF_SECURITY_ADMIN_USER=admin
|
|
||||||
volumes:
|
|
||||||
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
|
||||||
- ./grafana/dashboards:/etc/grafana/dashboards:ro
|
|
||||||
- grafana-data:/var/lib/grafana
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
depends_on:
|
|
||||||
- prometheus
|
|
||||||
- tempo
|
|
||||||
- loki
|
|
||||||
healthcheck:
|
|
||||||
test:
|
|
||||||
[ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
rustfs-data:
|
|
||||||
tempo-data:
|
|
||||||
jaeger-data:
|
|
||||||
prometheus-data:
|
|
||||||
loki-data:
|
|
||||||
grafana-data:
|
|
||||||
|
|
||||||
networks:
|
|
||||||
otel-network:
|
|
||||||
driver: bridge
|
|
||||||
name: "network_otel"
|
|
||||||
ipam:
|
|
||||||
config:
|
|
||||||
- subnet: 172.28.0.0/16
|
|
||||||
driver_opts:
|
|
||||||
com.docker.network.enable_ipv6: "true"
|
|
||||||
@@ -1,62 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
# Docker Compose override file for High Availability Tempo setup
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# docker-compose -f docker-compose-example-for-rustfs.yml \
|
|
||||||
# -f docker-compose-tempo-ha-override.yml up
|
|
||||||
|
|
||||||
services:
|
|
||||||
# Override Tempo to use high-availability configuration
|
|
||||||
tempo:
|
|
||||||
volumes:
|
|
||||||
- ./tempo-ha.yaml:/etc/tempo.yaml:ro
|
|
||||||
- tempo-data:/var/tempo
|
|
||||||
ports:
|
|
||||||
- "3200:3200" # Tempo HTTP
|
|
||||||
- "4317:4317" # OTLP gRPC
|
|
||||||
- "4318:4318" # OTLP HTTP
|
|
||||||
- "7946:7946" # Memberlist
|
|
||||||
- "14250:14250" # Jaeger gRPC
|
|
||||||
- "14268:14268" # Jaeger Thrift HTTP
|
|
||||||
- "9411:9411" # Zipkin
|
|
||||||
environment:
|
|
||||||
- TEMPO_MEMBERLIST_BIND_PORT=7946
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 5
|
|
||||||
start_period: 30s
|
|
||||||
depends_on:
|
|
||||||
- redpanda
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
tempo-data:
|
|
||||||
driver: local
|
|
||||||
driver_opts:
|
|
||||||
type: tmpfs
|
|
||||||
device: tmpfs
|
|
||||||
o: "size=4g" # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
|
|
||||||
|
|
||||||
# Network configuration remains the same
|
|
||||||
# networks:
|
|
||||||
# otel-network:
|
|
||||||
# driver: bridge
|
|
||||||
# name: "network_otel"
|
|
||||||
# ipam:
|
|
||||||
# config:
|
|
||||||
# - subnet: 172.28.0.0/16
|
|
||||||
|
|
||||||
@@ -14,198 +14,93 @@
|
|||||||
|
|
||||||
services:
|
services:
|
||||||
|
|
||||||
# --- Tracing ---
|
tempo-init:
|
||||||
|
image: busybox:latest
|
||||||
|
command: ["sh", "-c", "chown -R 10001:10001 /var/tempo"]
|
||||||
|
volumes:
|
||||||
|
- ./tempo-data:/var/tempo
|
||||||
|
user: root
|
||||||
|
networks:
|
||||||
|
- otel-network
|
||||||
|
restart: "no"
|
||||||
|
|
||||||
tempo:
|
tempo:
|
||||||
image: grafana/tempo:2.10.5
|
image: grafana/tempo:latest
|
||||||
container_name: tempo
|
user: "10001" # The container must be started with root to execute chown in the script
|
||||||
command: [ "-config.file=/etc/tempo.yaml" ]
|
command: [ "-config.file=/etc/tempo.yaml" ] # This is passed as a parameter to the entry point script
|
||||||
volumes:
|
volumes:
|
||||||
- ./tempo.yaml:/etc/tempo.yaml:ro
|
- ./tempo.yaml:/etc/tempo.yaml:ro
|
||||||
- tempo-data:/var/tempo
|
- ./tempo-data:/var/tempo
|
||||||
ports:
|
ports:
|
||||||
- "3200:3200" # tempo
|
- "3200:3200" # tempo
|
||||||
- "4317" # otlp grpc
|
- "24317:4317" # otlp grpc
|
||||||
- "4318" # otlp http
|
|
||||||
- "7946" # memberlist
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "/tempo", "-version" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
start_period: 15s
|
|
||||||
|
|
||||||
vulture:
|
|
||||||
image: grafana/tempo-vulture:latest
|
|
||||||
restart: always
|
|
||||||
command:
|
|
||||||
[
|
|
||||||
"-prometheus-listen-address=:8080",
|
|
||||||
"-tempo-query-url=http://tempo:3200",
|
|
||||||
"-tempo-push-url=http://tempo:4317",
|
|
||||||
]
|
|
||||||
depends_on:
|
|
||||||
- tempo
|
|
||||||
|
|
||||||
jaeger:
|
|
||||||
image: jaegertracing/jaeger:latest
|
|
||||||
container_name: jaeger
|
|
||||||
environment:
|
|
||||||
- SPAN_STORAGE_TYPE=badger
|
|
||||||
- BADGER_EPHEMERAL=false
|
|
||||||
- BADGER_DIRECTORY_VALUE=/badger/data
|
|
||||||
- BADGER_DIRECTORY_KEY=/badger/key
|
|
||||||
- COLLECTOR_OTLP_ENABLED=true
|
|
||||||
volumes:
|
|
||||||
- ./jaeger.yaml:/etc/jaeger/config.yml
|
|
||||||
- jaeger-data:/badger
|
|
||||||
ports:
|
|
||||||
- "16686:16686" # Web UI
|
|
||||||
- "18888:8888" # Metrics
|
|
||||||
- "4317" # otlp grpc
|
|
||||||
- "4318" # otlp http
|
|
||||||
command: [ "--config", "/etc/jaeger/config.yml" ]
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:8888/metrics" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
start_period: 15s
|
|
||||||
|
|
||||||
# --- Metrics ---
|
|
||||||
|
|
||||||
prometheus:
|
|
||||||
image: prom/prometheus:latest
|
|
||||||
container_name: prometheus
|
|
||||||
volumes:
|
|
||||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
|
||||||
- ./prometheus-rules:/etc/prometheus/rules:ro
|
|
||||||
- prometheus-data:/prometheus
|
|
||||||
ports:
|
|
||||||
- "9090:9090"
|
|
||||||
command:
|
|
||||||
- "--config.file=/etc/prometheus/prometheus.yml"
|
|
||||||
- "--web.enable-otlp-receiver" # Enable OTLP
|
|
||||||
- "--web.enable-remote-write-receiver" # Enable remote write
|
|
||||||
- "--enable-feature=promql-experimental-functions" # Enable info()
|
|
||||||
- "--storage.tsdb.retention.time=30d"
|
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
networks:
|
networks:
|
||||||
- otel-network
|
- otel-network
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
|
|
||||||
# --- Logging ---
|
|
||||||
|
|
||||||
loki:
|
|
||||||
image: grafana/loki:latest
|
|
||||||
container_name: loki
|
|
||||||
volumes:
|
|
||||||
- ./loki.yaml:/etc/loki/loki.yaml:ro
|
|
||||||
- loki-data:/loki
|
|
||||||
ports:
|
|
||||||
- "3100:3100"
|
|
||||||
command: -config.file=/etc/loki/loki.yaml
|
|
||||||
networks:
|
|
||||||
- otel-network
|
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "/usr/bin/loki", "--version" ]
|
|
||||||
interval: 15s
|
|
||||||
timeout: 10s
|
|
||||||
retries: 5
|
|
||||||
start_period: 60s
|
|
||||||
|
|
||||||
# --- Collection ---
|
|
||||||
|
|
||||||
otel-collector:
|
otel-collector:
|
||||||
image: otel/opentelemetry-collector-contrib:latest
|
image: otel/opentelemetry-collector-contrib:0.129.1
|
||||||
|
environment:
|
||||||
|
- TZ=Asia/Shanghai
|
||||||
volumes:
|
volumes:
|
||||||
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
|
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
|
||||||
ports:
|
ports:
|
||||||
- "1888:1888" # pprof
|
- "1888:1888"
|
||||||
- "8888:8888" # Prometheus metrics for Collector
|
- "8888:8888"
|
||||||
- "8889:8889" # Prometheus metrics for application indicators
|
- "8889:8889"
|
||||||
- "13133:13133" # health check
|
- "13133:13133"
|
||||||
- "4317:4317" # OTLP gRPC
|
- "4317:4317"
|
||||||
- "4318:4318" # OTLP HTTP
|
- "4318:4318"
|
||||||
- "55679:55679" # zpages
|
- "55679:55679"
|
||||||
networks:
|
networks:
|
||||||
- otel-network
|
- otel-network
|
||||||
restart: unless-stopped
|
jaeger:
|
||||||
depends_on:
|
image: jaegertracing/jaeger:2.8.0
|
||||||
- tempo
|
environment:
|
||||||
- jaeger
|
- TZ=Asia/Shanghai
|
||||||
- prometheus
|
|
||||||
- loki
|
|
||||||
healthcheck:
|
|
||||||
test: [ "CMD", "/otelcol-contrib", "--version" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
|
|
||||||
# --- Profiles ---
|
|
||||||
|
|
||||||
pyroscope:
|
|
||||||
image: grafana/pyroscope:latest
|
|
||||||
container_name: pyroscope
|
|
||||||
ports:
|
ports:
|
||||||
- "4040:4040"
|
- "16686:16686"
|
||||||
command:
|
- "14317:4317"
|
||||||
- -self-profiling.disable-push=true
|
- "14318:4318"
|
||||||
|
networks:
|
||||||
|
- otel-network
|
||||||
|
prometheus:
|
||||||
|
image: prom/prometheus:v3.4.2
|
||||||
|
environment:
|
||||||
|
- TZ=Asia/Shanghai
|
||||||
|
volumes:
|
||||||
|
- ./prometheus.yml:/etc/prometheus/prometheus.yml
|
||||||
|
ports:
|
||||||
|
- "9090:9090"
|
||||||
|
networks:
|
||||||
|
- otel-network
|
||||||
|
loki:
|
||||||
|
image: grafana/loki:3.5.1
|
||||||
|
environment:
|
||||||
|
- TZ=Asia/Shanghai
|
||||||
|
volumes:
|
||||||
|
- ./loki-config.yaml:/etc/loki/local-config.yaml
|
||||||
|
ports:
|
||||||
|
- "3100:3100"
|
||||||
|
command: -config.file=/etc/loki/local-config.yaml
|
||||||
networks:
|
networks:
|
||||||
- otel-network
|
- otel-network
|
||||||
restart: unless-stopped
|
|
||||||
|
|
||||||
# --- Visualization ---
|
|
||||||
|
|
||||||
grafana:
|
grafana:
|
||||||
image: grafana/grafana:latest
|
image: grafana/grafana:12.0.2
|
||||||
container_name: grafana
|
|
||||||
ports:
|
ports:
|
||||||
- "3000:3000"
|
- "3000:3000" # Web UI
|
||||||
|
volumes:
|
||||||
|
- ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
|
||||||
environment:
|
environment:
|
||||||
- GF_SECURITY_ADMIN_PASSWORD=admin
|
- GF_SECURITY_ADMIN_PASSWORD=admin
|
||||||
- GF_SECURITY_ADMIN_USER=admin
|
- TZ=Asia/Shanghai
|
||||||
volumes:
|
|
||||||
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
|
||||||
- ./grafana/dashboards:/etc/grafana/dashboards:ro
|
|
||||||
- grafana-data:/var/lib/grafana
|
|
||||||
networks:
|
networks:
|
||||||
- otel-network
|
- otel-network
|
||||||
restart: unless-stopped
|
|
||||||
depends_on:
|
|
||||||
- prometheus
|
|
||||||
- tempo
|
|
||||||
- loki
|
|
||||||
healthcheck:
|
|
||||||
test:
|
|
||||||
[ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
tempo-data:
|
|
||||||
jaeger-data:
|
|
||||||
prometheus-data:
|
|
||||||
loki-data:
|
|
||||||
grafana-data:
|
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
otel-network:
|
otel-network:
|
||||||
driver: bridge
|
driver: bridge
|
||||||
name: "network_otel"
|
name: "network_otel_config"
|
||||||
ipam:
|
|
||||||
config:
|
|
||||||
- subnet: 172.28.0.0/16
|
|
||||||
driver_opts:
|
driver_opts:
|
||||||
com.docker.network.enable_ipv6: "true"
|
com.docker.network.enable_ipv6: "true"
|
||||||
|
|||||||
@@ -0,0 +1,32 @@
|
|||||||
|
apiVersion: 1
|
||||||
|
|
||||||
|
datasources:
|
||||||
|
- name: Prometheus
|
||||||
|
type: prometheus
|
||||||
|
uid: prometheus
|
||||||
|
access: proxy
|
||||||
|
orgId: 1
|
||||||
|
url: http://prometheus:9090
|
||||||
|
basicAuth: false
|
||||||
|
isDefault: false
|
||||||
|
version: 1
|
||||||
|
editable: false
|
||||||
|
jsonData:
|
||||||
|
httpMethod: GET
|
||||||
|
- name: Tempo
|
||||||
|
type: tempo
|
||||||
|
access: proxy
|
||||||
|
orgId: 1
|
||||||
|
url: http://tempo:3200
|
||||||
|
basicAuth: false
|
||||||
|
isDefault: true
|
||||||
|
version: 1
|
||||||
|
editable: false
|
||||||
|
apiVersion: 1
|
||||||
|
uid: tempo
|
||||||
|
jsonData:
|
||||||
|
httpMethod: GET
|
||||||
|
serviceMap:
|
||||||
|
datasourceUid: prometheus
|
||||||
|
streamingEnabled:
|
||||||
|
search: true
|
||||||
@@ -1,500 +0,0 @@
|
|||||||
{
|
|
||||||
"annotations": {
|
|
||||||
"list": []
|
|
||||||
},
|
|
||||||
"editable": true,
|
|
||||||
"fiscalYearStartMonth": 0,
|
|
||||||
"graphTooltip": 1,
|
|
||||||
"id": null,
|
|
||||||
"links": [],
|
|
||||||
"liveNow": false,
|
|
||||||
"panels": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 20,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "line+area"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "transparent",
|
|
||||||
"value": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"color": "red",
|
|
||||||
"value": 1
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"id": 1,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max", "sum"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Bitrot Verify Failures",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{stage=\"bitrot_verify\"}[$__rate_interval])) by (path, reason)",
|
|
||||||
"legendFormat": "bitrot_fail {{path}} / {{reason}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"bitrot_verify_failed\"}[$__rate_interval])) by (path, stage)",
|
|
||||||
"legendFormat": "verify_failed {{path}} / {{stage}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "s",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"id": 2,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Bitrot Verify Duration (p50 / p95 / p99)",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
|
|
||||||
"legendFormat": "p50",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
|
|
||||||
"legendFormat": "p95",
|
|
||||||
"refId": "B"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
|
|
||||||
"legendFormat": "p99",
|
|
||||||
"refId": "C"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 20,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "line+area"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "transparent",
|
|
||||||
"value": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"color": "red",
|
|
||||||
"value": 1
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 8
|
|
||||||
},
|
|
||||||
"id": 3,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max", "sum"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Decode Errors",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"decode_error\"}[$__rate_interval])) by (path, stage)",
|
|
||||||
"legendFormat": "decode_error {{path}} / {{stage}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 20,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "line+area"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "transparent",
|
|
||||||
"value": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"color": "red",
|
|
||||||
"value": 1
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 8
|
|
||||||
},
|
|
||||||
"id": 4,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max", "sum"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Short Reads",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"short_read\"}[$__rate_interval])) by (path, stage)",
|
|
||||||
"legendFormat": "short_read {{path}} / {{stage}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 16
|
|
||||||
},
|
|
||||||
"id": 5,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Shard Read Outcomes by Role",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_shard_read_total[$__rate_interval])) by (path, role, outcome)",
|
|
||||||
"legendFormat": "{{path}} {{role}}/{{outcome}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 16
|
|
||||||
},
|
|
||||||
"id": 6,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Request Result Status Rate",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_request_results_total[$__rate_interval])) by (status)",
|
|
||||||
"legendFormat": "{{status}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"refresh": "30s",
|
|
||||||
"schemaVersion": 38,
|
|
||||||
"style": "dark",
|
|
||||||
"tags": ["rustfs", "get-optimization", "data-integrity"],
|
|
||||||
"templating": {
|
|
||||||
"list": [
|
|
||||||
{
|
|
||||||
"current": {
|
|
||||||
"selected": false,
|
|
||||||
"text": "Prometheus",
|
|
||||||
"value": "Prometheus"
|
|
||||||
},
|
|
||||||
"hide": 0,
|
|
||||||
"includeAll": false,
|
|
||||||
"multi": false,
|
|
||||||
"name": "DS_PROMETHEUS",
|
|
||||||
"options": [],
|
|
||||||
"query": "prometheus",
|
|
||||||
"refresh": 1,
|
|
||||||
"regex": "",
|
|
||||||
"skipUrlSync": false,
|
|
||||||
"type": "datasource"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"time": {
|
|
||||||
"from": "now-1h",
|
|
||||||
"to": "now"
|
|
||||||
},
|
|
||||||
"timepicker": {},
|
|
||||||
"timezone": "",
|
|
||||||
"title": "RustFS GET Data Integrity",
|
|
||||||
"uid": "rustfs-get-data-integrity",
|
|
||||||
"version": 1
|
|
||||||
}
|
|
||||||
@@ -1,716 +0,0 @@
|
|||||||
{
|
|
||||||
"annotations": {
|
|
||||||
"list": []
|
|
||||||
},
|
|
||||||
"editable": true,
|
|
||||||
"fiscalYearStartMonth": 0,
|
|
||||||
"graphTooltip": 1,
|
|
||||||
"id": null,
|
|
||||||
"links": [],
|
|
||||||
"liveNow": false,
|
|
||||||
"panels": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"id": 1,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_strategy_total[$__rate_interval])) or vector(0)",
|
|
||||||
"legendFormat": "{{strategy}} / {{mode}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "GET Reader Setup Strategy Rate",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"id": 2,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_strategy_by_size_total[$__rate_interval])) or vector(0)",
|
|
||||||
"legendFormat": "{{path}} / {{strategy}} / {{size_bucket}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "GET Reader Setup Strategy by Size",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "none",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 8
|
|
||||||
},
|
|
||||||
"id": 3,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_count[$__rate_interval])), 1) or vector(0)",
|
|
||||||
"legendFormat": "scheduled {{strategy}} / {{mode}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_count[$__rate_interval])), 1) or vector(0)",
|
|
||||||
"legendFormat": "attempted {{strategy}} / {{mode}}",
|
|
||||||
"refId": "B"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_count[$__rate_interval])), 1) or vector(0)",
|
|
||||||
"legendFormat": "ready {{strategy}} / {{mode}}",
|
|
||||||
"refId": "C"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_count[$__rate_interval])), 1) or vector(0)",
|
|
||||||
"legendFormat": "deferred {{strategy}} / {{mode}}",
|
|
||||||
"refId": "D"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "GET Reader Setup Fanout Average",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "none",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 8
|
|
||||||
},
|
|
||||||
"id": 4,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_count[$__rate_interval])), 1) or vector(0)",
|
|
||||||
"legendFormat": "scheduled {{path}} / {{strategy}} / {{size_bucket}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_count[$__rate_interval])), 1) or vector(0)",
|
|
||||||
"legendFormat": "ready {{path}} / {{strategy}} / {{size_bucket}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "GET Reader Setup Fanout by Size",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "s",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 16
|
|
||||||
},
|
|
||||||
"id": 5,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.95, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
|
|
||||||
"legendFormat": "p95 {{path}} / {{stage}} / {{size_bucket}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.99, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
|
|
||||||
"legendFormat": "p99 {{path}} / {{stage}} / {{size_bucket}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "GET Hot Stage Latency by Size",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "s",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 16
|
|
||||||
},
|
|
||||||
"id": 6,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.95, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
|
|
||||||
"legendFormat": "p95 {{path}} / {{stage}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.99, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
|
|
||||||
"legendFormat": "p99 {{path}} / {{stage}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "GET Reader, Handler and Body Latency",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 24
|
|
||||||
},
|
|
||||||
"id": 7,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (subpath) (rate(rustfs_io_get_object_direct_memory_subpath_total[$__rate_interval])) or vector(0)",
|
|
||||||
"legendFormat": "{{subpath}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (subpath, size_bucket) (rate(rustfs_io_get_object_direct_memory_subpath_by_size_total[$__rate_interval])) or vector(0)",
|
|
||||||
"legendFormat": "{{subpath}} / {{size_bucket}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "GET Direct-Memory Subpath Rate",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 24
|
|
||||||
},
|
|
||||||
"id": 8,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_mmap_page_faults_total[$__rate_interval])) or vector(0)",
|
|
||||||
"legendFormat": "mmap {{path}} / {{stage}} / {{kind}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_direct_read_page_faults_total[$__rate_interval])) or vector(0)",
|
|
||||||
"legendFormat": "direct-read {{path}} / {{stage}} / {{kind}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "GET mmap vs Direct-Read Page Fault Rate",
|
|
||||||
"type": "timeseries"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"refresh": "30s",
|
|
||||||
"schemaVersion": 38,
|
|
||||||
"style": "dark",
|
|
||||||
"tags": [
|
|
||||||
"rustfs",
|
|
||||||
"get",
|
|
||||||
"performance",
|
|
||||||
"attribution"
|
|
||||||
],
|
|
||||||
"templating": {
|
|
||||||
"list": [
|
|
||||||
{
|
|
||||||
"current": {
|
|
||||||
"selected": false,
|
|
||||||
"text": "Prometheus",
|
|
||||||
"value": "Prometheus"
|
|
||||||
},
|
|
||||||
"hide": 0,
|
|
||||||
"includeAll": false,
|
|
||||||
"multi": false,
|
|
||||||
"name": "DS_PROMETHEUS",
|
|
||||||
"options": [],
|
|
||||||
"query": "prometheus",
|
|
||||||
"refresh": 1,
|
|
||||||
"regex": "",
|
|
||||||
"skipUrlSync": false,
|
|
||||||
"type": "datasource"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"time": {
|
|
||||||
"from": "now-1h",
|
|
||||||
"to": "now"
|
|
||||||
},
|
|
||||||
"timezone": "",
|
|
||||||
"title": "RustFS GET Performance Attribution",
|
|
||||||
"uid": "rustfs-get-performance-attribution",
|
|
||||||
"version": 1,
|
|
||||||
"weekStart": ""
|
|
||||||
}
|
|
||||||
@@ -1,525 +0,0 @@
|
|||||||
{
|
|
||||||
"annotations": {
|
|
||||||
"list": []
|
|
||||||
},
|
|
||||||
"editable": true,
|
|
||||||
"fiscalYearStartMonth": 0,
|
|
||||||
"graphTooltip": 1,
|
|
||||||
"id": null,
|
|
||||||
"links": [],
|
|
||||||
"liveNow": false,
|
|
||||||
"panels": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "none",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 20,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"id": 1,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Concurrent GET Requests",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "rustfs_io_get_object_concurrent_requests",
|
|
||||||
"legendFormat": "concurrent GETs",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "percent",
|
|
||||||
"min": 0,
|
|
||||||
"max": 100,
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 20,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "line+area"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "transparent",
|
|
||||||
"value": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"color": "yellow",
|
|
||||||
"value": 70
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"color": "red",
|
|
||||||
"value": 90
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"id": 2,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "IO Queue Utilization",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "rustfs_io_queue_utilization_percent",
|
|
||||||
"legendFormat": "queue utilization %",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "rustfs_io_queue_permits_in_use",
|
|
||||||
"legendFormat": "permits in use",
|
|
||||||
"refId": "B"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "rustfs_io_queue_permits_available",
|
|
||||||
"legendFormat": "permits available",
|
|
||||||
"refId": "C"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "s",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 8
|
|
||||||
},
|
|
||||||
"id": 3,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Disk Permit Wait Duration (p50 / p95 / p99)",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
|
|
||||||
"legendFormat": "p50",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
|
|
||||||
"legendFormat": "p95",
|
|
||||||
"refId": "B"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
|
|
||||||
"legendFormat": "p99",
|
|
||||||
"refId": "C"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "bytes",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 15,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 8
|
|
||||||
},
|
|
||||||
"id": 4,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["lastNotNull"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "RSS Trend (Process Memory)",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "process_resident_memory_bytes{job=~\"rustfs.*\"}",
|
|
||||||
"legendFormat": "RSS {{instance}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "rustfs_process_memory_bytes",
|
|
||||||
"legendFormat": "RustFS memory {{instance}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 16
|
|
||||||
},
|
|
||||||
"id": 5,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "IO Queue Operations by Priority",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_queue_operations[$__rate_interval])) by (operation, priority)",
|
|
||||||
"legendFormat": "{{operation}} / {{priority}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 15,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 16
|
|
||||||
},
|
|
||||||
"id": 6,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["sum"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "IO Queue Congestion Events",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_queue_congestion_total[$__rate_interval]))",
|
|
||||||
"legendFormat": "congestion events/s",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_timeout_total[$__rate_interval])) by (stage)",
|
|
||||||
"legendFormat": "timeout {{stage}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"refresh": "30s",
|
|
||||||
"schemaVersion": 38,
|
|
||||||
"style": "dark",
|
|
||||||
"tags": ["rustfs", "get-optimization", "resource-impact"],
|
|
||||||
"templating": {
|
|
||||||
"list": [
|
|
||||||
{
|
|
||||||
"current": {
|
|
||||||
"selected": false,
|
|
||||||
"text": "Prometheus",
|
|
||||||
"value": "Prometheus"
|
|
||||||
},
|
|
||||||
"hide": 0,
|
|
||||||
"includeAll": false,
|
|
||||||
"multi": false,
|
|
||||||
"name": "DS_PROMETHEUS",
|
|
||||||
"options": [],
|
|
||||||
"query": "prometheus",
|
|
||||||
"refresh": 1,
|
|
||||||
"regex": "",
|
|
||||||
"skipUrlSync": false,
|
|
||||||
"type": "datasource"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"time": {
|
|
||||||
"from": "now-1h",
|
|
||||||
"to": "now"
|
|
||||||
},
|
|
||||||
"timepicker": {},
|
|
||||||
"timezone": "",
|
|
||||||
"title": "RustFS GET Resource Impact",
|
|
||||||
"uid": "rustfs-get-resource-impact",
|
|
||||||
"version": 1
|
|
||||||
}
|
|
||||||
@@ -1,530 +0,0 @@
|
|||||||
{
|
|
||||||
"annotations": {
|
|
||||||
"list": []
|
|
||||||
},
|
|
||||||
"editable": true,
|
|
||||||
"fiscalYearStartMonth": 0,
|
|
||||||
"graphTooltip": 1,
|
|
||||||
"id": null,
|
|
||||||
"links": [],
|
|
||||||
"liveNow": false,
|
|
||||||
"panels": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "s",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"axisLabel": "",
|
|
||||||
"axisColorMode": "text",
|
|
||||||
"scaleDistribution": {
|
|
||||||
"type": "linear"
|
|
||||||
},
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"id": 1,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "GET Latency by Reader Path (p50 / p95 / p99)",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
|
|
||||||
"legendFormat": "p50 {{path}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
|
|
||||||
"legendFormat": "p95 {{path}}",
|
|
||||||
"refId": "B"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
|
|
||||||
"legendFormat": "p99 {{path}}",
|
|
||||||
"refId": "C"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "percentunit",
|
|
||||||
"min": 0,
|
|
||||||
"max": 1,
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 15,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"color": "yellow",
|
|
||||||
"value": 0.3
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"color": "red",
|
|
||||||
"value": 0.7
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"id": 2,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Early-Stop Hit Rate",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_metadata_early_stop_total{decision=\"hit\"}[$__rate_interval])) by (path) / clamp_min(sum(rate(rustfs_io_get_object_metadata_early_stop_total[$__rate_interval])) by (path), 1)",
|
|
||||||
"legendFormat": "hit rate {{path}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "percentunit",
|
|
||||||
"min": 0,
|
|
||||||
"max": 1,
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 15,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 8
|
|
||||||
},
|
|
||||||
"id": 3,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Codec Streaming Usage Rate",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])) by (decision) / clamp_min(sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])), 1)",
|
|
||||||
"legendFormat": "codec_streaming {{decision}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])) by (strategy) / clamp_min(sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])), 1)",
|
|
||||||
"legendFormat": "stream_strategy {{strategy}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "bars",
|
|
||||||
"fillOpacity": 80,
|
|
||||||
"lineWidth": 0,
|
|
||||||
"pointSize": 5,
|
|
||||||
"showPoints": "never",
|
|
||||||
"spanNulls": false,
|
|
||||||
"stacking": {
|
|
||||||
"mode": "normal",
|
|
||||||
"group": "A"
|
|
||||||
},
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"color": "red",
|
|
||||||
"value": 1
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 8
|
|
||||||
},
|
|
||||||
"id": 4,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["sum"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "Pipeline Failure Rate by Stage",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total[$__rate_interval])) by (stage, reason)",
|
|
||||||
"legendFormat": "{{stage}} / {{reason}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "reqps",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 16
|
|
||||||
},
|
|
||||||
"id": 5,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "GET Request Rate by Path",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_io_get_object_reader_path_total[$__rate_interval])) by (path)",
|
|
||||||
"legendFormat": "{{path}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 16
|
|
||||||
},
|
|
||||||
"id": 6,
|
|
||||||
"options": {
|
|
||||||
"reduceOptions": {
|
|
||||||
"values": false,
|
|
||||||
"calcs": ["lastNotNull"],
|
|
||||||
"fields": ""
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
},
|
|
||||||
"legend": {
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom",
|
|
||||||
"calcs": ["mean", "max"]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"title": "GET Total Latency (p50 / p95 / p99)",
|
|
||||||
"type": "timeseries",
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
|
|
||||||
"legendFormat": "p50",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
|
|
||||||
"legendFormat": "p95",
|
|
||||||
"refId": "B"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
|
|
||||||
"legendFormat": "p99",
|
|
||||||
"refId": "C"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"refresh": "30s",
|
|
||||||
"schemaVersion": 38,
|
|
||||||
"style": "dark",
|
|
||||||
"tags": ["rustfs", "get-optimization"],
|
|
||||||
"templating": {
|
|
||||||
"list": [
|
|
||||||
{
|
|
||||||
"current": {
|
|
||||||
"selected": false,
|
|
||||||
"text": "Prometheus",
|
|
||||||
"value": "Prometheus"
|
|
||||||
},
|
|
||||||
"hide": 0,
|
|
||||||
"includeAll": false,
|
|
||||||
"multi": false,
|
|
||||||
"name": "DS_PROMETHEUS",
|
|
||||||
"options": [],
|
|
||||||
"query": "prometheus",
|
|
||||||
"refresh": 1,
|
|
||||||
"regex": "",
|
|
||||||
"skipUrlSync": false,
|
|
||||||
"type": "datasource"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"time": {
|
|
||||||
"from": "now-1h",
|
|
||||||
"to": "now"
|
|
||||||
},
|
|
||||||
"timepicker": {},
|
|
||||||
"timezone": "",
|
|
||||||
"title": "RustFS GET Rollout Health",
|
|
||||||
"uid": "rustfs-get-rollout-health",
|
|
||||||
"version": 1
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,534 +0,0 @@
|
|||||||
{
|
|
||||||
"annotations": {
|
|
||||||
"list": []
|
|
||||||
},
|
|
||||||
"editable": true,
|
|
||||||
"fiscalYearStartMonth": 0,
|
|
||||||
"graphTooltip": 1,
|
|
||||||
"id": null,
|
|
||||||
"links": [],
|
|
||||||
"liveNow": false,
|
|
||||||
"panels": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"id": 1,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (path, eager_status, size_bucket, buffer_bucket, large_concurrency_tuning) (rate(rustfs_s3_put_object_diagnostics_total[$__rate_interval])) or vector(0)",
|
|
||||||
"legendFormat": "{{path}} / {{eager_status}} / {{size_bucket}} / {{buffer_bucket}} / large={{large_concurrency_tuning}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "PUT Diagnostics Decision Rate",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 0
|
|
||||||
},
|
|
||||||
"id": 2,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (path) (rate(rustfs_s3_put_object_path_total[$__rate_interval])) or vector(0)",
|
|
||||||
"legendFormat": "{{path}}",
|
|
||||||
"refId": "A"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "PUT Path Rate",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ms",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 8
|
|
||||||
},
|
|
||||||
"id": 3,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.95, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
|
|
||||||
"legendFormat": "p95 {{stage}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.99, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
|
|
||||||
"legendFormat": "p99 {{stage}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "PUT Hot Stage Latency",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "bytes",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 8
|
|
||||||
},
|
|
||||||
"id": 4,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.50, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
|
|
||||||
"legendFormat": "p50 {{path}} / {{size_bucket}}",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "histogram_quantile(0.95, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
|
|
||||||
"legendFormat": "p95 {{path}} / {{size_bucket}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "PUT Selected Buffer Size",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "percentunit",
|
|
||||||
"min": 0,
|
|
||||||
"max": 1,
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 15,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 0,
|
|
||||||
"y": 16
|
|
||||||
},
|
|
||||||
"id": 5,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_s3_put_object_zero_copy_eligible_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
|
|
||||||
"legendFormat": "eligible / total",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_s3_put_object_zero_copy_enabled_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
|
|
||||||
"legendFormat": "enabled / total",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "PUT Zero-Copy Eligibility Ratio",
|
|
||||||
"type": "timeseries"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"fieldConfig": {
|
|
||||||
"defaults": {
|
|
||||||
"unit": "ops",
|
|
||||||
"custom": {
|
|
||||||
"drawStyle": "line",
|
|
||||||
"lineInterpolation": "smooth",
|
|
||||||
"fillOpacity": 10,
|
|
||||||
"pointSize": 5,
|
|
||||||
"lineWidth": 1,
|
|
||||||
"showPoints": "auto",
|
|
||||||
"spanNulls": false,
|
|
||||||
"thresholdsStyle": {
|
|
||||||
"mode": "off"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"thresholds": {
|
|
||||||
"mode": "absolute",
|
|
||||||
"steps": [
|
|
||||||
{
|
|
||||||
"color": "green",
|
|
||||||
"value": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"overrides": []
|
|
||||||
},
|
|
||||||
"gridPos": {
|
|
||||||
"h": 8,
|
|
||||||
"w": 12,
|
|
||||||
"x": 12,
|
|
||||||
"y": 16
|
|
||||||
},
|
|
||||||
"id": 6,
|
|
||||||
"options": {
|
|
||||||
"legend": {
|
|
||||||
"calcs": [
|
|
||||||
"mean",
|
|
||||||
"max"
|
|
||||||
],
|
|
||||||
"displayMode": "table",
|
|
||||||
"placement": "bottom"
|
|
||||||
},
|
|
||||||
"reduceOptions": {
|
|
||||||
"calcs": [
|
|
||||||
"lastNotNull"
|
|
||||||
],
|
|
||||||
"fields": "",
|
|
||||||
"values": false
|
|
||||||
},
|
|
||||||
"tooltip": {
|
|
||||||
"mode": "multi",
|
|
||||||
"sort": "desc"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"targets": [
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum(rate(rustfs_s3_put_object_total[$__rate_interval])) or vector(0)",
|
|
||||||
"legendFormat": "PUT total",
|
|
||||||
"refId": "A"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"datasource": {
|
|
||||||
"type": "prometheus",
|
|
||||||
"uid": "${DS_PROMETHEUS}"
|
|
||||||
},
|
|
||||||
"expr": "sum by (stage) (rate(rustfs_system_storage_erasure_write_quorum_failures_total[$__rate_interval])) or vector(0)",
|
|
||||||
"legendFormat": "write quorum failure {{stage}}",
|
|
||||||
"refId": "B"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"title": "PUT Throughput and Quorum Failure Signals",
|
|
||||||
"type": "timeseries"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"refresh": "30s",
|
|
||||||
"schemaVersion": 38,
|
|
||||||
"style": "dark",
|
|
||||||
"tags": [
|
|
||||||
"rustfs",
|
|
||||||
"put",
|
|
||||||
"performance",
|
|
||||||
"attribution"
|
|
||||||
],
|
|
||||||
"templating": {
|
|
||||||
"list": [
|
|
||||||
{
|
|
||||||
"current": {
|
|
||||||
"selected": false,
|
|
||||||
"text": "Prometheus",
|
|
||||||
"value": "Prometheus"
|
|
||||||
},
|
|
||||||
"hide": 0,
|
|
||||||
"includeAll": false,
|
|
||||||
"multi": false,
|
|
||||||
"name": "DS_PROMETHEUS",
|
|
||||||
"options": [],
|
|
||||||
"query": "prometheus",
|
|
||||||
"refresh": 1,
|
|
||||||
"regex": "",
|
|
||||||
"skipUrlSync": false,
|
|
||||||
"type": "datasource"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"time": {
|
|
||||||
"from": "now-1h",
|
|
||||||
"to": "now"
|
|
||||||
},
|
|
||||||
"timezone": "",
|
|
||||||
"title": "RustFS PUT Performance Attribution",
|
|
||||||
"uid": "rustfs-put-performance-attribution",
|
|
||||||
"version": 1,
|
|
||||||
"weekStart": ""
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,25 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
apiVersion: 1
|
|
||||||
|
|
||||||
providers:
|
|
||||||
- name: "default"
|
|
||||||
orgId: 1
|
|
||||||
folder: ""
|
|
||||||
type: file
|
|
||||||
disableDeletion: false
|
|
||||||
updateIntervalSeconds: 10
|
|
||||||
options:
|
|
||||||
path: /etc/grafana/dashboards
|
|
||||||
@@ -1,98 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
apiVersion: 1
|
|
||||||
|
|
||||||
datasources:
|
|
||||||
- name: Prometheus
|
|
||||||
type: prometheus
|
|
||||||
uid: prometheus
|
|
||||||
url: http://prometheus:9090
|
|
||||||
access: proxy
|
|
||||||
isDefault: true
|
|
||||||
editable: false
|
|
||||||
jsonData:
|
|
||||||
httpMethod: GET
|
|
||||||
exemplarTraceIdDestinations:
|
|
||||||
- name: trace_id
|
|
||||||
datasourceUid: tempo
|
|
||||||
|
|
||||||
- name: Tempo
|
|
||||||
type: tempo
|
|
||||||
uid: tempo
|
|
||||||
access: proxy
|
|
||||||
url: http://tempo:3200
|
|
||||||
isDefault: false
|
|
||||||
editable: false
|
|
||||||
jsonData:
|
|
||||||
httpMethod: GET
|
|
||||||
serviceMap:
|
|
||||||
datasourceUid: prometheus
|
|
||||||
tracesToLogs:
|
|
||||||
datasourceUid: loki
|
|
||||||
tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
|
|
||||||
mappedTags: [ { key: 'service.name', value: 'app' } ]
|
|
||||||
spanStartTimeShift: '-1h'
|
|
||||||
spanEndTimeShift: '1h'
|
|
||||||
filterByTraceID: true
|
|
||||||
filterBySpanID: false
|
|
||||||
tracesToMetrics:
|
|
||||||
datasourceUid: prometheus
|
|
||||||
tags: [ { key: 'service.name' }, { key: 'job' } ]
|
|
||||||
queries:
|
|
||||||
- name: 'Service-Level Latency'
|
|
||||||
query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
|
|
||||||
- name: 'Service-Level Calls'
|
|
||||||
query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
|
|
||||||
- name: 'Service-Level Errors'
|
|
||||||
query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
|
|
||||||
nodeGraph:
|
|
||||||
enabled: true
|
|
||||||
|
|
||||||
- name: Loki
|
|
||||||
type: loki
|
|
||||||
uid: loki
|
|
||||||
url: http://loki:3100
|
|
||||||
basicAuth: false
|
|
||||||
isDefault: false
|
|
||||||
editable: false
|
|
||||||
jsonData:
|
|
||||||
derivedFields:
|
|
||||||
- datasourceUid: tempo
|
|
||||||
matcherRegex: 'trace_id=(\w+)'
|
|
||||||
name: 'TraceID'
|
|
||||||
url: '$${__value.raw}'
|
|
||||||
|
|
||||||
- name: Jaeger
|
|
||||||
type: jaeger
|
|
||||||
uid: jaeger
|
|
||||||
url: http://jaeger:16686
|
|
||||||
access: proxy
|
|
||||||
isDefault: false
|
|
||||||
editable: false
|
|
||||||
jsonData:
|
|
||||||
tracesToLogs:
|
|
||||||
datasourceUid: loki
|
|
||||||
tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
|
|
||||||
mappedTags: [ { key: 'service.name', value: 'app' } ]
|
|
||||||
spanStartTimeShift: '1s'
|
|
||||||
spanEndTimeShift: '-1s'
|
|
||||||
filterByTraceID: true
|
|
||||||
filterBySpanID: false
|
|
||||||
|
|
||||||
- name: Pyroscope
|
|
||||||
type: grafana-pyroscope-datasource
|
|
||||||
url: http://pyroscope:4040
|
|
||||||
jsonData:
|
|
||||||
minStep: '15s'
|
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
# Copyright 2024 RustFS Team
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
service:
|
||||||
|
extensions: [ jaeger_storage, jaeger_query, remote_sampling, healthcheckv2 ]
|
||||||
|
pipelines:
|
||||||
|
traces:
|
||||||
|
receivers: [ otlp, jaeger, zipkin ]
|
||||||
|
processors: [ batch, adaptive_sampling ]
|
||||||
|
exporters: [ jaeger_storage_exporter ]
|
||||||
|
telemetry:
|
||||||
|
resource:
|
||||||
|
service.name: jaeger
|
||||||
|
metrics:
|
||||||
|
level: detailed
|
||||||
|
readers:
|
||||||
|
- pull:
|
||||||
|
exporter:
|
||||||
|
prometheus:
|
||||||
|
host: 0.0.0.0
|
||||||
|
port: 8888
|
||||||
|
logs:
|
||||||
|
level: debug
|
||||||
|
# TODO Initialize telemetry tracer once OTEL released new feature.
|
||||||
|
# https://github.com/open-telemetry/opentelemetry-collector/issues/10663
|
||||||
|
|
||||||
|
extensions:
|
||||||
|
healthcheckv2:
|
||||||
|
use_v2: true
|
||||||
|
http:
|
||||||
|
|
||||||
|
# pprof:
|
||||||
|
# endpoint: 0.0.0.0:1777
|
||||||
|
# zpages:
|
||||||
|
# endpoint: 0.0.0.0:55679
|
||||||
|
|
||||||
|
jaeger_query:
|
||||||
|
storage:
|
||||||
|
traces: some_store
|
||||||
|
traces_archive: another_store
|
||||||
|
ui:
|
||||||
|
config_file: ./cmd/jaeger/config-ui.json
|
||||||
|
log_access: true
|
||||||
|
# The maximum duration that is considered for clock skew adjustments.
|
||||||
|
# Defaults to 0 seconds, which means it's disabled.
|
||||||
|
max_clock_skew_adjust: 0s
|
||||||
|
grpc:
|
||||||
|
endpoint: 0.0.0.0:16685
|
||||||
|
http:
|
||||||
|
endpoint: 0.0.0.0:16686
|
||||||
|
|
||||||
|
jaeger_storage:
|
||||||
|
backends:
|
||||||
|
some_store:
|
||||||
|
memory:
|
||||||
|
max_traces: 1000000
|
||||||
|
another_store:
|
||||||
|
memory:
|
||||||
|
max_traces: 1000000
|
||||||
|
metric_backends:
|
||||||
|
some_metrics_storage:
|
||||||
|
prometheus:
|
||||||
|
endpoint: http://prometheus:9090
|
||||||
|
normalize_calls: true
|
||||||
|
normalize_duration: true
|
||||||
|
|
||||||
|
remote_sampling:
|
||||||
|
# You can either use file or adaptive sampling strategy in remote_sampling
|
||||||
|
# file:
|
||||||
|
# path: ./cmd/jaeger/sampling-strategies.json
|
||||||
|
adaptive:
|
||||||
|
sampling_store: some_store
|
||||||
|
initial_sampling_probability: 0.1
|
||||||
|
http:
|
||||||
|
grpc:
|
||||||
|
|
||||||
|
receivers:
|
||||||
|
otlp:
|
||||||
|
protocols:
|
||||||
|
grpc:
|
||||||
|
http:
|
||||||
|
|
||||||
|
jaeger:
|
||||||
|
protocols:
|
||||||
|
grpc:
|
||||||
|
thrift_binary:
|
||||||
|
thrift_compact:
|
||||||
|
thrift_http:
|
||||||
|
|
||||||
|
zipkin:
|
||||||
|
|
||||||
|
processors:
|
||||||
|
batch:
|
||||||
|
# Adaptive Sampling Processor is required to support adaptive sampling.
|
||||||
|
# It expects remote_sampling extension with `adaptive:` config to be enabled.
|
||||||
|
adaptive_sampling:
|
||||||
|
|
||||||
|
exporters:
|
||||||
|
jaeger_storage_exporter:
|
||||||
|
trace_storage: some_store
|
||||||
|
|
||||||
@@ -1,74 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
service:
|
|
||||||
extensions: [jaeger_storage, jaeger_query]
|
|
||||||
pipelines:
|
|
||||||
traces:
|
|
||||||
receivers: [otlp]
|
|
||||||
processors: [batch]
|
|
||||||
exporters: [jaeger_storage_exporter, spanmetrics]
|
|
||||||
metrics/spanmetrics:
|
|
||||||
receivers: [spanmetrics]
|
|
||||||
exporters: [prometheus]
|
|
||||||
telemetry:
|
|
||||||
resource:
|
|
||||||
service.name: jaeger
|
|
||||||
metrics:
|
|
||||||
level: detailed
|
|
||||||
readers:
|
|
||||||
- pull:
|
|
||||||
exporter:
|
|
||||||
prometheus:
|
|
||||||
host: 0.0.0.0
|
|
||||||
port: 8888
|
|
||||||
logs:
|
|
||||||
level: DEBUG
|
|
||||||
|
|
||||||
extensions:
|
|
||||||
jaeger_query:
|
|
||||||
storage:
|
|
||||||
traces: some_storage
|
|
||||||
metrics: some_metrics_storage
|
|
||||||
jaeger_storage:
|
|
||||||
backends:
|
|
||||||
some_storage:
|
|
||||||
memory:
|
|
||||||
max_traces: 100000
|
|
||||||
metric_backends:
|
|
||||||
some_metrics_storage:
|
|
||||||
prometheus:
|
|
||||||
endpoint: http://prometheus:9090
|
|
||||||
normalize_calls: true
|
|
||||||
normalize_duration: true
|
|
||||||
|
|
||||||
connectors:
|
|
||||||
spanmetrics:
|
|
||||||
|
|
||||||
receivers:
|
|
||||||
otlp:
|
|
||||||
protocols:
|
|
||||||
grpc:
|
|
||||||
endpoint: "0.0.0.0:4317"
|
|
||||||
http:
|
|
||||||
endpoint: "0.0.0.0:4318"
|
|
||||||
|
|
||||||
processors:
|
|
||||||
batch:
|
|
||||||
|
|
||||||
exporters:
|
|
||||||
jaeger_storage_exporter:
|
|
||||||
trace_storage: some_storage
|
|
||||||
prometheus:
|
|
||||||
endpoint: "0.0.0.0:8889"
|
|
||||||
@@ -11,21 +11,22 @@
|
|||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
auth_enabled: false
|
auth_enabled: false
|
||||||
|
|
||||||
server:
|
server:
|
||||||
http_listen_port: 3100
|
http_listen_port: 3100
|
||||||
grpc_listen_port: 9095
|
grpc_listen_port: 9096
|
||||||
log_level: info
|
log_level: debug
|
||||||
grpc_server_max_concurrent_streams: 1000
|
grpc_server_max_concurrent_streams: 1000
|
||||||
|
|
||||||
common:
|
common:
|
||||||
instance_addr: 127.0.0.1
|
instance_addr: 127.0.0.1
|
||||||
path_prefix: /loki
|
path_prefix: /tmp/loki
|
||||||
storage:
|
storage:
|
||||||
filesystem:
|
filesystem:
|
||||||
chunks_directory: /loki/chunks
|
chunks_directory: /tmp/loki/chunks
|
||||||
rules_directory: /loki/rules
|
rules_directory: /tmp/loki/rules
|
||||||
replication_factor: 1
|
replication_factor: 1
|
||||||
ring:
|
ring:
|
||||||
kvstore:
|
kvstore:
|
||||||
@@ -38,6 +39,9 @@ query_range:
|
|||||||
enabled: true
|
enabled: true
|
||||||
max_size_mb: 100
|
max_size_mb: 100
|
||||||
|
|
||||||
|
limits_config:
|
||||||
|
metric_aggregation_enabled: true
|
||||||
|
|
||||||
schema_config:
|
schema_config:
|
||||||
configs:
|
configs:
|
||||||
- from: 2020-10-24
|
- from: 2020-10-24
|
||||||
@@ -48,16 +52,26 @@ schema_config:
|
|||||||
prefix: index_
|
prefix: index_
|
||||||
period: 24h
|
period: 24h
|
||||||
|
|
||||||
limits_config:
|
|
||||||
reject_old_samples: true
|
|
||||||
reject_old_samples_max_age: 168h
|
|
||||||
allow_structured_metadata: true
|
|
||||||
max_line_size: 256KB
|
|
||||||
|
|
||||||
pattern_ingester:
|
pattern_ingester:
|
||||||
enabled: true
|
enabled: true
|
||||||
metric_aggregation:
|
metric_aggregation:
|
||||||
loki_address: localhost:3100
|
loki_address: localhost:3100
|
||||||
|
|
||||||
|
ruler:
|
||||||
|
alertmanager_url: http://localhost:9093
|
||||||
|
|
||||||
frontend:
|
frontend:
|
||||||
encoding: protobuf
|
encoding: protobuf
|
||||||
|
|
||||||
|
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
|
||||||
|
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
|
||||||
|
#
|
||||||
|
# Statistics help us better understand how Loki is used, and they show us performance
|
||||||
|
# levels for most users. This helps us prioritize features and documentation.
|
||||||
|
# For more information on what's sent, look at
|
||||||
|
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
|
||||||
|
# Refer to the buildReport method to see what goes into a report.
|
||||||
|
#
|
||||||
|
# If you would like to disable reporting, uncomment the following lines:
|
||||||
|
#analytics:
|
||||||
|
# reporting_enabled: false
|
||||||
@@ -15,118 +15,67 @@
|
|||||||
receivers:
|
receivers:
|
||||||
otlp:
|
otlp:
|
||||||
protocols:
|
protocols:
|
||||||
grpc:
|
grpc: # OTLP gRPC 接收器
|
||||||
endpoint: 0.0.0.0:4317
|
endpoint: 0.0.0.0:4317
|
||||||
http:
|
http: # OTLP HTTP 接收器
|
||||||
endpoint: 0.0.0.0:4318
|
endpoint: 0.0.0.0:4318
|
||||||
|
|
||||||
processors:
|
processors:
|
||||||
batch:
|
batch: # 批处理处理器，提升吞吐量
|
||||||
timeout: 1s
|
timeout: 5s
|
||||||
send_batch_size: 1024
|
send_batch_size: 1000
|
||||||
memory_limiter:
|
memory_limiter:
|
||||||
check_interval: 1s
|
check_interval: 1s
|
||||||
limit_mib: 1024
|
limit_mib: 512
|
||||||
spike_limit_mib: 256
|
|
||||||
transform/logs:
|
|
||||||
error_mode: ignore
|
|
||||||
log_statements:
|
|
||||||
- context: log
|
|
||||||
statements:
|
|
||||||
- set(attributes["message"], body.string) where IsString(body)
|
|
||||||
- set(attributes["log.body"], body.string) where IsString(body)
|
|
||||||
- set(body, Concat([body, " event=", attributes["event"]], "")) where IsString(body) and attributes["event"] != nil
|
|
||||||
- set(body, Concat([body, " component=", attributes["component"]], "")) where IsString(body) and attributes["component"] != nil
|
|
||||||
- set(body, Concat([body, " subsystem=", attributes["subsystem"]], "")) where IsString(body) and attributes["subsystem"] != nil
|
|
||||||
- set(body, Concat([body, " state=", attributes["state"]], "")) where IsString(body) and attributes["state"] != nil
|
|
||||||
- set(body, Concat([body, " result=", attributes["result"]], "")) where IsString(body) and attributes["result"] != nil
|
|
||||||
- set(body, Concat([body, " reason=", attributes["reason"]], "")) where IsString(body) and attributes["reason"] != nil
|
|
||||||
- set(body, Concat([body, " failure_reason=", attributes["failure_reason"]], "")) where IsString(body) and attributes["failure_reason"] != nil
|
|
||||||
- set(body, Concat([body, " rpc_path=", attributes["rpc_path"]], "")) where IsString(body) and attributes["rpc_path"] != nil
|
|
||||||
- set(body, Concat([body, " rpc_service=", attributes["rpc_service"]], "")) where IsString(body) and attributes["rpc_service"] != nil
|
|
||||||
- set(body, Concat([body, " rpc_method=", attributes["rpc_method"]], "")) where IsString(body) and attributes["rpc_method"] != nil
|
|
||||||
- set(body, Concat([body, " expected_audience=", attributes["expected_audience"]], "")) where IsString(body) and attributes["expected_audience"] != nil
|
|
||||||
- set(body, Concat([body, " peer_addr=", attributes["peer_addr"]], "")) where IsString(body) and attributes["peer_addr"] != nil
|
|
||||||
- set(body, Concat([body, " replay_scope_bootstrap_allowed=", attributes["replay_scope_bootstrap_allowed"]], "")) where IsString(body) and attributes["replay_scope_bootstrap_allowed"] != nil
|
|
||||||
- set(body, Concat([body, " error=", attributes["error"]], "")) where IsString(body) and attributes["error"] != nil
|
|
||||||
- set(body, Concat([body, " exception_message=", attributes["exception.message"]], "")) where IsString(body) and attributes["exception.message"] != nil
|
|
||||||
|
|
||||||
exporters:
|
exporters:
|
||||||
otlp/tempo:
|
otlp/traces: # OTLP 导出器，用于跟踪数据
|
||||||
endpoint: "tempo:4317"
|
endpoint: "jaeger:4317" # Jaeger 的 OTLP gRPC 端点
|
||||||
|
tls:
|
||||||
|
insecure: true # 开发环境禁用 TLS，生产环境需配置证书
|
||||||
|
otlp/tempo: # OTLP 导出器，用于跟踪数据
|
||||||
|
endpoint: "tempo:4317" # tempo 的 OTLP gRPC 端点
|
||||||
|
tls:
|
||||||
|
insecure: true # 开发环境禁用 TLS，生产环境需配置证书
|
||||||
|
prometheus: # Prometheus 导出器，用于指标数据
|
||||||
|
endpoint: "0.0.0.0:8889" # Prometheus 刮取端点
|
||||||
|
namespace: "rustfs" # 指标前缀
|
||||||
|
send_timestamps: true # 发送时间戳
|
||||||
|
# enable_open_metrics: true
|
||||||
|
otlphttp/loki: # Loki 导出器，用于日志数据
|
||||||
|
# endpoint: "http://loki:3100/otlp/v1/logs"
|
||||||
|
endpoint: "http://loki:3100/otlp/v1/logs"
|
||||||
tls:
|
tls:
|
||||||
insecure: true
|
insecure: true
|
||||||
compression: gzip
|
|
||||||
retry_on_failure:
|
|
||||||
enabled: true
|
|
||||||
initial_interval: 1s
|
|
||||||
max_interval: 30s
|
|
||||||
max_elapsed_time: 300s
|
|
||||||
sending_queue:
|
|
||||||
enabled: true
|
|
||||||
num_consumers: 10
|
|
||||||
queue_size: 5000
|
|
||||||
|
|
||||||
otlp/jaeger:
|
|
||||||
endpoint: "jaeger:4317"
|
|
||||||
tls:
|
|
||||||
insecure: true
|
|
||||||
compression: gzip
|
|
||||||
retry_on_failure:
|
|
||||||
enabled: true
|
|
||||||
initial_interval: 1s
|
|
||||||
max_interval: 30s
|
|
||||||
max_elapsed_time: 300s
|
|
||||||
sending_queue:
|
|
||||||
enabled: true
|
|
||||||
num_consumers: 10
|
|
||||||
queue_size: 5000
|
|
||||||
|
|
||||||
prometheus:
|
|
||||||
endpoint: "0.0.0.0:8889"
|
|
||||||
send_timestamps: true
|
|
||||||
metric_expiration: 5m
|
|
||||||
resource_to_telemetry_conversion:
|
|
||||||
enabled: true
|
|
||||||
|
|
||||||
otlphttp/loki:
|
|
||||||
endpoint: "http://loki:3100/otlp"
|
|
||||||
tls:
|
|
||||||
insecure: true
|
|
||||||
compression: gzip
|
|
||||||
|
|
||||||
extensions:
|
extensions:
|
||||||
health_check:
|
health_check:
|
||||||
endpoint: 0.0.0.0:13133
|
|
||||||
pprof:
|
pprof:
|
||||||
endpoint: 0.0.0.0:1888
|
|
||||||
zpages:
|
zpages:
|
||||||
endpoint: 0.0.0.0:55679
|
|
||||||
|
|
||||||
service:
|
service:
|
||||||
extensions: [ health_check, pprof, zpages ]
|
extensions: [ health_check, pprof, zpages ] # 启用扩展
|
||||||
pipelines:
|
pipelines:
|
||||||
traces:
|
traces:
|
||||||
receivers: [ otlp ]
|
receivers: [ otlp ]
|
||||||
processors: [ memory_limiter, batch ]
|
processors: [ memory_limiter,batch ]
|
||||||
exporters: [ otlp/tempo, otlp/jaeger ]
|
exporters: [ otlp/traces,otlp/tempo ]
|
||||||
metrics:
|
metrics:
|
||||||
receivers: [ otlp ]
|
receivers: [ otlp ]
|
||||||
processors: [ batch ]
|
processors: [ batch ]
|
||||||
exporters: [ prometheus ]
|
exporters: [ prometheus ]
|
||||||
logs:
|
logs:
|
||||||
receivers: [ otlp ]
|
receivers: [ otlp ]
|
||||||
processors: [ batch, transform/logs ]
|
processors: [ batch ]
|
||||||
exporters: [ otlphttp/loki ]
|
exporters: [ otlphttp/loki ]
|
||||||
telemetry:
|
telemetry:
|
||||||
logs:
|
logs:
|
||||||
level: "info"
|
level: "info" # Collector 日志级别
|
||||||
encoding: "json"
|
|
||||||
metrics:
|
metrics:
|
||||||
level: "normal"
|
level: "detailed" # 可以是 basic, normal, detailed
|
||||||
readers:
|
readers:
|
||||||
- pull:
|
- periodic:
|
||||||
exporter:
|
exporter:
|
||||||
prometheus:
|
otlp:
|
||||||
host: '0.0.0.0'
|
protocol: http/protobuf
|
||||||
port: 8888
|
endpoint: http://otel-collector:4318
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,53 +0,0 @@
|
|||||||
groups:
|
|
||||||
- name: rustfs-dashboard
|
|
||||||
interval: 30s
|
|
||||||
rules:
|
|
||||||
- record: rustfs:http_server_requests:rate5m
|
|
||||||
expr: sum by (job) (rate(rustfs_http_server_requests_total[5m]))
|
|
||||||
|
|
||||||
- record: rustfs:http_server_request_duration_seconds:p50_5m
|
|
||||||
expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
|
|
||||||
- record: rustfs:http_server_request_duration_seconds:p95_5m
|
|
||||||
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
|
|
||||||
- record: rustfs:http_server_request_duration_seconds:p99_5m
|
|
||||||
expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
|
|
||||||
|
|
||||||
- record: rustfs:http_server_response_body_size_bytes:p50_5m
|
|
||||||
expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
|
|
||||||
- record: rustfs:http_server_response_body_size_bytes:p95_5m
|
|
||||||
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
|
|
||||||
- record: rustfs:http_server_response_body_size_bytes:p99_5m
|
|
||||||
expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
|
|
||||||
|
|
||||||
- record: rustfs:log_cleaner_runs:rate15m
|
|
||||||
expr: sum by (job) (rate(rustfs_log_cleaner_runs_total[15m]))
|
|
||||||
- record: rustfs:log_cleaner_failure_ratio:rate5m
|
|
||||||
expr: sum by (job) (rate(rustfs_log_cleaner_run_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_runs_total[5m])), 1e-9)
|
|
||||||
- record: rustfs:log_cleaner_rotation_failure_ratio:rate5m
|
|
||||||
expr: sum by (job) (rate(rustfs_log_cleaner_rotation_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_rotation_total[5m])), 1e-9)
|
|
||||||
- record: rustfs:log_cleaner_rotation_duration_seconds:p95_5m
|
|
||||||
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_rotation_duration_seconds_bucket[5m])))
|
|
||||||
- record: rustfs:log_cleaner_compress_duration_seconds:p95_5m
|
|
||||||
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])))
|
|
||||||
|
|
||||||
- record: rustfs:scanner_objects_scanned:rate5m
|
|
||||||
expr: sum by (job) (rate(rustfs_scanner_objects_scanned_total[5m]))
|
|
||||||
- record: rustfs:scanner_directories_scanned:rate5m
|
|
||||||
expr: sum by (job) (rate(rustfs_scanner_directories_scanned_total[5m]))
|
|
||||||
- record: rustfs:scanner_buckets_scanned:rate5m
|
|
||||||
expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
|
|
||||||
- record: rustfs:scanner_cycles_success:rate5m
|
|
||||||
expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))
|
|
||||||
|
|
||||||
- record: rustfs:log_chain_op_event_mismatch:rate5m
|
|
||||||
expr: sum by (job) (rate(rustfs_log_chain_op_event_mismatch_total[5m]))
|
|
||||||
|
|
||||||
- alert: RustFSLogChainOpEventMismatchDetected
|
|
||||||
expr: rustfs:log_chain_op_event_mismatch:rate5m > 0
|
|
||||||
for: 10m
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
component: s3-log-chain
|
|
||||||
annotations:
|
|
||||||
summary: "RustFS log-chain op/event mismatch detected"
|
|
||||||
description: "job={{ $labels.job }} has non-zero rustfs_log_chain_op_event_mismatch_total rate for more than 10m. Check s3 op/event mapping changes."
|
|
||||||
@@ -1,260 +0,0 @@
|
|||||||
# =============================================================================
|
|
||||||
# RustFS GET Optimization — Prometheus Alerting Rules
|
|
||||||
# =============================================================================
|
|
||||||
#
|
|
||||||
# Import into Prometheus:
|
|
||||||
# 1. Copy this file to your Prometheus rules directory
|
|
||||||
# 2. Add to prometheus.yml:
|
|
||||||
# rule_files:
|
|
||||||
# - "prometheus-alert-rules.yaml"
|
|
||||||
# 3. Validate: promtool check rules prometheus-alert-rules.yaml
|
|
||||||
# 4. Reload: curl -X POST http://localhost:9090/-/reload
|
|
||||||
#
|
|
||||||
# All metric names match those registered in crates/io-metrics/src/lib.rs
|
|
||||||
# and documented in crates/ecstore/src/diagnostics/get.rs.
|
|
||||||
#
|
|
||||||
# Baseline comparison uses "offset 1d" — adjust to "offset 7d" for weekly
|
|
||||||
# seasonality if your traffic pattern varies by day of week.
|
|
||||||
# =============================================================================
|
|
||||||
|
|
||||||
groups:
|
|
||||||
# ==========================================================================
|
|
||||||
# Critical alerts — immediate action required
|
|
||||||
# ==========================================================================
|
|
||||||
- name: rustfs-get-optimization-critical
|
|
||||||
interval: 30s
|
|
||||||
rules:
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 1. GetP99Regression
|
|
||||||
# GET p99 latency exceeds 2x the baseline (same time yesterday)
|
|
||||||
# sustained for 10 minutes.
|
|
||||||
# Action: Roll back the GET optimization immediately.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: GetP99Regression
|
|
||||||
expr: |
|
|
||||||
histogram_quantile(0.99,
|
|
||||||
sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m])) by (le)
|
|
||||||
)
|
|
||||||
>
|
|
||||||
2
|
|
||||||
*
|
|
||||||
histogram_quantile(0.99,
|
|
||||||
sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m] offset 1d)) by (le)
|
|
||||||
)
|
|
||||||
for: 10m
|
|
||||||
labels:
|
|
||||||
severity: critical
|
|
||||||
team: rustfs-storage
|
|
||||||
area: get-optimization
|
|
||||||
annotations:
|
|
||||||
summary: "GET p99 latency regression detected (>2x baseline for 10m)"
|
|
||||||
description: >-
|
|
||||||
The 99th-percentile GET object latency is {{ $value | humanizeDuration }}
|
|
||||||
which is more than double the baseline measured 24 hours ago.
|
|
||||||
This indicates a severe performance regression introduced by
|
|
||||||
a recent GET optimization change.
|
|
||||||
runbook_url: "https://internal.wiki/runbooks/rustfs/get-p99-regression"
|
|
||||||
action: >
|
|
||||||
1. Verify the regression is not caused by external factors
|
|
||||||
(disk health, network, load spike).
|
|
||||||
2. If confirmed optimization-related, roll back:
|
|
||||||
- Set RUSTFS_GET_CODEC_STREAMING=0
|
|
||||||
- Set RUSTFS_GET_METADATA_EARLY_STOP=0
|
|
||||||
- Restart affected nodes.
|
|
||||||
3. Collect flamegraphs and open a P0 incident.
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 2. PipelineFailureSpike
|
|
||||||
# Pipeline failure rate exceeds 5x the baseline sustained for
|
|
||||||
# 5 minutes. Covers all failure reasons: bitrot_mismatch,
|
|
||||||
# decode_error, downstream_closed, io, read_quorum, timeout, etc.
|
|
||||||
# Action: Investigate pipeline health and roll back if needed.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: PipelineFailureSpike
|
|
||||||
expr: |
|
|
||||||
sum(rate(rustfs_io_get_object_pipeline_failures_total[5m]))
|
|
||||||
>
|
|
||||||
5
|
|
||||||
*
|
|
||||||
sum(rate(rustfs_io_get_object_pipeline_failures_total[5m] offset 1d))
|
|
||||||
for: 5m
|
|
||||||
labels:
|
|
||||||
severity: critical
|
|
||||||
team: rustfs-storage
|
|
||||||
area: get-optimization
|
|
||||||
annotations:
|
|
||||||
summary: "GET pipeline failure rate spike (>5x baseline for 5m)"
|
|
||||||
description: >-
|
|
||||||
The GET pipeline failure rate is {{ $value | printf "%.2f" }}/s,
|
|
||||||
more than 5x the baseline from 24 hours ago.
|
|
||||||
Failure reasons may include: bitrot_mismatch, decode_error,
|
|
||||||
downstream_closed, io, range_or_length_invalid, read_quorum,
|
|
||||||
short_read, timeout, unknown.
|
|
||||||
runbook_url: "https://internal.wiki/runbooks/rustfs/pipeline-failure-spike"
|
|
||||||
action: >
|
|
||||||
1. Check Grafana "GET Data Integrity" dashboard for failure
|
|
||||||
breakdown by reason label.
|
|
||||||
2. If decode_error or bitrot_mismatch dominates, stop
|
|
||||||
optimization and investigate data integrity.
|
|
||||||
3. If io or timeout dominates, check disk and network health.
|
|
||||||
4. Roll back optimization if failures persist.
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 3. BitrotMismatchSpike
|
|
||||||
# Bitrot verification mismatch rate exceeds 3x baseline for
|
|
||||||
# 5 minutes. This is a data-integrity signal — shard checksums
|
|
||||||
# do not match after read.
|
|
||||||
#
|
|
||||||
# The "bitrot_mismatch" reason is recorded on the
|
|
||||||
# rustfs_io_get_object_pipeline_failures_total counter when a
|
|
||||||
# StorageError::FileCorrupt or DiskError::FileCorrupt /
|
|
||||||
# DiskError::PartMissingOrCorrupt is classified during the GET
|
|
||||||
# pipeline (see classify_storage_error / classify_disk_error in
|
|
||||||
# crates/ecstore/src/diagnostics/get.rs).
|
|
||||||
#
|
|
||||||
# Action: Stop optimization, investigate data integrity urgently.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: BitrotMismatchSpike
|
|
||||||
expr: |
|
|
||||||
sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m]))
|
|
||||||
>
|
|
||||||
3
|
|
||||||
*
|
|
||||||
sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m] offset 1d))
|
|
||||||
for: 5m
|
|
||||||
labels:
|
|
||||||
severity: critical
|
|
||||||
team: rustfs-storage
|
|
||||||
area: get-optimization
|
|
||||||
annotations:
|
|
||||||
summary: "Bitrot mismatch rate spike (>3x baseline for 5m)"
|
|
||||||
description: >-
|
|
||||||
The rate of pipeline failures classified as bitrot_mismatch is
|
|
||||||
{{ $value | printf "%.2f" }}/s, more than 3x the baseline from
|
|
||||||
24 hours ago. This indicates shard checksum verification
|
|
||||||
failures (FileCorrupt / PartMissingOrCorrupt) which may point
|
|
||||||
to data corruption introduced by the GET optimization pipeline
|
|
||||||
(e.g., incorrect decode, buffer reuse bug).
|
|
||||||
runbook_url: "https://internal.wiki/runbooks/rustfs/bitrot-mismatch-spike"
|
|
||||||
action: >
|
|
||||||
1. Immediately disable codec streaming:
|
|
||||||
RUSTFS_GET_CODEC_STREAMING=0
|
|
||||||
2. Run "mc admin scan" on affected buckets to verify on-disk
|
|
||||||
integrity independent of the GET path.
|
|
||||||
3. Compare xl.meta checksums across erasure shards.
|
|
||||||
4. If corruption confirmed, initiate data recovery from parity.
|
|
||||||
5. Do NOT re-enable optimization until root cause is identified.
|
|
||||||
|
|
||||||
# ==========================================================================
|
|
||||||
# Warning alerts — investigation needed
|
|
||||||
# ==========================================================================
|
|
||||||
- name: rustfs-get-optimization-warning
|
|
||||||
interval: 30s
|
|
||||||
rules:
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 4. EarlyStopInsufficientQuorum
|
|
||||||
# The metadata early-stop path is hitting "insufficient_quorum"
|
|
||||||
# at a rate above 0.1/s for 5 minutes. This means too many
|
|
||||||
# disks are failing to return valid metadata in time.
|
|
||||||
# Action: Check disk health and metadata fanout latency.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: EarlyStopInsufficientQuorum
|
|
||||||
expr: |
|
|
||||||
sum(rate(rustfs_io_get_object_metadata_early_stop_total{reason="insufficient_quorum"}[5m]))
|
|
||||||
> 0.1
|
|
||||||
for: 5m
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
team: rustfs-storage
|
|
||||||
area: get-optimization
|
|
||||||
annotations:
|
|
||||||
summary: "Early-stop insufficient quorum rate elevated (>0.1/s for 5m)"
|
|
||||||
description: >-
|
|
||||||
The metadata early-stop path is returning "insufficient_quorum"
|
|
||||||
at {{ $value | printf "%.3f" }}/s. This means the bounded
|
|
||||||
metadata fanout cannot gather enough valid responses before
|
|
||||||
the quorum deadline, suggesting disk or network issues.
|
|
||||||
runbook_url: "https://internal.wiki/runbooks/rustfs/early-stop-quorum"
|
|
||||||
action: >
|
|
||||||
1. Check disk health: mc admin info --json | jq '.disks'
|
|
||||||
2. Review rustfs_io_get_object_metadata_response_total by
|
|
||||||
outcome (error, timeout, disk_not_found) in Grafana.
|
|
||||||
3. Check rustfs_io_disk_permit_wait_duration_seconds for
|
|
||||||
I/O scheduler saturation.
|
|
||||||
4. If disks are healthy, consider increasing the early-stop
|
|
||||||
timeout or temporarily disabling early-stop.
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 5. CodecStreamingFallbackSpike
|
|
||||||
# The codec streaming fallback rate is >10x baseline for 10
|
|
||||||
# minutes. This means the optimized codec streaming path is
|
|
||||||
# being bypassed much more often than expected.
|
|
||||||
# Action: Check fallback reasons and object eligibility.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: CodecStreamingFallbackSpike
|
|
||||||
expr: |
|
|
||||||
sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
|
|
||||||
>
|
|
||||||
10
|
|
||||||
*
|
|
||||||
sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m] offset 1d))
|
|
||||||
for: 10m
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
team: rustfs-storage
|
|
||||||
area: get-optimization
|
|
||||||
annotations:
|
|
||||||
summary: "Codec streaming fallback rate spike (>10x baseline for 10m)"
|
|
||||||
description: >-
|
|
||||||
The codec streaming fallback rate is {{ $value | printf "%.2f" }}/s,
|
|
||||||
more than 10x the baseline from 24 hours ago. Fallback reasons
|
|
||||||
are labeled by "reason" — check Grafana for breakdown.
|
|
||||||
Common reasons: object too small, multipart not supported,
|
|
||||||
unsupported erasure layout, feature flag disabled.
|
|
||||||
runbook_url: "https://internal.wiki/runbooks/rustfs/codec-fallback-spike"
|
|
||||||
action: >
|
|
||||||
1. Query by reason label:
|
|
||||||
sum by (reason) (rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
|
|
||||||
2. If dominated by a single reason, investigate why that
|
|
||||||
condition became more frequent (e.g., workload change,
|
|
||||||
configuration drift).
|
|
||||||
3. Cross-reference with rustfs_io_get_object_reader_path_total
|
|
||||||
to verify the fallback path (legacy_duplex) is healthy.
|
|
||||||
4. If fallback is expected (e.g., workload shifted to small
|
|
||||||
objects), update the alert baseline.
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 6. IoQueueSaturation
|
|
||||||
# I/O queue utilization exceeds 90% for 5 minutes. High
|
|
||||||
# utilization causes disk permit wait latency to increase and
|
|
||||||
# can cascade into pipeline timeouts.
|
|
||||||
# Action: Check disk load and consider reducing concurrency.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: IoQueueSaturation
|
|
||||||
expr: |
|
|
||||||
rustfs_io_queue_utilization_percent > 90
|
|
||||||
for: 5m
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
team: rustfs-storage
|
|
||||||
area: get-optimization
|
|
||||||
annotations:
|
|
||||||
summary: "I/O queue utilization >90% for 5m"
|
|
||||||
description: >-
|
|
||||||
The I/O queue utilization is {{ $value | printf "%.1f" }}%,
|
|
||||||
sustained above 90% for 5 minutes. This indicates the disk
|
|
||||||
I/O scheduler is near saturation, which will increase
|
|
||||||
rustfs_io_disk_permit_wait_duration_seconds and may trigger
|
|
||||||
pipeline timeouts.
|
|
||||||
runbook_url: "https://internal.wiki/runbooks/rustfs/io-queue-saturation"
|
|
||||||
action: >
|
|
||||||
1. Check disk I/O metrics (iostat, node_exporter) for
|
|
||||||
individual disk saturation.
|
|
||||||
2. Review rustfs_io_queue_permits_in_use vs
|
|
||||||
rustfs_io_queue_permits_available for permit exhaustion.
|
|
||||||
3. Check rustfs_io_starvation_events for priority starvation.
|
|
||||||
4. If GET optimization increased concurrency, consider:
|
|
||||||
- Reducing RUSTFS_GET_PIPELINE_PARALLELISM
|
|
||||||
- Lowering RUSTFS_IO_QUEUE_PERMITS
|
|
||||||
5. If caused by background operations (ILM, healing), throttle
|
|
||||||
those before adjusting GET concurrency.
|
|
||||||
@@ -1,251 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
# =============================================================================
|
|
||||||
# RustFS KMS backend — Prometheus alerting rules
|
|
||||||
# =============================================================================
|
|
||||||
#
|
|
||||||
# Metric source: the KMS operation-policy choke point in
|
|
||||||
# crates/kms/src/policy.rs, except KmsKeyRotationOverdue, which reads the
|
|
||||||
# label-less key-lifecycle gauge published by the deletion worker's sweep
|
|
||||||
# (crates/kms/src/deletion_worker.rs). All label values are bounded static
|
|
||||||
# strings (operation, op_class, outcome, error_class, backend, scope); key
|
|
||||||
# identifiers, key material, and tokens never appear in labels.
|
|
||||||
#
|
|
||||||
# Response procedures: docs/operations/kms-observability-runbook.md
|
|
||||||
#
|
|
||||||
# IMPORTANT — threshold status: every numeric threshold below is a
|
|
||||||
# conservative default chosen without a production baseline. Calibrate against
|
|
||||||
# a staging baseline before relying on these alerts for paging, and prefer
|
|
||||||
# loosening over tightening until the baseline exists. Formal SLO targets are
|
|
||||||
# deliberately not encoded here (see rustfs/backlog#1584).
|
|
||||||
#
|
|
||||||
# NOTE: prometheus.yml loads /etc/prometheus/rules/*.yml — keep the .yml
|
|
||||||
# extension or the file is silently ignored by the docker-compose stack.
|
|
||||||
#
|
|
||||||
# Validate: promtool check rules rustfs-kms-alerts.yml
|
|
||||||
# =============================================================================
|
|
||||||
|
|
||||||
groups:
|
|
||||||
# ==========================================================================
|
|
||||||
# Critical alerts — immediate action required
|
|
||||||
# ==========================================================================
|
|
||||||
- name: rustfs-kms-critical
|
|
||||||
interval: 30s
|
|
||||||
rules:
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 1. KmsBackendFatalErrors
|
|
||||||
# Any attempt failure classified as fatal (non-retryable): auth
|
|
||||||
# or permission errors, malformed requests, missing keys. The
|
|
||||||
# policy never retries these, so even a low rate means real
|
|
||||||
# operations are failing right now.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: KmsBackendFatalErrors
|
|
||||||
expr: |
|
|
||||||
sum by (operation) (rate(rustfs_kms_backend_attempt_failures_total{error_class="fatal"}[5m])) > 0
|
|
||||||
for: 5m
|
|
||||||
labels:
|
|
||||||
severity: critical
|
|
||||||
component: kms
|
|
||||||
annotations:
|
|
||||||
summary: "KMS backend fatal errors on operation {{ $labels.operation }}"
|
|
||||||
description: >-
|
|
||||||
Attempt failures classified as fatal are occurring at
|
|
||||||
{{ $value | printf "%.3f" }}/s on operation
|
|
||||||
{{ $labels.operation }}. Fatal failures are not retried:
|
|
||||||
each one is a KMS backend call that failed permanently
|
|
||||||
(authentication, permissions, malformed request, or a
|
|
||||||
missing key/version).
|
|
||||||
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendfatalerrors"
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 2. KmsBackendHighErrorRate
|
|
||||||
# Sustained share of operations terminating without success
|
|
||||||
# (fatal, budget/deadline exhaustion, admission backpressure,
|
|
||||||
# or an open circuit). The cancelled outcome is excluded because
|
|
||||||
# shutdowns legitimately produce it.
|
|
||||||
# The traffic guard keeps a single failure on a near-idle
|
|
||||||
# cluster from firing the alert.
|
|
||||||
# Threshold: 5% for 10m — conservative default, calibrate
|
|
||||||
# against a staging baseline.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: KmsBackendHighErrorRate
|
|
||||||
expr: |
|
|
||||||
(
|
|
||||||
sum(rate(rustfs_kms_backend_operations_total{outcome!~"success|cancelled"}[5m]))
|
|
||||||
/
|
|
||||||
clamp_min(sum(rate(rustfs_kms_backend_operations_total[5m])), 1e-9)
|
|
||||||
) > 0.05
|
|
||||||
and
|
|
||||||
sum(rate(rustfs_kms_backend_operations_total[5m])) > 0.02
|
|
||||||
for: 10m
|
|
||||||
labels:
|
|
||||||
severity: critical
|
|
||||||
component: kms
|
|
||||||
annotations:
|
|
||||||
summary: "KMS backend non-success ratio above 5% for 10m"
|
|
||||||
description: >-
|
|
||||||
{{ $value | humanizePercentage }} of KMS backend operations
|
|
||||||
are terminating in fatal, budget_exhausted,
|
|
||||||
deadline_exceeded, backpressure_timeout,
|
|
||||||
backpressure_rejected, or circuit_open. Object encryption
|
|
||||||
and decryption paths depending on the KMS are degraded or
|
|
||||||
failing.
|
|
||||||
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
|
|
||||||
|
|
||||||
# ==========================================================================
|
|
||||||
# Warning alerts — investigation needed
|
|
||||||
# ==========================================================================
|
|
||||||
- name: rustfs-kms-warning
|
|
||||||
interval: 30s
|
|
||||||
rules:
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 3. KmsBackendP99LatencyHigh
|
|
||||||
# p99 wall-clock duration of whole operations (attempts plus
|
|
||||||
# backoff) is sustained above 2 seconds. Because the histogram
|
|
||||||
# includes retries, a high p99 usually means the retry policy
|
|
||||||
# is absorbing backend failures, not that every call is slow.
|
|
||||||
# Threshold: 2s for 10m — conservative default, calibrate
|
|
||||||
# against a staging baseline.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: KmsBackendP99LatencyHigh
|
|
||||||
expr: |
|
|
||||||
histogram_quantile(0.99,
|
|
||||||
sum by (le) (rate(rustfs_kms_backend_operation_duration_seconds_bucket[5m]))
|
|
||||||
) > 2
|
|
||||||
for: 10m
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
component: kms
|
|
||||||
annotations:
|
|
||||||
summary: "KMS backend operation p99 latency above 2s for 10m"
|
|
||||||
description: >-
|
|
||||||
The 99th-percentile KMS backend operation duration is
|
|
||||||
{{ $value | humanizeDuration }}, including retries and
|
|
||||||
backoff. Encryption and decryption latency is leaking into
|
|
||||||
S3 request latency.
|
|
||||||
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendp99latencyhigh"
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 4. KmsBackendAttemptFailureSpike
|
|
||||||
# Aggregate attempt-failure rate (all error classes) sustained
|
|
||||||
# above an absolute floor. An absolute threshold is used instead
|
|
||||||
# of an offset-1d baseline ratio because fresh deployments have
|
|
||||||
# no baseline and an empty offset vector would keep a ratio
|
|
||||||
# alert from ever firing; switch to a baseline-relative form
|
|
||||||
# (see rustfs-get-optimization-alerts.yaml for the pattern)
|
|
||||||
# once a stable staging baseline exists.
|
|
||||||
# Threshold: 0.5/s for 10m — conservative default, calibrate
|
|
||||||
# against a staging baseline.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: KmsBackendAttemptFailureSpike
|
|
||||||
expr: |
|
|
||||||
sum(rate(rustfs_kms_backend_attempt_failures_total[5m])) > 0.5
|
|
||||||
for: 10m
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
component: kms
|
|
||||||
annotations:
|
|
||||||
summary: "KMS backend attempt failures above 0.5/s for 10m"
|
|
||||||
description: >-
|
|
||||||
KMS backend attempts are failing at
|
|
||||||
{{ $value | printf "%.2f" }}/s across all error classes.
|
|
||||||
The retry policy may still be masking these from callers —
|
|
||||||
check the error-class breakdown before it stops absorbing
|
|
||||||
them.
|
|
||||||
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendattemptfailurespike"
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 5. KmsBackendRetryBudgetExhausted
|
|
||||||
# Operations are running out of retry budget (budget_exhausted)
|
|
||||||
# or operation deadline (deadline_exceeded). These surface to
|
|
||||||
# callers as failed KMS operations even though every individual
|
|
||||||
# failure was retryable — the backend is unhealthy for longer
|
|
||||||
# than the policy can bridge.
|
|
||||||
# Threshold: 0.05/s for 10m — conservative default, calibrate
|
|
||||||
# against a staging baseline.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: KmsBackendRetryBudgetExhausted
|
|
||||||
expr: |
|
|
||||||
sum by (outcome) (rate(rustfs_kms_backend_operations_total{outcome=~"budget_exhausted|deadline_exceeded"}[5m])) > 0.05
|
|
||||||
for: 10m
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
component: kms
|
|
||||||
annotations:
|
|
||||||
summary: "KMS backend operations exhausting retry budget ({{ $labels.outcome }})"
|
|
||||||
description: >-
|
|
||||||
KMS backend operations are terminating as
|
|
||||||
{{ $labels.outcome }} at {{ $value | printf "%.3f" }}/s.
|
|
||||||
Retryable failures are outlasting the retry budget, so
|
|
||||||
callers are seeing hard failures.
|
|
||||||
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 6. KmsBackendCircuitOpen
|
|
||||||
# Direct circuit-state signal, independent of operation traffic.
|
|
||||||
# A transient open can recover on its first half-open probe; alert
|
|
||||||
# only when the circuit remains open or half-open for one minute.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: KmsBackendCircuitOpen
|
|
||||||
expr: |
|
|
||||||
rustfs_kms_backend_circuit_open > 0
|
|
||||||
for: 1m
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
component: kms
|
|
||||||
annotations:
|
|
||||||
summary: "KMS backend circuit open ({{ $labels.backend }}/{{ $labels.scope }})"
|
|
||||||
description: >-
|
|
||||||
The KMS backend circuit for {{ $labels.backend }} scope
|
|
||||||
{{ $labels.scope }} has remained open or half-open for one
|
|
||||||
minute. Operations in this scope can terminate as
|
|
||||||
circuit_open until the half-open probe succeeds or returns
|
|
||||||
a non-retryable failure.
|
|
||||||
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
# 7. KmsKeyRotationOverdue
|
|
||||||
# The least recently rotated usable key has gone more than 400
|
|
||||||
# days without a rotation (measured from creation for keys with
|
|
||||||
# no recorded rotation). Direct gauge state published by the
|
|
||||||
# deletion worker's sweep, so no traffic guard applies; the
|
|
||||||
# one-hour hold only bridges scrape gaps. The worker runs only
|
|
||||||
# on backends with the schedule_deletion capability, so on the
|
|
||||||
# Static backend the series never exists and this alert cannot
|
|
||||||
# fire — that backend cannot rotate either; see the rotation
|
|
||||||
# driver matrix in docs/operations/kms-backend-security.md.
|
|
||||||
# Threshold: 400 days — conservative default sitting above a
|
|
||||||
# one-year rotation policy. Align it with the rotation period
|
|
||||||
# your compliance policy requires, and with
|
|
||||||
# RUSTFS_KMS_ROTATION_MAX_AGE_SECS so the per-key rotation_due
|
|
||||||
# verdict and this aggregate alert agree.
|
|
||||||
# ------------------------------------------------------------------
|
|
||||||
- alert: KmsKeyRotationOverdue
|
|
||||||
expr: |
|
|
||||||
rustfs_kms_oldest_key_rotation_age_seconds > (400 * 86400)
|
|
||||||
for: 1h
|
|
||||||
labels:
|
|
||||||
severity: warning
|
|
||||||
component: kms
|
|
||||||
annotations:
|
|
||||||
summary: "Oldest KMS key unrotated for more than 400 days"
|
|
||||||
description: >-
|
|
||||||
The least recently rotated usable KMS key was last rotated
|
|
||||||
{{ $value | humanizeDuration }} ago (measured from creation
|
|
||||||
for keys with no recorded rotation). List keys through the
|
|
||||||
admin API and read rotation_due / rotation_due_reason for
|
|
||||||
the per-key verdict; an "unsupported" reason means the
|
|
||||||
backend cannot rotate at all.
|
|
||||||
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmskeyrotationoverdue"
|
|
||||||
@@ -13,72 +13,16 @@
|
|||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
|
|
||||||
global:
|
global:
|
||||||
scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
|
scrape_interval: 5s # 刮取间隔
|
||||||
evaluation_interval: 15s
|
|
||||||
external_labels:
|
|
||||||
cluster: 'rustfs-dev' # Label to identify the cluster
|
|
||||||
replica: '1' # Replica identifier
|
|
||||||
|
|
||||||
rule_files:
|
|
||||||
- /etc/prometheus/rules/*.yml
|
|
||||||
|
|
||||||
scrape_configs:
|
scrape_configs:
|
||||||
- job_name: 'otel-collector'
|
- job_name: 'otel-collector'
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets: [ 'otel-collector:8888' ] # Scrape metrics from Collector
|
- targets: [ 'otel-collector:8888' ] # 从 Collector 刮取指标
|
||||||
scrape_interval: 10s
|
- job_name: 'otel-metrics'
|
||||||
|
|
||||||
- job_name: 'rustfs-app-metrics'
|
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets: [ 'otel-collector:8889' ] # Application indicators
|
- targets: [ 'otel-collector:8889' ] # 应用指标
|
||||||
scrape_interval: 15s
|
|
||||||
metric_relabel_configs:
|
|
||||||
- source_labels: [ __name__ ]
|
|
||||||
regex: 'go_.*'
|
|
||||||
action: drop # Drop Go runtime metrics if not needed
|
|
||||||
|
|
||||||
- job_name: 'tempo'
|
- job_name: 'tempo'
|
||||||
static_configs:
|
static_configs:
|
||||||
- targets: [ 'tempo:3200' ] # Scrape metrics from Tempo
|
- targets: [ 'tempo:3200' ]
|
||||||
|
|
||||||
- job_name: 'jaeger'
|
|
||||||
static_configs:
|
|
||||||
- targets: [ 'jaeger:14269' ] # Jaeger admin port (14269 is standard for admin/metrics)
|
|
||||||
|
|
||||||
- job_name: 'loki'
|
|
||||||
static_configs:
|
|
||||||
- targets: [ 'loki:3100' ]
|
|
||||||
|
|
||||||
- job_name: 'prometheus'
|
|
||||||
static_configs:
|
|
||||||
- targets: [ 'localhost:9090' ]
|
|
||||||
|
|
||||||
- job_name: 'vulture'
|
|
||||||
static_configs:
|
|
||||||
- targets:
|
|
||||||
- 'vulture:8080'
|
|
||||||
|
|
||||||
otlp:
|
|
||||||
promote_resource_attributes:
|
|
||||||
- service.instance.id
|
|
||||||
- service.name
|
|
||||||
- service.namespace
|
|
||||||
- cloud.availability_zone
|
|
||||||
- cloud.region
|
|
||||||
- container.name
|
|
||||||
- deployment.environment.name
|
|
||||||
- k8s.cluster.name
|
|
||||||
- k8s.container.name
|
|
||||||
- k8s.cronjob.name
|
|
||||||
- k8s.daemonset.name
|
|
||||||
- k8s.deployment.name
|
|
||||||
- k8s.job.name
|
|
||||||
- k8s.namespace.name
|
|
||||||
- k8s.pod.name
|
|
||||||
- k8s.replicaset.name
|
|
||||||
- k8s.statefulset.name
|
|
||||||
translation_strategy: NoUTF8EscapingWithSuffixes
|
|
||||||
|
|
||||||
storage:
|
|
||||||
tsdb:
|
|
||||||
out_of_order_time_window: 30m
|
|
||||||
|
|||||||
@@ -1,3 +0,0 @@
|
|||||||
kafka:
|
|
||||||
brokers:
|
|
||||||
- redpanda:9092
|
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
*
|
||||||
@@ -1,286 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
# High Availability Tempo Configuration for docker-compose-example-for-rustfs.yml
|
|
||||||
# Features:
|
|
||||||
# - Distributed architecture with multiple components
|
|
||||||
# - Kafka-based ingestion for fault tolerance
|
|
||||||
# - Replication factor of 3 for data resilience
|
|
||||||
# - Query frontend for load balancing
|
|
||||||
# - Metrics generation from traces
|
|
||||||
# - WAL for durability
|
|
||||||
|
|
||||||
partition_ring_live_store: true
|
|
||||||
stream_over_http_enabled: true
|
|
||||||
|
|
||||||
server:
|
|
||||||
http_listen_port: 3200
|
|
||||||
http_server_read_timeout: 30s
|
|
||||||
http_server_write_timeout: 30s
|
|
||||||
grpc_server_max_recv_msg_size: 4194304 # 4MB
|
|
||||||
grpc_server_max_send_msg_size: 4194304
|
|
||||||
log_level: info
|
|
||||||
log_format: json
|
|
||||||
|
|
||||||
# Memberlist configuration for distributed mode
|
|
||||||
memberlist:
|
|
||||||
node_name: tempo
|
|
||||||
bind_port: 7946
|
|
||||||
join_members:
|
|
||||||
- tempo:7946
|
|
||||||
retransmit_factor: 4
|
|
||||||
node_timeout: 15s
|
|
||||||
retransmit_interval: 300ms
|
|
||||||
dead_node_reclaim_time: 30s
|
|
||||||
|
|
||||||
# Distributor configuration - receives traces and routes to ingesters
|
|
||||||
distributor:
|
|
||||||
ingester_write_path_enabled: true
|
|
||||||
kafka_write_path_enabled: true
|
|
||||||
rate_limit_bytes: 10MB
|
|
||||||
rate_limit_enabled: true
|
|
||||||
receivers:
|
|
||||||
otlp:
|
|
||||||
protocols:
|
|
||||||
grpc:
|
|
||||||
endpoint: "0.0.0.0:4317"
|
|
||||||
max_concurrent_streams: 0
|
|
||||||
max_receive_message_size: 4194304
|
|
||||||
http:
|
|
||||||
endpoint: "0.0.0.0:4318"
|
|
||||||
cors:
|
|
||||||
allowed_origins:
|
|
||||||
- "*"
|
|
||||||
max_age: 86400
|
|
||||||
jaeger:
|
|
||||||
protocols:
|
|
||||||
grpc:
|
|
||||||
endpoint: "0.0.0.0:14250"
|
|
||||||
thrift_http:
|
|
||||||
endpoint: "0.0.0.0:14268"
|
|
||||||
zipkin:
|
|
||||||
endpoint: "0.0.0.0:9411"
|
|
||||||
ring:
|
|
||||||
kvstore:
|
|
||||||
store: memberlist
|
|
||||||
heartbeat_timeout: 5s
|
|
||||||
replication_factor: 3
|
|
||||||
heartbeat_interval: 5s
|
|
||||||
|
|
||||||
# Ingester configuration - stores traces and querying
|
|
||||||
ingester:
|
|
||||||
lifecycler:
|
|
||||||
address: tempo
|
|
||||||
ring:
|
|
||||||
kvstore:
|
|
||||||
store: memberlist
|
|
||||||
replication_factor: 3
|
|
||||||
max_cache_freshness_per_sec: 10s
|
|
||||||
heartbeat_interval: 5s
|
|
||||||
heartbeat_timeout: 5s
|
|
||||||
num_tokens: 128
|
|
||||||
tokens_file_path: /var/tempo/tokens.json
|
|
||||||
claim_on_rollout: true
|
|
||||||
trace_idle_period: 20s
|
|
||||||
max_block_bytes: 10_000_000
|
|
||||||
max_block_duration: 10m
|
|
||||||
chunk_size_bytes: 1_000_000
|
|
||||||
chunk_encoding: snappy
|
|
||||||
wal:
|
|
||||||
checkpoint_duration: 5s
|
|
||||||
max_wal_blocks: 4
|
|
||||||
metrics:
|
|
||||||
enabled: true
|
|
||||||
level: block
|
|
||||||
target_info_duration: 15m
|
|
||||||
|
|
||||||
# WAL configuration for data durability
|
|
||||||
wal:
|
|
||||||
checkpoint_duration: 5s
|
|
||||||
flush_on_shutdown: true
|
|
||||||
path: /var/tempo/wal
|
|
||||||
|
|
||||||
# Kafka ingestion configuration - for high throughput scenarios
|
|
||||||
ingest:
|
|
||||||
enabled: true
|
|
||||||
kafka:
|
|
||||||
brokers: [ redpanda:9092 ]
|
|
||||||
topic: tempo-ingest
|
|
||||||
encoding: protobuf
|
|
||||||
consumer_group: tempo-ingest-consumer
|
|
||||||
session_timeout: 10s
|
|
||||||
rebalance_timeout: 1m
|
|
||||||
partition: auto
|
|
||||||
verbosity: 2
|
|
||||||
|
|
||||||
# Query frontend configuration - distributed querying
|
|
||||||
query_frontend:
|
|
||||||
compression: gzip
|
|
||||||
downstream_url: http://localhost:3200
|
|
||||||
log_queries_longer_than: 5s
|
|
||||||
cache_uncompressed_bytes: 100MB
|
|
||||||
max_outstanding_requests_per_tenant: 100
|
|
||||||
max_query_length: 48h
|
|
||||||
max_query_lookback: 30d
|
|
||||||
default_result_cache_ttl: 1m
|
|
||||||
result_cache:
|
|
||||||
cache:
|
|
||||||
enable_fifocache: true
|
|
||||||
default_validity: 1m
|
|
||||||
rf1_after: "1999-01-01T00:00:00Z"
|
|
||||||
mcp_server:
|
|
||||||
enabled: true
|
|
||||||
|
|
||||||
# Querier configuration - queries traces
|
|
||||||
querier:
|
|
||||||
frontend_worker:
|
|
||||||
frontend_address: localhost:3200
|
|
||||||
grpc_client_config:
|
|
||||||
max_recv_msg_size: 104857600
|
|
||||||
max_concurrent_queries: 20
|
|
||||||
max_metric_bytes_per_trace: 1MB
|
|
||||||
|
|
||||||
# Query scheduler configuration - for distributed querying
|
|
||||||
query_scheduler:
|
|
||||||
use_scheduler_ring: false
|
|
||||||
|
|
||||||
# Metrics generator configuration - generates metrics from traces
|
|
||||||
metrics_generator:
|
|
||||||
enabled: true
|
|
||||||
registry:
|
|
||||||
enabled: true
|
|
||||||
external_labels:
|
|
||||||
source: tempo
|
|
||||||
cluster: rustfs-docker-ha
|
|
||||||
environment: production
|
|
||||||
storage:
|
|
||||||
path: /var/tempo/generator/wal
|
|
||||||
remote_write:
|
|
||||||
- url: http://prometheus:9090/api/v1/write
|
|
||||||
send_exemplars: true
|
|
||||||
resource_to_telemetry_conversion:
|
|
||||||
enabled: true
|
|
||||||
processor:
|
|
||||||
batch:
|
|
||||||
timeout: 10s
|
|
||||||
send_batch_size: 1024
|
|
||||||
memory_limiter:
|
|
||||||
check_interval: 5s
|
|
||||||
limit_mib: 512
|
|
||||||
spike_limit_mib: 128
|
|
||||||
processors:
|
|
||||||
- span-metrics
|
|
||||||
- local-blocks
|
|
||||||
- service-graphs
|
|
||||||
generate_native_histograms: both
|
|
||||||
|
|
||||||
# Backend worker configuration
|
|
||||||
backend_worker:
|
|
||||||
backend_scheduler_addr: localhost:3200
|
|
||||||
compaction:
|
|
||||||
block_retention: 24h
|
|
||||||
compacted_block_retention: 1h
|
|
||||||
ring:
|
|
||||||
kvstore:
|
|
||||||
store: memberlist
|
|
||||||
|
|
||||||
# Backend scheduler configuration
|
|
||||||
backend_scheduler:
|
|
||||||
enabled: true
|
|
||||||
provider:
|
|
||||||
compaction:
|
|
||||||
compaction:
|
|
||||||
block_retention: 24h
|
|
||||||
compacted_block_retention: 1h
|
|
||||||
concurrency: 25
|
|
||||||
v2_out_path: /var/tempo/blocks/compaction
|
|
||||||
|
|
||||||
# Storage configuration - local backend with proper retention
|
|
||||||
storage:
|
|
||||||
trace:
|
|
||||||
backend: local
|
|
||||||
wal:
|
|
||||||
path: /var/tempo/wal
|
|
||||||
checkpoint_duration: 5s
|
|
||||||
flush_on_shutdown: true
|
|
||||||
local:
|
|
||||||
path: /var/tempo/blocks
|
|
||||||
bloom_filter_false_positive: 0.05
|
|
||||||
bloom_shift: 4
|
|
||||||
index:
|
|
||||||
downsample_bytes: 1000000
|
|
||||||
page_size_bytes: 0
|
|
||||||
cache_size_bytes: 0
|
|
||||||
pool:
|
|
||||||
max_workers: 400
|
|
||||||
queue_depth: 10000
|
|
||||||
|
|
||||||
# Compactor configuration - manages block compaction
|
|
||||||
compactor:
|
|
||||||
compaction:
|
|
||||||
block_retention: 168h # 7 days
|
|
||||||
compacted_block_retention: 1h
|
|
||||||
concurrency: 25
|
|
||||||
v2_out_path: /var/tempo/blocks/compaction
|
|
||||||
shard_count: 32
|
|
||||||
max_block_bytes: 107374182400 # 100GB
|
|
||||||
max_compaction_objects: 6000000
|
|
||||||
max_time_per_tenant: 5m
|
|
||||||
block_size_bytes: 107374182400
|
|
||||||
ring:
|
|
||||||
kvstore:
|
|
||||||
store: memberlist
|
|
||||||
heartbeat_interval: 5s
|
|
||||||
heartbeat_timeout: 5s
|
|
||||||
|
|
||||||
# Limits configuration - rate limiting and quotas
|
|
||||||
limits:
|
|
||||||
max_traces_per_user: 10000
|
|
||||||
max_bytes_per_trace: 10485760 # 10MB
|
|
||||||
max_search_bytes_per_trace: 0
|
|
||||||
forgiving_oversize_traces: true
|
|
||||||
rate_limit_bytes: 10MB
|
|
||||||
rate_limit_enabled: true
|
|
||||||
ingestion_burst_size_bytes: 20MB
|
|
||||||
ingestion_rate_limit_bytes: 10MB
|
|
||||||
max_bytes_per_second: 10485760
|
|
||||||
metrics_generator_max_active_series: 10000
|
|
||||||
metrics_generator_max_churned_series: 10000
|
|
||||||
metrics_generator_forta_out_of_order_ttl: 5m
|
|
||||||
|
|
||||||
# Override configuration
|
|
||||||
overrides:
|
|
||||||
defaults:
|
|
||||||
metrics_generator:
|
|
||||||
processors:
|
|
||||||
- span-metrics
|
|
||||||
- local-blocks
|
|
||||||
- service-graphs
|
|
||||||
generate_native_histograms: both
|
|
||||||
max_active_series: 10000
|
|
||||||
max_churned_series: 10000
|
|
||||||
|
|
||||||
# Usage reporting configuration
|
|
||||||
usage_report:
|
|
||||||
reporting_enabled: false
|
|
||||||
|
|
||||||
# Tracing configuration for debugging
|
|
||||||
tracing:
|
|
||||||
enabled: true
|
|
||||||
jaeger:
|
|
||||||
sampler:
|
|
||||||
name: probabilistic
|
|
||||||
param: 0.1
|
|
||||||
reporter_log_spans: false
|
|
||||||
|
|
||||||
@@ -1,72 +1,55 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
stream_over_http_enabled: true
|
stream_over_http_enabled: true
|
||||||
|
|
||||||
server:
|
server:
|
||||||
http_listen_port: 3200
|
http_listen_port: 3200
|
||||||
log_level: info
|
log_level: info
|
||||||
|
|
||||||
memberlist:
|
query_frontend:
|
||||||
node_name: tempo
|
search:
|
||||||
bind_port: 7946
|
duration_slo: 5s
|
||||||
join_members:
|
throughput_bytes_slo: 1.073741824e+09
|
||||||
- tempo:7946
|
metadata_slo:
|
||||||
|
duration_slo: 5s
|
||||||
|
throughput_bytes_slo: 1.073741824e+09
|
||||||
|
trace_by_id:
|
||||||
|
duration_slo: 5s
|
||||||
|
|
||||||
distributor:
|
distributor:
|
||||||
receivers:
|
receivers:
|
||||||
otlp:
|
otlp:
|
||||||
protocols:
|
protocols:
|
||||||
grpc:
|
grpc:
|
||||||
endpoint: "0.0.0.0:4317"
|
endpoint: "tempo:4317"
|
||||||
http:
|
|
||||||
endpoint: "0.0.0.0:4318"
|
|
||||||
|
|
||||||
ingester:
|
ingester:
|
||||||
max_block_duration: 5m
|
max_block_duration: 5m # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally
|
||||||
|
|
||||||
|
compactor:
|
||||||
|
compaction:
|
||||||
|
block_retention: 1h # overall Tempo trace retention. set for demo purposes
|
||||||
|
|
||||||
metrics_generator:
|
metrics_generator:
|
||||||
registry:
|
registry:
|
||||||
external_labels:
|
external_labels:
|
||||||
source: tempo
|
source: tempo
|
||||||
cluster: docker-compose
|
cluster: docker-compose
|
||||||
traces_storage:
|
|
||||||
path: /var/tempo/generator/traces
|
|
||||||
storage:
|
storage:
|
||||||
path: /var/tempo/generator/wal
|
path: /var/tempo/generator/wal
|
||||||
remote_write:
|
remote_write:
|
||||||
- url: http://prometheus:9090/api/v1/write
|
- url: http://prometheus:9090/api/v1/write
|
||||||
send_exemplars: true
|
send_exemplars: true
|
||||||
|
traces_storage:
|
||||||
query_frontend:
|
path: /var/tempo/generator/traces
|
||||||
rf1_after: "1999-01-01T00:00:00Z"
|
|
||||||
mcp_server:
|
|
||||||
enabled: true
|
|
||||||
|
|
||||||
storage:
|
storage:
|
||||||
trace:
|
trace:
|
||||||
backend: local
|
backend: local # backend configuration to use
|
||||||
wal:
|
wal:
|
||||||
path: /var/tempo/wal # where to store the wal locally
|
path: /var/tempo/wal # where to store the wal locally
|
||||||
local:
|
local:
|
||||||
path: /var/tempo/blocks # where to store the traces locally
|
path: /var/tempo/blocks
|
||||||
|
|
||||||
overrides:
|
overrides:
|
||||||
defaults:
|
defaults:
|
||||||
metrics_generator:
|
metrics_generator:
|
||||||
processors: [ "span-metrics", "service-graphs", "local-blocks" ]
|
processors: [ service-graphs, span-metrics, local-blocks ] # enables metrics generator
|
||||||
generate_native_histograms: both
|
generate_native_histograms: both
|
||||||
|
|
||||||
usage_report:
|
|
||||||
reporting_enabled: false
|
|
||||||
|
|||||||
@@ -5,57 +5,71 @@
|
|||||||
|
|
||||||
English | [中文](README_ZH.md)
|
English | [中文](README_ZH.md)
|
||||||
|
|
||||||
This directory contains the configuration for an **alternative** observability stack using OpenObserve.
|
This directory contains the configuration files for setting up an observability stack with OpenObserve and OpenTelemetry
|
||||||
|
Collector.
|
||||||
|
|
||||||
## ⚠️ Note
|
### Overview
|
||||||
|
|
||||||
For the **recommended** observability stack (Prometheus, Grafana, Tempo, Loki), please see `../observability/`.
|
This setup provides a complete observability solution for your applications:
|
||||||
|
|
||||||
## 🌟 Overview
|
- **OpenObserve**: A modern, open-source observability platform for logs, metrics, and traces.
|
||||||
|
- **OpenTelemetry Collector**: Collects and processes telemetry data before sending it to OpenObserve.
|
||||||
|
|
||||||
OpenObserve is a lightweight, all-in-one observability platform that handles logs, metrics, and traces in a single binary. This setup is ideal for:
|
### Setup Instructions
|
||||||
- Resource-constrained environments.
|
|
||||||
- Quick setup and testing.
|
|
||||||
- Users who prefer a unified UI.
|
|
||||||
|
|
||||||
## 🚀 Quick Start
|
1. **Prerequisites**:
|
||||||
|
- Docker and Docker Compose installed
|
||||||
|
- Sufficient memory resources (minimum 2GB recommended)
|
||||||
|
|
||||||
### 1. Start Services
|
2. **Starting the Services**:
|
||||||
|
```bash
|
||||||
|
cd .docker/openobserve-otel
|
||||||
|
docker compose -f docker-compose.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Accessing the Dashboard**:
|
||||||
|
- OpenObserve UI: http://localhost:5080
|
||||||
|
- Default credentials:
|
||||||
|
- Username: root@rustfs.com
|
||||||
|
- Password: rustfs123
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
#### OpenObserve Configuration
|
||||||
|
|
||||||
|
The OpenObserve service is configured with:
|
||||||
|
|
||||||
|
- Root user credentials
|
||||||
|
- Data persistence through a volume mount
|
||||||
|
- Memory cache enabled
|
||||||
|
- Health checks
|
||||||
|
- Exposed ports:
|
||||||
|
- 5080: HTTP API and UI
|
||||||
|
- 5081: OTLP gRPC
|
||||||
|
|
||||||
|
#### OpenTelemetry Collector Configuration
|
||||||
|
|
||||||
|
The collector is configured to:
|
||||||
|
|
||||||
|
- Receive telemetry data via OTLP (HTTP and gRPC)
|
||||||
|
- Collect logs from files
|
||||||
|
- Process data in batches
|
||||||
|
- Export data to OpenObserve
|
||||||
|
- Manage memory usage
|
||||||
|
|
||||||
|
### Integration with Your Application
|
||||||
|
|
||||||
|
To send telemetry data from your application, configure your OpenTelemetry SDK to send data to:
|
||||||
|
|
||||||
|
- OTLP gRPC: `localhost:4317`
|
||||||
|
- OTLP HTTP: `localhost:4318`
|
||||||
|
|
||||||
|
For example, in a Rust application using the `rustfs-obs` library:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd .docker/openobserve-otel
|
export RUSTFS_OBS_ENDPOINT=http://localhost:4317
|
||||||
docker compose up -d
|
export RUSTFS_OBS_SERVICE_NAME=yourservice
|
||||||
|
export RUSTFS_OBS_SERVICE_VERSION=1.0.0
|
||||||
|
export RUSTFS_OBS_ENVIRONMENT=development
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Access Dashboard
|
|
||||||
|
|
||||||
- **URL**: [http://localhost:5080](http://localhost:5080)
|
|
||||||
- **Username**: `root@rustfs.com`
|
|
||||||
- **Password**: `rustfs123`
|
|
||||||
|
|
||||||
## 🛠️ Configuration
|
|
||||||
|
|
||||||
### OpenObserve
|
|
||||||
|
|
||||||
- **Persistence**: Data is persisted to a Docker volume.
|
|
||||||
- **Ports**:
|
|
||||||
- `5080`: HTTP API and UI
|
|
||||||
- `5081`: OTLP gRPC
|
|
||||||
|
|
||||||
### OpenTelemetry Collector
|
|
||||||
|
|
||||||
- **Receivers**: OTLP (gRPC `4317`, HTTP `4318`)
|
|
||||||
- **Exporters**: Sends data to OpenObserve.
|
|
||||||
|
|
||||||
## 🔗 Integration
|
|
||||||
|
|
||||||
Configure your application to send OTLP data to the collector:
|
|
||||||
|
|
||||||
- **Endpoint**: `http://localhost:4318` (HTTP) or `localhost:4317` (gRPC)
|
|
||||||
|
|
||||||
Example for RustFS:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export RUSTFS_OBS_ENDPOINT=http://localhost:4318
|
|
||||||
export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
|
|
||||||
```
|
|
||||||
|
|||||||
@@ -5,57 +5,71 @@
|
|||||||
|
|
||||||
[English](README.md) | 中文
|
[English](README.md) | 中文
|
||||||
|
|
||||||
本目录包含使用 OpenObserve 的**替代**可观测性技术栈配置。
|
## 中文
|
||||||
|
|
||||||
## ⚠️ 注意
|
本目录包含搭建 OpenObserve 和 OpenTelemetry Collector 可观测性栈的配置文件。
|
||||||
|
|
||||||
对于**推荐**的可观测性技术栈（Prometheus, Grafana, Tempo, Loki），请参阅 `../observability/`。
|
### 概述
|
||||||
|
|
||||||
## 🌟 概览
|
此设置为应用程序提供了完整的可观测性解决方案：
|
||||||
|
|
||||||
OpenObserve 是一个轻量级、一体化的可观测性平台，在一个二进制文件中处理日志、指标和追踪。此设置非常适合：
|
- **OpenObserve**：现代化、开源的可观测性平台，用于日志、指标和追踪。
|
||||||
- 资源受限的环境。
|
- **OpenTelemetry Collector**：收集和处理遥测数据，然后将其发送到 OpenObserve。
|
||||||
- 快速设置和测试。
|
|
||||||
- 喜欢统一 UI 的用户。
|
|
||||||
|
|
||||||
## 🚀 快速开始
|
### 设置说明
|
||||||
|
|
||||||
### 1. 启动服务
|
1. **前提条件**：
|
||||||
|
- 已安装 Docker 和 Docker Compose
|
||||||
|
- 足够的内存资源（建议至少 2GB）
|
||||||
|
|
||||||
|
2. **启动服务**：
|
||||||
|
```bash
|
||||||
|
cd .docker/openobserve-otel
|
||||||
|
docker compose -f docker-compose.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **访问仪表板**：
|
||||||
|
- OpenObserve UI：http://localhost:5080
|
||||||
|
- 默认凭据：
|
||||||
|
- 用户名：root@rustfs.com
|
||||||
|
- 密码：rustfs123
|
||||||
|
|
||||||
|
### 配置
|
||||||
|
|
||||||
|
#### OpenObserve 配置
|
||||||
|
|
||||||
|
OpenObserve 服务配置：
|
||||||
|
|
||||||
|
- 根用户凭据
|
||||||
|
- 通过卷挂载实现数据持久化
|
||||||
|
- 启用内存缓存
|
||||||
|
- 健康检查
|
||||||
|
- 暴露端口：
|
||||||
|
- 5080：HTTP API 和 UI
|
||||||
|
- 5081：OTLP gRPC
|
||||||
|
|
||||||
|
#### OpenTelemetry Collector 配置
|
||||||
|
|
||||||
|
收集器配置为：
|
||||||
|
|
||||||
|
- 通过 OTLP（HTTP 和 gRPC）接收遥测数据
|
||||||
|
- 从文件中收集日志
|
||||||
|
- 批处理数据
|
||||||
|
- 将数据导出到 OpenObserve
|
||||||
|
- 管理内存使用
|
||||||
|
|
||||||
|
### 与应用程序集成
|
||||||
|
|
||||||
|
要从应用程序发送遥测数据，将 OpenTelemetry SDK 配置为发送数据到：
|
||||||
|
|
||||||
|
- OTLP gRPC:`localhost:4317`
|
||||||
|
- OTLP HTTP:`localhost:4318`
|
||||||
|
|
||||||
|
例如，在使用 `rustfs-obs` 库的 Rust 应用程序中：
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd .docker/openobserve-otel
|
export RUSTFS_OBS_ENDPOINT=http://localhost:4317
|
||||||
docker compose up -d
|
export RUSTFS_OBS_SERVICE_NAME=yourservice
|
||||||
```
|
export RUSTFS_OBS_SERVICE_VERSION=1.0.0
|
||||||
|
export RUSTFS_OBS_ENVIRONMENT=development
|
||||||
### 2. 访问仪表盘
|
|
||||||
|
|
||||||
- **URL**: [http://localhost:5080](http://localhost:5080)
|
|
||||||
- **用户名**: `root@rustfs.com`
|
|
||||||
- **密码**: `rustfs123`
|
|
||||||
|
|
||||||
## 🛠️ 配置
|
|
||||||
|
|
||||||
### OpenObserve
|
|
||||||
|
|
||||||
- **持久化**: 数据持久化到 Docker 卷。
|
|
||||||
- **端口**:
|
|
||||||
- `5080`: HTTP API 和 UI
|
|
||||||
- `5081`: OTLP gRPC
|
|
||||||
|
|
||||||
### OpenTelemetry Collector
|
|
||||||
|
|
||||||
- **接收器**: OTLP (gRPC `4317`, HTTP `4318`)
|
|
||||||
- **导出器**: 将数据发送到 OpenObserve。
|
|
||||||
|
|
||||||
## 🔗 集成
|
|
||||||
|
|
||||||
配置您的应用程序将 OTLP 数据发送到收集器：
|
|
||||||
|
|
||||||
- **端点**: `http://localhost:4318` (HTTP) 或 `localhost:4317` (gRPC)
|
|
||||||
|
|
||||||
RustFS 示例：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export RUSTFS_OBS_ENDPOINT=http://localhost:4318
|
|
||||||
export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
|
|
||||||
```
|
```
|
||||||
@@ -1,91 +0,0 @@
|
|||||||
# Rio compatibility compose files
|
|
||||||
|
|
||||||
These compose files prepare 4-node, 4-disk clusters for rio/rio-v2 storage format compatibility checks. All disks are bind-mounted under `.docker/compat/data` so the on-disk files remain available on the host.
|
|
||||||
|
|
||||||
## Clusters
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d --build
|
|
||||||
docker compose -f .docker/compat/docker-compose.minio.yml up -d
|
|
||||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
|
||||||
```
|
|
||||||
|
|
||||||
Default API endpoints:
|
|
||||||
|
|
||||||
- RustFS `1.0.0-beta.5`: `http://127.0.0.1:9100`
|
|
||||||
- MinIO: `http://127.0.0.1:9200`
|
|
||||||
- current main with `rio-v2`: `http://127.0.0.1:9300`
|
|
||||||
|
|
||||||
## Reading old datasets with rio-v2
|
|
||||||
|
|
||||||
Stop the writer cluster before mounting its disks into the rio-v2 cluster.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml down
|
|
||||||
RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
|
|
||||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
|
||||||
|
|
||||||
docker compose -f .docker/compat/docker-compose.minio.yml down
|
|
||||||
RUSTFS_RIO_V2_DATASET=./data/minio \
|
|
||||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
|
||||||
```
|
|
||||||
|
|
||||||
## 200G object mix
|
|
||||||
|
|
||||||
Use the same bucket/object matrix against the beta5 and MinIO endpoints, then read it back through the rio-v2 endpoint. A practical 200G mix is:
|
|
||||||
|
|
||||||
- 1 KiB x 1024
|
|
||||||
- 1 MiB x 1024
|
|
||||||
- 64 MiB x 512
|
|
||||||
- 1 GiB x 64
|
|
||||||
- 8 GiB x 12
|
|
||||||
- 6 GiB x 1
|
|
||||||
|
|
||||||
Compression is enabled by default for RustFS and MinIO. Server-side KMS/SSE settings are intentionally left to environment variables or mounted key directories so real key material is not committed. For SSE-C cases, run the clusters with TLS because MinIO requires HTTPS for SSE-C.
|
|
||||||
|
|
||||||
## High-concurrency write/read stress
|
|
||||||
|
|
||||||
Use `run_rw_compat_stress.sh` to generate a manifest on an old endpoint, then verify the same objects through the rio-v2 endpoint after mounting the old disks.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
.docker/compat/run_rw_compat_stress.sh \
|
|
||||||
--mode write \
|
|
||||||
--endpoint http://127.0.0.1:9100 \
|
|
||||||
--access-key rustfsadmin \
|
|
||||||
--secret-key rustfsadmin \
|
|
||||||
--bucket compat-beta5 \
|
|
||||||
--concurrency 96
|
|
||||||
|
|
||||||
RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
|
|
||||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
|
||||||
|
|
||||||
.docker/compat/run_rw_compat_stress.sh \
|
|
||||||
--mode verify \
|
|
||||||
--endpoint http://127.0.0.1:9300 \
|
|
||||||
--access-key rustfsadmin \
|
|
||||||
--secret-key rustfsadmin \
|
|
||||||
--bucket compat-beta5 \
|
|
||||||
--concurrency 96 \
|
|
||||||
--manifest target/compat/rw-stress-YYYYmmdd-HHMMSS/manifest.csv
|
|
||||||
```
|
|
||||||
|
|
||||||
For encrypted datasets, add `--encryption sse-s3`, `--encryption sse-kms --sse-kms-key-id <key-id>`, or `--encryption sse-c --sse-c-key-file <raw-32-byte-key-file>` to both the write and verify commands.
|
|
||||||
|
|
||||||
## 5 GiB encrypted compatibility run
|
|
||||||
|
|
||||||
The `5g` profile covers 1 KiB, 1 MiB, 16 MiB, 64 MiB, and 1 GiB objects and totals exactly 5 GiB. Generate `compat-key.key` under `.docker/compat/kms/rustfs-compat`, enable local KMS on both RustFS clusters, then use the same encryption arguments while writing with beta5 and verifying with rio-v2. Set non-default local test credentials first because distributed listeners reject the built-in default credentials.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export COMPAT_ACCESS_KEY='<non-default-access-key>'
|
|
||||||
export COMPAT_SECRET_KEY='<non-default-secret-key>'
|
|
||||||
|
|
||||||
RUSTFS_ACCESS_KEY="$COMPAT_ACCESS_KEY" RUSTFS_SECRET_KEY="$COMPAT_SECRET_KEY" \
|
|
||||||
RUSTFS_KMS_ENABLE=true RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true \
|
|
||||||
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d
|
|
||||||
|
|
||||||
.docker/compat/run_rw_compat_stress.sh \
|
|
||||||
--mode write --endpoint http://127.0.0.1:9100 \
|
|
||||||
--access-key "$COMPAT_ACCESS_KEY" --secret-key "$COMPAT_SECRET_KEY" \
|
|
||||||
--bucket compat-beta5-sse-s3 --profile 5g --concurrency 16 \
|
|
||||||
--encryption sse-s3
|
|
||||||
```
|
|
||||||
@@ -1,88 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0.
|
|
||||||
|
|
||||||
x-minio-env: &minio-env
|
|
||||||
MINIO_ROOT_USER: "${MINIO_ROOT_USER:-minioadmin}"
|
|
||||||
MINIO_ROOT_PASSWORD: "${MINIO_ROOT_PASSWORD:-minioadmin}"
|
|
||||||
MINIO_COMPRESSION_ENABLE: "${MINIO_COMPRESSION_ENABLE:-on}"
|
|
||||||
MINIO_COMPRESSION_ALLOW_ENCRYPTION: "${MINIO_COMPRESSION_ALLOW_ENCRYPTION:-on}"
|
|
||||||
MINIO_COMPRESSION_EXTENSIONS: "${MINIO_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
|
|
||||||
MINIO_COMPRESSION_MIME_TYPES: "${MINIO_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
|
|
||||||
|
|
||||||
x-minio-node: &minio-node
|
|
||||||
image: "${MINIO_IMAGE:-quay.io/minio/minio:latest}"
|
|
||||||
command: server --console-address ":9001" "http://minio{1...4}:9000/data/disk{1...4}"
|
|
||||||
environment: *minio-env
|
|
||||||
networks:
|
|
||||||
- minio-compat-net
|
|
||||||
restart: unless-stopped
|
|
||||||
|
|
||||||
services:
|
|
||||||
minio-permission-helper:
|
|
||||||
image: alpine:3.23
|
|
||||||
command: sh -c "mkdir -p /compat-data && chown -R 1000:1000 /compat-data"
|
|
||||||
volumes:
|
|
||||||
- ./data/minio:/compat-data
|
|
||||||
restart: "no"
|
|
||||||
|
|
||||||
minio1:
|
|
||||||
<<: *minio-node
|
|
||||||
hostname: minio1
|
|
||||||
depends_on:
|
|
||||||
minio-permission-helper:
|
|
||||||
condition: service_completed_successfully
|
|
||||||
volumes:
|
|
||||||
- ./data/minio/node1/disk1:/data/disk1
|
|
||||||
- ./data/minio/node1/disk2:/data/disk2
|
|
||||||
- ./data/minio/node1/disk3:/data/disk3
|
|
||||||
- ./data/minio/node1/disk4:/data/disk4
|
|
||||||
ports:
|
|
||||||
- "${MINIO_API_PORT:-9200}:9000"
|
|
||||||
- "${MINIO_CONSOLE_PORT:-9201}:9001"
|
|
||||||
|
|
||||||
minio2:
|
|
||||||
<<: *minio-node
|
|
||||||
hostname: minio2
|
|
||||||
depends_on:
|
|
||||||
minio-permission-helper:
|
|
||||||
condition: service_completed_successfully
|
|
||||||
volumes:
|
|
||||||
- ./data/minio/node2/disk1:/data/disk1
|
|
||||||
- ./data/minio/node2/disk2:/data/disk2
|
|
||||||
- ./data/minio/node2/disk3:/data/disk3
|
|
||||||
- ./data/minio/node2/disk4:/data/disk4
|
|
||||||
ports:
|
|
||||||
- "${MINIO_NODE2_PORT:-9202}:9000"
|
|
||||||
|
|
||||||
minio3:
|
|
||||||
<<: *minio-node
|
|
||||||
hostname: minio3
|
|
||||||
depends_on:
|
|
||||||
minio-permission-helper:
|
|
||||||
condition: service_completed_successfully
|
|
||||||
volumes:
|
|
||||||
- ./data/minio/node3/disk1:/data/disk1
|
|
||||||
- ./data/minio/node3/disk2:/data/disk2
|
|
||||||
- ./data/minio/node3/disk3:/data/disk3
|
|
||||||
- ./data/minio/node3/disk4:/data/disk4
|
|
||||||
ports:
|
|
||||||
- "${MINIO_NODE3_PORT:-9203}:9000"
|
|
||||||
|
|
||||||
minio4:
|
|
||||||
<<: *minio-node
|
|
||||||
hostname: minio4
|
|
||||||
depends_on:
|
|
||||||
minio-permission-helper:
|
|
||||||
condition: service_completed_successfully
|
|
||||||
volumes:
|
|
||||||
- ./data/minio/node4/disk1:/data/disk1
|
|
||||||
- ./data/minio/node4/disk2:/data/disk2
|
|
||||||
- ./data/minio/node4/disk3:/data/disk3
|
|
||||||
- ./data/minio/node4/disk4:/data/disk4
|
|
||||||
ports:
|
|
||||||
- "${MINIO_NODE4_PORT:-9204}:9000"
|
|
||||||
|
|
||||||
networks:
|
|
||||||
minio-compat-net:
|
|
||||||
driver: bridge
|
|
||||||
@@ -1,104 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0.
|
|
||||||
|
|
||||||
x-rustfs-env: &rustfs-env
|
|
||||||
RUSTFS_VOLUMES: "http://rustfs-beta5-node{1...4}:9000/data/disk{1...4}"
|
|
||||||
RUSTFS_ADDRESS: ":9000"
|
|
||||||
RUSTFS_CONSOLE_ADDRESS: ":9001"
|
|
||||||
RUSTFS_CONSOLE_ENABLE: "true"
|
|
||||||
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
|
|
||||||
RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
|
|
||||||
RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
|
|
||||||
RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
|
|
||||||
RUSTFS_OBS_LOG_DIRECTORY: "/logs"
|
|
||||||
RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
|
|
||||||
RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
|
|
||||||
RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
|
|
||||||
RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
|
|
||||||
RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
|
|
||||||
RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
|
|
||||||
RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
|
|
||||||
RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
|
|
||||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
|
|
||||||
|
|
||||||
x-rustfs-node: &rustfs-node
|
|
||||||
image: "${RUSTFS_BETA5_IMAGE:-rustfs/rustfs:1.0.0-beta.5}"
|
|
||||||
build:
|
|
||||||
context: ../..
|
|
||||||
dockerfile: Dockerfile
|
|
||||||
args:
|
|
||||||
RELEASE: "1.0.0-beta.5"
|
|
||||||
environment: *rustfs-env
|
|
||||||
depends_on:
|
|
||||||
rustfs-beta5-permission-helper:
|
|
||||||
condition: service_completed_successfully
|
|
||||||
networks:
|
|
||||||
- rustfs-beta5-net
|
|
||||||
restart: unless-stopped
|
|
||||||
|
|
||||||
services:
|
|
||||||
rustfs-beta5-permission-helper:
|
|
||||||
image: alpine:3.23
|
|
||||||
command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
|
|
||||||
volumes:
|
|
||||||
- ./data/rustfs-beta5:/compat-data
|
|
||||||
- ./kms/rustfs-compat:/kms
|
|
||||||
restart: "no"
|
|
||||||
|
|
||||||
rustfs-beta5-node1:
|
|
||||||
<<: *rustfs-node
|
|
||||||
hostname: rustfs-beta5-node1
|
|
||||||
volumes:
|
|
||||||
- ./data/rustfs-beta5/node1/disk1:/data/disk1
|
|
||||||
- ./data/rustfs-beta5/node1/disk2:/data/disk2
|
|
||||||
- ./data/rustfs-beta5/node1/disk3:/data/disk3
|
|
||||||
- ./data/rustfs-beta5/node1/disk4:/data/disk4
|
|
||||||
- ./data/rustfs-beta5/logs/node1:/logs
|
|
||||||
- ./kms/rustfs-compat:/kms
|
|
||||||
ports:
|
|
||||||
- "${RUSTFS_BETA5_API_PORT:-9100}:9000"
|
|
||||||
- "${RUSTFS_BETA5_CONSOLE_PORT:-9101}:9001"
|
|
||||||
|
|
||||||
rustfs-beta5-node2:
|
|
||||||
<<: *rustfs-node
|
|
||||||
hostname: rustfs-beta5-node2
|
|
||||||
volumes:
|
|
||||||
- ./data/rustfs-beta5/node2/disk1:/data/disk1
|
|
||||||
- ./data/rustfs-beta5/node2/disk2:/data/disk2
|
|
||||||
- ./data/rustfs-beta5/node2/disk3:/data/disk3
|
|
||||||
- ./data/rustfs-beta5/node2/disk4:/data/disk4
|
|
||||||
- ./data/rustfs-beta5/logs/node2:/logs
|
|
||||||
- ./kms/rustfs-compat:/kms
|
|
||||||
ports:
|
|
||||||
- "${RUSTFS_BETA5_NODE2_PORT:-9102}:9000"
|
|
||||||
|
|
||||||
rustfs-beta5-node3:
|
|
||||||
<<: *rustfs-node
|
|
||||||
hostname: rustfs-beta5-node3
|
|
||||||
volumes:
|
|
||||||
- ./data/rustfs-beta5/node3/disk1:/data/disk1
|
|
||||||
- ./data/rustfs-beta5/node3/disk2:/data/disk2
|
|
||||||
- ./data/rustfs-beta5/node3/disk3:/data/disk3
|
|
||||||
- ./data/rustfs-beta5/node3/disk4:/data/disk4
|
|
||||||
- ./data/rustfs-beta5/logs/node3:/logs
|
|
||||||
- ./kms/rustfs-compat:/kms
|
|
||||||
ports:
|
|
||||||
- "${RUSTFS_BETA5_NODE3_PORT:-9103}:9000"
|
|
||||||
|
|
||||||
rustfs-beta5-node4:
|
|
||||||
<<: *rustfs-node
|
|
||||||
hostname: rustfs-beta5-node4
|
|
||||||
volumes:
|
|
||||||
- ./data/rustfs-beta5/node4/disk1:/data/disk1
|
|
||||||
- ./data/rustfs-beta5/node4/disk2:/data/disk2
|
|
||||||
- ./data/rustfs-beta5/node4/disk3:/data/disk3
|
|
||||||
- ./data/rustfs-beta5/node4/disk4:/data/disk4
|
|
||||||
- ./data/rustfs-beta5/logs/node4:/logs
|
|
||||||
- ./kms/rustfs-compat:/kms
|
|
||||||
ports:
|
|
||||||
- "${RUSTFS_BETA5_NODE4_PORT:-9104}:9000"
|
|
||||||
|
|
||||||
networks:
|
|
||||||
rustfs-beta5-net:
|
|
||||||
driver: bridge
|
|
||||||
@@ -1,115 +0,0 @@
|
|||||||
# Copyright 2024 RustFS Team
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0.
|
|
||||||
|
|
||||||
x-rustfs-rio-v2-env: &rustfs-rio-v2-env
|
|
||||||
RUSTFS_VOLUMES: "http://rustfs-rio-v2-node{1...4}:9000/data/disk{1...4}"
|
|
||||||
RUSTFS_ADDRESS: ":9000"
|
|
||||||
RUSTFS_CONSOLE_ADDRESS: ":9001"
|
|
||||||
RUSTFS_CONSOLE_ENABLE: "true"
|
|
||||||
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
|
|
||||||
RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
|
|
||||||
RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
|
|
||||||
RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
|
|
||||||
RUSTFS_OBS_LOG_DIRECTORY: "/logs"
|
|
||||||
RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
|
|
||||||
RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
|
|
||||||
RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
|
|
||||||
RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
|
|
||||||
RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
|
|
||||||
RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
|
|
||||||
RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
|
|
||||||
RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
|
|
||||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
|
|
||||||
|
|
||||||
x-rustfs-rio-v2-node: &rustfs-rio-v2-node
|
|
||||||
image: "${RUSTFS_RIO_V2_IMAGE:-rustfs/rustfs:compat-rio-v2}"
|
|
||||||
entrypoint:
|
|
||||||
- /bin/sh
|
|
||||||
- -c
|
|
||||||
- |
|
|
||||||
until getent hosts rustfs-rio-v2-node1 >/dev/null &&
|
|
||||||
getent hosts rustfs-rio-v2-node2 >/dev/null &&
|
|
||||||
getent hosts rustfs-rio-v2-node3 >/dev/null &&
|
|
||||||
getent hosts rustfs-rio-v2-node4 >/dev/null; do
|
|
||||||
sleep 1
|
|
||||||
done
|
|
||||||
exec /entrypoint.sh /usr/bin/rustfs
|
|
||||||
build:
|
|
||||||
context: ../..
|
|
||||||
dockerfile: Dockerfile.source
|
|
||||||
args:
|
|
||||||
RUSTFS_BUILD_FEATURES: "rio-v2"
|
|
||||||
environment: *rustfs-rio-v2-env
|
|
||||||
depends_on:
|
|
||||||
rustfs-rio-v2-permission-helper:
|
|
||||||
condition: service_completed_successfully
|
|
||||||
networks:
|
|
||||||
- rustfs-rio-v2-net
|
|
||||||
restart: unless-stopped
|
|
||||||
|
|
||||||
services:
|
|
||||||
rustfs-rio-v2-permission-helper:
|
|
||||||
image: alpine:3.23
|
|
||||||
command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
|
|
||||||
volumes:
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}:/compat-data
|
|
||||||
- ./kms/rustfs-compat:/kms
|
|
||||||
restart: "no"
|
|
||||||
|
|
||||||
rustfs-rio-v2-node1:
|
|
||||||
<<: *rustfs-rio-v2-node
|
|
||||||
hostname: rustfs-rio-v2-node1
|
|
||||||
volumes:
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk1:/data/disk1
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk2:/data/disk2
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk3:/data/disk3
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk4:/data/disk4
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node1:/logs
|
|
||||||
- ./kms/rustfs-compat:/kms
|
|
||||||
ports:
|
|
||||||
- "${RUSTFS_RIO_V2_API_PORT:-9300}:9000"
|
|
||||||
- "${RUSTFS_RIO_V2_CONSOLE_PORT:-9301}:9001"
|
|
||||||
|
|
||||||
rustfs-rio-v2-node2:
|
|
||||||
<<: *rustfs-rio-v2-node
|
|
||||||
hostname: rustfs-rio-v2-node2
|
|
||||||
volumes:
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk1:/data/disk1
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk2:/data/disk2
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk3:/data/disk3
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk4:/data/disk4
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node2:/logs
|
|
||||||
- ./kms/rustfs-compat:/kms
|
|
||||||
ports:
|
|
||||||
- "${RUSTFS_RIO_V2_NODE2_PORT:-9302}:9000"
|
|
||||||
|
|
||||||
rustfs-rio-v2-node3:
|
|
||||||
<<: *rustfs-rio-v2-node
|
|
||||||
hostname: rustfs-rio-v2-node3
|
|
||||||
volumes:
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk1:/data/disk1
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk2:/data/disk2
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk3:/data/disk3
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk4:/data/disk4
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node3:/logs
|
|
||||||
- ./kms/rustfs-compat:/kms
|
|
||||||
ports:
|
|
||||||
- "${RUSTFS_RIO_V2_NODE3_PORT:-9303}:9000"
|
|
||||||
|
|
||||||
rustfs-rio-v2-node4:
|
|
||||||
<<: *rustfs-rio-v2-node
|
|
||||||
hostname: rustfs-rio-v2-node4
|
|
||||||
volumes:
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk1:/data/disk1
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk2:/data/disk2
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk3:/data/disk3
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk4:/data/disk4
|
|
||||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node4:/logs
|
|
||||||
- ./kms/rustfs-compat:/kms
|
|
||||||
ports:
|
|
||||||
- "${RUSTFS_RIO_V2_NODE4_PORT:-9304}:9000"
|
|
||||||
|
|
||||||
networks:
|
|
||||||
rustfs-rio-v2-net:
|
|
||||||
driver: bridge
|
|
||||||
@@ -1,639 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
# High-concurrency S3 read/write stress runner for rio/rio-v2 format compatibility.
|
|
||||||
# Write a manifest on an old endpoint, then verify the same manifest through rio-v2.
|
|
||||||
|
|
||||||
MODE="write"
|
|
||||||
ENDPOINT=""
|
|
||||||
ACCESS_KEY="${AWS_ACCESS_KEY_ID:-}"
|
|
||||||
SECRET_KEY="${AWS_SECRET_ACCESS_KEY:-}"
|
|
||||||
BUCKET="compat-rw-stress"
|
|
||||||
REGION="us-east-1"
|
|
||||||
CONCURRENCY=64
|
|
||||||
OUT_DIR=""
|
|
||||||
WORK_DIR=""
|
|
||||||
MANIFEST=""
|
|
||||||
PROFILE="200g"
|
|
||||||
OBJECT_SPEC=""
|
|
||||||
DATA_PATTERN="compressible"
|
|
||||||
ENCRYPTION="none"
|
|
||||||
SSE_KMS_KEY_ID=""
|
|
||||||
SSE_C_KEY_FILE=""
|
|
||||||
CLIENT="mc"
|
|
||||||
AWS_BIN="${AWS_BIN:-aws}"
|
|
||||||
MC_BIN="${MC_BIN:-}"
|
|
||||||
KEEP_PAYLOADS=false
|
|
||||||
DRY_RUN=false
|
|
||||||
RESUME=false
|
|
||||||
|
|
||||||
usage() {
|
|
||||||
cat <<'USAGE'
|
|
||||||
Usage:
|
|
||||||
.docker/compat/run_rw_compat_stress.sh --mode <write|verify|mixed> \
|
|
||||||
--endpoint <url> --access-key <ak> --secret-key <sk> [options]
|
|
||||||
|
|
||||||
Modes:
|
|
||||||
write Create bucket, upload objects concurrently, and write manifest.csv.
|
|
||||||
verify Read objects concurrently from --endpoint and verify against manifest.csv.
|
|
||||||
mixed Write and verify against the same endpoint.
|
|
||||||
|
|
||||||
Required:
|
|
||||||
--endpoint S3 endpoint URL, for example http://127.0.0.1:9100
|
|
||||||
--access-key S3 access key
|
|
||||||
--secret-key S3 secret key
|
|
||||||
|
|
||||||
Core options:
|
|
||||||
--bucket Bucket name (default: compat-rw-stress)
|
|
||||||
--region Region (default: us-east-1)
|
|
||||||
--concurrency Parallel object operations (default: 64)
|
|
||||||
--out-dir Output directory (default: target/compat/rw-stress-<timestamp>)
|
|
||||||
--work-dir Payload scratch directory (default: <out-dir>/payloads)
|
|
||||||
--manifest Manifest path (default: <out-dir>/manifest.csv for write/mixed)
|
|
||||||
--profile compact | 5g | 200g (default: 200g)
|
|
||||||
--object-spec Override profile. Format: size:count,size:count
|
|
||||||
Example: 1KiB:1024,1MiB:1024,64MiB:512,1GiB:64
|
|
||||||
--data-pattern compressible | random | mixed (default: compressible)
|
|
||||||
--keep-payloads Do not delete local payload files after upload
|
|
||||||
--resume Skip write tasks that already have rows in <out-dir>/tasks/write-rows
|
|
||||||
--dry-run Print planned tasks and commands without executing S3 operations
|
|
||||||
--client mc | aws (default: mc)
|
|
||||||
--mc-bin Path to mc binary (default: first tmp/mc.* or mc in PATH)
|
|
||||||
--aws-bin Path to aws binary (used with --client aws)
|
|
||||||
|
|
||||||
Encryption options:
|
|
||||||
--encryption none | sse-s3 | sse-kms | sse-c (default: none)
|
|
||||||
--sse-kms-key-id KMS key id for --encryption sse-kms
|
|
||||||
--sse-c-key-file Raw 32-byte SSE-C key file for --encryption sse-c
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
# Generate the old RustFS beta5 dataset.
|
|
||||||
.docker/compat/run_rw_compat_stress.sh \
|
|
||||||
--mode write --endpoint http://127.0.0.1:9100 \
|
|
||||||
--access-key rustfsadmin --secret-key rustfsadmin \
|
|
||||||
--bucket compat-beta5 --concurrency 96
|
|
||||||
|
|
||||||
# Verify that dataset after mounting beta5 disks into the rio-v2 compose.
|
|
||||||
.docker/compat/run_rw_compat_stress.sh \
|
|
||||||
--mode verify --endpoint http://127.0.0.1:9300 \
|
|
||||||
--access-key rustfsadmin --secret-key rustfsadmin \
|
|
||||||
--bucket compat-beta5 --concurrency 96 \
|
|
||||||
--manifest target/compat/rw-stress-YYYYmmdd-HHMMSS/manifest.csv
|
|
||||||
|
|
||||||
# Faster smoke run.
|
|
||||||
.docker/compat/run_rw_compat_stress.sh \
|
|
||||||
--mode mixed --endpoint http://127.0.0.1:9300 \
|
|
||||||
--access-key rustfsadmin --secret-key rustfsadmin \
|
|
||||||
--profile compact --concurrency 16
|
|
||||||
USAGE
|
|
||||||
}
|
|
||||||
|
|
||||||
die() {
|
|
||||||
echo "ERROR: $*" >&2
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|
||||||
require_cmd() {
|
|
||||||
command -v "$1" >/dev/null 2>&1 || die "command not found: $1"
|
|
||||||
}
|
|
||||||
|
|
||||||
parse_args() {
|
|
||||||
while [[ $# -gt 0 ]]; do
|
|
||||||
case "$1" in
|
|
||||||
--mode) MODE="$2"; shift 2 ;;
|
|
||||||
--endpoint) ENDPOINT="$2"; shift 2 ;;
|
|
||||||
--access-key) ACCESS_KEY="$2"; shift 2 ;;
|
|
||||||
--secret-key) SECRET_KEY="$2"; shift 2 ;;
|
|
||||||
--bucket) BUCKET="$2"; shift 2 ;;
|
|
||||||
--region) REGION="$2"; shift 2 ;;
|
|
||||||
--concurrency) CONCURRENCY="$2"; shift 2 ;;
|
|
||||||
--out-dir) OUT_DIR="$2"; shift 2 ;;
|
|
||||||
--work-dir) WORK_DIR="$2"; shift 2 ;;
|
|
||||||
--manifest) MANIFEST="$2"; shift 2 ;;
|
|
||||||
--profile) PROFILE="$2"; shift 2 ;;
|
|
||||||
--object-spec) OBJECT_SPEC="$2"; shift 2 ;;
|
|
||||||
--data-pattern) DATA_PATTERN="$2"; shift 2 ;;
|
|
||||||
--encryption) ENCRYPTION="$2"; shift 2 ;;
|
|
||||||
--sse-kms-key-id) SSE_KMS_KEY_ID="$2"; shift 2 ;;
|
|
||||||
--sse-c-key-file) SSE_C_KEY_FILE="$2"; shift 2 ;;
|
|
||||||
--client) CLIENT="$2"; shift 2 ;;
|
|
||||||
--mc-bin) MC_BIN="$2"; shift 2 ;;
|
|
||||||
--aws-bin) AWS_BIN="$2"; shift 2 ;;
|
|
||||||
--keep-payloads) KEEP_PAYLOADS=true; shift ;;
|
|
||||||
--resume) RESUME=true; shift ;;
|
|
||||||
--dry-run) DRY_RUN=true; shift ;;
|
|
||||||
-h|--help) usage; exit 0 ;;
|
|
||||||
*) die "unknown arg: $1" ;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
validate_args() {
|
|
||||||
[[ "$MODE" =~ ^(write|verify|mixed)$ ]] || die "--mode must be write, verify, or mixed"
|
|
||||||
[[ "$CLIENT" =~ ^(mc|aws)$ ]] || die "--client must be mc or aws"
|
|
||||||
[[ "$PROFILE" =~ ^(compact|5g|200g)$ ]] || die "--profile must be compact, 5g, or 200g"
|
|
||||||
[[ "$DATA_PATTERN" =~ ^(compressible|random|mixed)$ ]] || die "--data-pattern must be compressible, random, or mixed"
|
|
||||||
[[ "$ENCRYPTION" =~ ^(none|sse-s3|sse-kms|sse-c)$ ]] || die "--encryption must be none, sse-s3, sse-kms, or sse-c"
|
|
||||||
[[ -n "$ENDPOINT" && -n "$ACCESS_KEY" && -n "$SECRET_KEY" ]] || die "--endpoint/--access-key/--secret-key are required"
|
|
||||||
[[ "$CONCURRENCY" =~ ^[0-9]+$ && "$CONCURRENCY" -gt 0 ]] || die "--concurrency must be a positive integer"
|
|
||||||
if [[ "$ENCRYPTION" == "sse-kms" && -z "$SSE_KMS_KEY_ID" ]]; then
|
|
||||||
die "--sse-kms-key-id is required for --encryption sse-kms"
|
|
||||||
fi
|
|
||||||
if [[ "$ENCRYPTION" == "sse-c" ]]; then
|
|
||||||
[[ -n "$SSE_C_KEY_FILE" && -f "$SSE_C_KEY_FILE" ]] || die "--sse-c-key-file must point to an existing key file"
|
|
||||||
fi
|
|
||||||
if [[ "$MODE" == "verify" && -z "$MANIFEST" ]]; then
|
|
||||||
die "--manifest is required for --mode verify"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
setup_paths() {
|
|
||||||
if [[ -z "$OUT_DIR" ]]; then
|
|
||||||
OUT_DIR="target/compat/rw-stress-$(date +%Y%m%d-%H%M%S)"
|
|
||||||
fi
|
|
||||||
if [[ -z "$WORK_DIR" ]]; then
|
|
||||||
WORK_DIR="$OUT_DIR/payloads"
|
|
||||||
fi
|
|
||||||
if [[ -z "$MANIFEST" ]]; then
|
|
||||||
MANIFEST="$OUT_DIR/manifest.csv"
|
|
||||||
fi
|
|
||||||
|
|
||||||
mkdir -p "$OUT_DIR" "$WORK_DIR" "$OUT_DIR/tasks" "$OUT_DIR/logs"
|
|
||||||
TASKS_FILE="$OUT_DIR/tasks/tasks.tsv"
|
|
||||||
WRITE_ROWS_DIR="$OUT_DIR/tasks/write-rows"
|
|
||||||
VERIFY_ROWS_DIR="$OUT_DIR/tasks/verify-rows"
|
|
||||||
MC_CONFIG_DIR_LOCAL="$OUT_DIR/mc-config"
|
|
||||||
MC_ALIAS="compat"
|
|
||||||
mkdir -p "$WRITE_ROWS_DIR" "$VERIFY_ROWS_DIR"
|
|
||||||
}
|
|
||||||
|
|
||||||
resolve_mc_bin() {
|
|
||||||
if [[ -n "$MC_BIN" ]]; then
|
|
||||||
echo "$MC_BIN"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
local candidate
|
|
||||||
candidate="$(find tmp -maxdepth 1 -type f -name 'mc.*' -perm -111 2>/dev/null | sort | tail -n 1 || true)"
|
|
||||||
if [[ -n "$candidate" ]]; then
|
|
||||||
echo "$candidate"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
command -v mc 2>/dev/null || true
|
|
||||||
}
|
|
||||||
|
|
||||||
size_to_bytes() {
|
|
||||||
local raw="$1"
|
|
||||||
local num unit
|
|
||||||
if [[ "$raw" =~ ^([0-9]+)(B|KiB|MiB|GiB|KB|MB|GB)?$ ]]; then
|
|
||||||
num="${BASH_REMATCH[1]}"
|
|
||||||
unit="${BASH_REMATCH[2]:-B}"
|
|
||||||
else
|
|
||||||
die "invalid size: $raw"
|
|
||||||
fi
|
|
||||||
|
|
||||||
case "$unit" in
|
|
||||||
B) echo "$num" ;;
|
|
||||||
KiB) echo $((num * 1024)) ;;
|
|
||||||
MiB) echo $((num * 1024 * 1024)) ;;
|
|
||||||
GiB) echo $((num * 1024 * 1024 * 1024)) ;;
|
|
||||||
KB) echo $((num * 1000)) ;;
|
|
||||||
MB) echo $((num * 1000 * 1000)) ;;
|
|
||||||
GB) echo $((num * 1000 * 1000 * 1000)) ;;
|
|
||||||
*) die "invalid size unit: $unit" ;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
profile_spec() {
|
|
||||||
if [[ -n "$OBJECT_SPEC" ]]; then
|
|
||||||
echo "$OBJECT_SPEC"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
case "$PROFILE" in
|
|
||||||
compact)
|
|
||||||
echo "1KiB:64,1MiB:64,16MiB:16,128MiB:4"
|
|
||||||
;;
|
|
||||||
5g)
|
|
||||||
echo "1KiB:1024,1MiB:255,16MiB:64,64MiB:28,1GiB:2"
|
|
||||||
;;
|
|
||||||
200g)
|
|
||||||
echo "1KiB:1024,1MiB:1024,64MiB:512,1GiB:64,8GiB:12,6GiB:1"
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
content_for_index() {
|
|
||||||
local index="$1"
|
|
||||||
case $((index % 3)) in
|
|
||||||
0) echo "txt|text/plain" ;;
|
|
||||||
1) echo "json|application/json" ;;
|
|
||||||
2) echo "bin|binary/octet-stream" ;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
generate_tasks() {
|
|
||||||
local spec item size count bytes i content ext mime key seed index=0
|
|
||||||
spec="$(profile_spec)"
|
|
||||||
: > "$TASKS_FILE"
|
|
||||||
|
|
||||||
IFS=',' read -r -a items <<< "$spec"
|
|
||||||
for item in "${items[@]}"; do
|
|
||||||
item="${item//[[:space:]]/}"
|
|
||||||
[[ -n "$item" ]] || continue
|
|
||||||
[[ "$item" =~ ^([^:]+):([0-9]+)$ ]] || die "invalid object spec item: $item"
|
|
||||||
size="${BASH_REMATCH[1]}"
|
|
||||||
count="${BASH_REMATCH[2]}"
|
|
||||||
bytes="$(size_to_bytes "$size")"
|
|
||||||
|
|
||||||
for ((i = 1; i <= count; i++)); do
|
|
||||||
content="$(content_for_index "$index")"
|
|
||||||
ext="${content%%|*}"
|
|
||||||
mime="${content#*|}"
|
|
||||||
key="rw-stress/${size}/obj-$(printf '%06d' "$i").${ext}"
|
|
||||||
seed="${BUCKET}:${key}:${bytes}"
|
|
||||||
printf '%s\t%s\t%s\t%s\t%s\n' "$key" "$size" "$bytes" "$mime" "$seed" >> "$TASKS_FILE"
|
|
||||||
index=$((index + 1))
|
|
||||||
done
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
write_row_file_for_key() {
|
|
||||||
local key="$1"
|
|
||||||
echo "$WRITE_ROWS_DIR/${key//\//_}.csv"
|
|
||||||
}
|
|
||||||
|
|
||||||
prepare_write_input() {
|
|
||||||
WRITE_INPUT="$TASKS_FILE"
|
|
||||||
if [[ "$RESUME" != "true" ]]; then
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
WRITE_INPUT="$OUT_DIR/tasks/write-input.tsv"
|
|
||||||
: > "$WRITE_INPUT"
|
|
||||||
|
|
||||||
local line key _size _bytes _mime _seed row_file
|
|
||||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
|
||||||
IFS=$'\t' read -r key _size _bytes _mime _seed <<< "$line"
|
|
||||||
row_file="$(write_row_file_for_key "$key")"
|
|
||||||
[[ -s "$row_file" ]] && continue
|
|
||||||
printf '%s\n' "$line" >> "$WRITE_INPUT"
|
|
||||||
done < "$TASKS_FILE"
|
|
||||||
}
|
|
||||||
|
|
||||||
print_plan() {
|
|
||||||
local total_objects total_bytes profile_label
|
|
||||||
if [[ -f "$TASKS_FILE" ]]; then
|
|
||||||
total_objects="$(wc -l < "$TASKS_FILE" | tr -d ' ')"
|
|
||||||
total_bytes="$(awk -F '\t' '{sum += $3} END {print sum + 0}' "$TASKS_FILE")"
|
|
||||||
profile_label="$(profile_spec)"
|
|
||||||
else
|
|
||||||
total_objects="$(awk 'END {count = NR - 1; if (count < 0) count = 0; print count}' "$MANIFEST")"
|
|
||||||
total_bytes="$(awk -F ',' 'NR > 1 {sum += $3} END {print sum + 0}' "$MANIFEST")"
|
|
||||||
profile_label="from manifest"
|
|
||||||
fi
|
|
||||||
|
|
||||||
cat <<PLAN
|
|
||||||
Mode: $MODE
|
|
||||||
Endpoint: $ENDPOINT
|
|
||||||
Bucket: $BUCKET
|
|
||||||
Profile: $profile_label
|
|
||||||
Objects: $total_objects
|
|
||||||
Bytes: $total_bytes
|
|
||||||
Concurrency: $CONCURRENCY
|
|
||||||
Encryption: $ENCRYPTION
|
|
||||||
Client: $CLIENT
|
|
||||||
Manifest: $MANIFEST
|
|
||||||
Out dir: $OUT_DIR
|
|
||||||
Work dir: $WORK_DIR
|
|
||||||
PLAN
|
|
||||||
}
|
|
||||||
|
|
||||||
aws_base() {
|
|
||||||
AWS_ACCESS_KEY_ID="$ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SECRET_KEY" AWS_DEFAULT_REGION="$REGION" \
|
|
||||||
"$AWS_BIN" --endpoint-url "$ENDPOINT" "$@"
|
|
||||||
}
|
|
||||||
|
|
||||||
mc_base() {
|
|
||||||
"$MC_BIN" --config-dir "$MC_CONFIG_DIR_LOCAL" "$@"
|
|
||||||
}
|
|
||||||
|
|
||||||
aws_cp_args() {
|
|
||||||
case "$ENCRYPTION" in
|
|
||||||
none) ;;
|
|
||||||
sse-s3) printf '%s\n' "--sse" "AES256" ;;
|
|
||||||
sse-kms) printf '%s\n' "--sse" "aws:kms" "--sse-kms-key-id" "$SSE_KMS_KEY_ID" ;;
|
|
||||||
sse-c) printf '%s\n' "--sse-c" "AES256" "--sse-c-key" "fileb://$SSE_C_KEY_FILE" ;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
mc_enc_target() {
|
|
||||||
printf '%s/%s/rw-stress/' "$MC_ALIAS" "$BUCKET"
|
|
||||||
}
|
|
||||||
|
|
||||||
mc_cp_args() {
|
|
||||||
local target
|
|
||||||
target="$(mc_enc_target)"
|
|
||||||
case "$ENCRYPTION" in
|
|
||||||
none) ;;
|
|
||||||
sse-s3) printf '%s\n' "--enc-s3" "$target" ;;
|
|
||||||
sse-kms) printf '%s\n' "--enc-kms" "${target}=${SSE_KMS_KEY_ID}" ;;
|
|
||||||
sse-c)
|
|
||||||
local key_b64
|
|
||||||
key_b64="$(base64 < "$SSE_C_KEY_FILE" | tr -d '\n')"
|
|
||||||
printf '%s\n' "--enc-c" "${target}=${key_b64}"
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
aws_get_args() {
|
|
||||||
if [[ "$ENCRYPTION" == "sse-c" ]]; then
|
|
||||||
printf '%s\n' "--sse-c" "AES256" "--sse-c-key" "fileb://$SSE_C_KEY_FILE"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
mc_get_args() {
|
|
||||||
if [[ "$ENCRYPTION" == "sse-c" ]]; then
|
|
||||||
local target key_b64
|
|
||||||
target="$(mc_enc_target)"
|
|
||||||
key_b64="$(base64 < "$SSE_C_KEY_FILE" | tr -d '\n')"
|
|
||||||
printf '%s\n' "--enc-c" "${target}=${key_b64}"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
setup_mc_alias() {
|
|
||||||
if [[ "$CLIENT" != "mc" || "$DRY_RUN" == "true" ]]; then
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
mkdir -p "$MC_CONFIG_DIR_LOCAL"
|
|
||||||
mc_base alias set "$MC_ALIAS" "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY" --api S3v4 --path auto >/dev/null
|
|
||||||
}
|
|
||||||
|
|
||||||
create_bucket_if_needed() {
|
|
||||||
if [[ "$DRY_RUN" == "true" ]]; then
|
|
||||||
echo "[DRY-RUN] create bucket if missing: $BUCKET"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$CLIENT" == "mc" ]]; then
|
|
||||||
mc_base mb --ignore-existing --region "$REGION" "$MC_ALIAS/$BUCKET" >/dev/null
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
if aws_base s3api head-bucket --bucket "$BUCKET" >/dev/null 2>&1; then
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
aws_base s3api create-bucket --bucket "$BUCKET" >/dev/null
|
|
||||||
}
|
|
||||||
|
|
||||||
payload_path_for_key() {
|
|
||||||
local key="$1"
|
|
||||||
echo "$WORK_DIR/${key//\//_}"
|
|
||||||
}
|
|
||||||
|
|
||||||
generate_payload() {
|
|
||||||
local file="$1"
|
|
||||||
local bytes="$2"
|
|
||||||
local seed="$3"
|
|
||||||
local pattern="$DATA_PATTERN"
|
|
||||||
mkdir -p "$(dirname "$file")"
|
|
||||||
|
|
||||||
if [[ "$pattern" == "mixed" ]]; then
|
|
||||||
if [[ $((bytes % 2)) -eq 0 ]]; then
|
|
||||||
pattern="compressible"
|
|
||||||
else
|
|
||||||
pattern="random"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$pattern" == "random" ]]; then
|
|
||||||
head -c "$bytes" /dev/zero | openssl enc -aes-256-ctr -nosalt -pass "pass:$seed" -out "$file"
|
|
||||||
else
|
|
||||||
yes "$seed payload-for-rio-compatibility" | tr '\n' ' ' | head -c "$bytes" > "$file"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
sha256_file() {
|
|
||||||
shasum -a 256 "$1" | awk '{print $1}'
|
|
||||||
}
|
|
||||||
|
|
||||||
sha256_object() {
|
|
||||||
local key="$1"
|
|
||||||
shift
|
|
||||||
if [[ "$CLIENT" == "mc" ]]; then
|
|
||||||
mc_base cat "$@" "$MC_ALIAS/$BUCKET/$key" | shasum -a 256 | awk '{print $1}'
|
|
||||||
else
|
|
||||||
aws_base s3 cp "s3://$BUCKET/$key" - --no-progress "$@" | shasum -a 256 | awk '{print $1}'
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
collect_args() {
|
|
||||||
local generator="$1"
|
|
||||||
extra_args=()
|
|
||||||
while IFS= read -r arg; do
|
|
||||||
extra_args+=("$arg")
|
|
||||||
done < <("$generator")
|
|
||||||
}
|
|
||||||
|
|
||||||
write_one() {
|
|
||||||
local line="$1"
|
|
||||||
local key size bytes mime seed file sha row_file
|
|
||||||
local -a extra_args
|
|
||||||
IFS=$'\t' read -r key size bytes mime seed <<< "$line"
|
|
||||||
file="$(payload_path_for_key "$key")"
|
|
||||||
row_file="$(write_row_file_for_key "$key")"
|
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == "true" ]]; then
|
|
||||||
echo "[DRY-RUN] upload $bytes bytes to s3://$BUCKET/$key content-type=$mime"
|
|
||||||
printf '%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "DRY_RUN" "$ENCRYPTION" > "$row_file"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
generate_payload "$file" "$bytes" "$seed"
|
|
||||||
sha="$(sha256_file "$file")"
|
|
||||||
|
|
||||||
if [[ "$CLIENT" == "mc" ]]; then
|
|
||||||
collect_args mc_cp_args
|
|
||||||
mc_base cp --quiet --attr "Content-Type=$mime" ${extra_args[@]+"${extra_args[@]}"} "$file" "$MC_ALIAS/$BUCKET/$key" \
|
|
||||||
> "$OUT_DIR/logs/${key//\//_}.put.log" 2>&1
|
|
||||||
else
|
|
||||||
collect_args aws_cp_args
|
|
||||||
aws_base s3 cp "$file" "s3://$BUCKET/$key" --no-progress --content-type "$mime" ${extra_args[@]+"${extra_args[@]}"} \
|
|
||||||
> "$OUT_DIR/logs/${key//\//_}.put.log" 2>&1
|
|
||||||
fi
|
|
||||||
|
|
||||||
printf '%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$sha" "$ENCRYPTION" > "$row_file"
|
|
||||||
if [[ "$KEEP_PAYLOADS" != "true" ]]; then
|
|
||||||
rm -f "$file"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
verify_one() {
|
|
||||||
local line="$1"
|
|
||||||
local key size bytes mime expected encryption actual row_file
|
|
||||||
local -a extra_args
|
|
||||||
IFS=',' read -r key size bytes mime expected encryption <<< "$line"
|
|
||||||
row_file="$VERIFY_ROWS_DIR/${key//\//_}.csv"
|
|
||||||
|
|
||||||
if [[ "$DRY_RUN" == "true" ]]; then
|
|
||||||
echo "[DRY-RUN] verify s3://$BUCKET/$key expected=$expected"
|
|
||||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "DRY_RUN" "dry-run" > "$row_file"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$CLIENT" == "mc" ]]; then
|
|
||||||
collect_args mc_get_args
|
|
||||||
else
|
|
||||||
collect_args aws_get_args
|
|
||||||
fi
|
|
||||||
if ! actual="$(sha256_object "$key" ${extra_args[@]+"${extra_args[@]}"})"; then
|
|
||||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "ERROR" "download failed" > "$row_file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$actual" == "$expected" ]]; then
|
|
||||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "$actual" "ok" > "$row_file"
|
|
||||||
else
|
|
||||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "$actual" "sha256-mismatch" > "$row_file"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
run_parallel_tasks() {
|
|
||||||
local action="$1"
|
|
||||||
local input_file="$2"
|
|
||||||
local failure_file="$OUT_DIR/tasks/${action}.failed"
|
|
||||||
local line
|
|
||||||
rm -f "$failure_file"
|
|
||||||
|
|
||||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
|
||||||
while [[ "$(jobs -rp | wc -l | tr -d ' ')" -ge "$CONCURRENCY" ]]; do
|
|
||||||
sleep 0.2
|
|
||||||
done
|
|
||||||
|
|
||||||
({
|
|
||||||
if [[ "$action" == "write" ]]; then
|
|
||||||
write_one "$line"
|
|
||||||
else
|
|
||||||
verify_one "$line"
|
|
||||||
fi
|
|
||||||
} || touch "$failure_file") &
|
|
||||||
done < "$input_file"
|
|
||||||
|
|
||||||
wait
|
|
||||||
[[ ! -f "$failure_file" ]]
|
|
||||||
}
|
|
||||||
|
|
||||||
combine_write_manifest() {
|
|
||||||
echo "key,size,bytes,content_type,sha256,encryption" > "$MANIFEST"
|
|
||||||
find "$WRITE_ROWS_DIR" -type f -name '*.csv' -print0 | sort -z | xargs -0 cat >> "$MANIFEST"
|
|
||||||
|
|
||||||
local expected actual
|
|
||||||
expected="$(wc -l < "$TASKS_FILE" | tr -d ' ')"
|
|
||||||
actual="$(( $(wc -l < "$MANIFEST" | tr -d ' ') - 1 ))"
|
|
||||||
if [[ "$actual" -ne "$expected" ]]; then
|
|
||||||
die "manifest row count mismatch: expected $expected, got $actual"
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
prepare_verify_input() {
|
|
||||||
VERIFY_INPUT="$OUT_DIR/tasks/verify-input.csv"
|
|
||||||
tail -n +2 "$MANIFEST" > "$VERIFY_INPUT"
|
|
||||||
}
|
|
||||||
|
|
||||||
combine_verify_summary() {
|
|
||||||
VERIFY_SUMMARY="$OUT_DIR/verify-summary.csv"
|
|
||||||
echo "key,size,bytes,content_type,expected_sha256,actual_sha256,status" > "$VERIFY_SUMMARY"
|
|
||||||
find "$VERIFY_ROWS_DIR" -type f -name '*.csv' -print0 | sort -z | xargs -0 cat >> "$VERIFY_SUMMARY"
|
|
||||||
|
|
||||||
local failed
|
|
||||||
if [[ "$DRY_RUN" == "true" ]]; then
|
|
||||||
echo "Verification dry run complete. Summary: $VERIFY_SUMMARY"
|
|
||||||
return
|
|
||||||
fi
|
|
||||||
|
|
||||||
failed="$(awk -F ',' 'NR > 1 && $7 != "ok" {count++} END {print count + 0}' "$VERIFY_SUMMARY")"
|
|
||||||
local expected actual
|
|
||||||
expected="$(wc -l < "$VERIFY_INPUT" | tr -d ' ')"
|
|
||||||
actual="$(( $(wc -l < "$VERIFY_SUMMARY" | tr -d ' ') - 1 ))"
|
|
||||||
if [[ "$actual" -ne "$expected" ]]; then
|
|
||||||
echo "Verification row count mismatch: expected $expected, got $actual" >&2
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
if [[ "$failed" -ne 0 ]]; then
|
|
||||||
echo "Verification failed: $failed object(s). See $VERIFY_SUMMARY" >&2
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
echo "Verification passed. Summary: $VERIFY_SUMMARY"
|
|
||||||
}
|
|
||||||
|
|
||||||
export_functions() {
|
|
||||||
export ENDPOINT ACCESS_KEY SECRET_KEY BUCKET REGION ENCRYPTION SSE_KMS_KEY_ID SSE_C_KEY_FILE AWS_BIN
|
|
||||||
export CLIENT MC_BIN MC_CONFIG_DIR_LOCAL MC_ALIAS
|
|
||||||
export WORK_DIR OUT_DIR WRITE_ROWS_DIR VERIFY_ROWS_DIR DATA_PATTERN KEEP_PAYLOADS DRY_RUN
|
|
||||||
export -f aws_base mc_base aws_cp_args aws_get_args mc_enc_target mc_cp_args mc_get_args payload_path_for_key generate_payload sha256_file sha256_object write_one verify_one
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
|
||||||
parse_args "$@"
|
|
||||||
validate_args
|
|
||||||
setup_paths
|
|
||||||
|
|
||||||
if [[ "$DRY_RUN" != "true" ]]; then
|
|
||||||
if [[ "$CLIENT" == "mc" ]]; then
|
|
||||||
MC_BIN="$(resolve_mc_bin)"
|
|
||||||
[[ -n "$MC_BIN" ]] || die "mc binary not found; pass --mc-bin or put mc in PATH"
|
|
||||||
require_cmd "$MC_BIN"
|
|
||||||
else
|
|
||||||
require_cmd "$AWS_BIN"
|
|
||||||
fi
|
|
||||||
require_cmd shasum
|
|
||||||
require_cmd head
|
|
||||||
require_cmd yes
|
|
||||||
require_cmd openssl
|
|
||||||
elif [[ "$CLIENT" == "mc" ]]; then
|
|
||||||
MC_BIN="$(resolve_mc_bin)"
|
|
||||||
fi
|
|
||||||
|
|
||||||
setup_mc_alias
|
|
||||||
|
|
||||||
if [[ "$MODE" == "write" || "$MODE" == "mixed" ]]; then
|
|
||||||
local write_failed=false
|
|
||||||
generate_tasks
|
|
||||||
prepare_write_input
|
|
||||||
print_plan
|
|
||||||
create_bucket_if_needed
|
|
||||||
export_functions
|
|
||||||
if ! run_parallel_tasks write "$WRITE_INPUT"; then
|
|
||||||
write_failed=true
|
|
||||||
fi
|
|
||||||
combine_write_manifest
|
|
||||||
echo "Write manifest: $MANIFEST"
|
|
||||||
if [[ "$write_failed" == "true" ]]; then
|
|
||||||
die "one or more write tasks failed; see $OUT_DIR/logs"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$MODE" == "verify" || "$MODE" == "mixed" ]]; then
|
|
||||||
local verify_failed=false
|
|
||||||
[[ -f "$MANIFEST" ]] || die "manifest not found: $MANIFEST"
|
|
||||||
if [[ "$MODE" == "verify" ]]; then
|
|
||||||
cp "$MANIFEST" "$OUT_DIR/manifest.csv"
|
|
||||||
MANIFEST="$OUT_DIR/manifest.csv"
|
|
||||||
fi
|
|
||||||
prepare_verify_input
|
|
||||||
print_plan
|
|
||||||
export_functions
|
|
||||||
if ! run_parallel_tasks verify "$VERIFY_INPUT"; then
|
|
||||||
verify_failed=true
|
|
||||||
fi
|
|
||||||
combine_verify_summary
|
|
||||||
if [[ "$verify_failed" == "true" ]]; then
|
|
||||||
die "one or more verify tasks failed; see $VERIFY_SUMMARY"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
@@ -1,52 +0,0 @@
|
|||||||
services:
|
|
||||||
rustfs:
|
|
||||||
image: rustfs/rustfs:1.0.0-alpha.99-glibc
|
|
||||||
container_name: rustfs-issue-2715-test
|
|
||||||
security_opt:
|
|
||||||
- "no-new-privileges:true"
|
|
||||||
ports:
|
|
||||||
- "19000:9000"
|
|
||||||
- "19001:9001"
|
|
||||||
environment:
|
|
||||||
- RUSTFS_VOLUMES=/data/rustfs{0...8}
|
|
||||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
|
||||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
|
||||||
- RUSTFS_CONSOLE_ENABLE=true
|
|
||||||
- RUSTFS_CORS_ALLOWED_ORIGINS=*
|
|
||||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
|
||||||
- RUSTFS_ACCESS_KEY=admin
|
|
||||||
- RUSTFS_SECRET_KEY=admin
|
|
||||||
- RUSTFS_OBS_LOGGER_LEVEL=info
|
|
||||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
|
||||||
- RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
|
|
||||||
- RUSTFS_STORAGE_CLASS_STANDARD=EC:2
|
|
||||||
- RUSTFS_STORAGE_CLASS_RRS=EC:1
|
|
||||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
|
|
||||||
- RUSTFS_OBS_LOG_DIRECTORY=/opt/rustfs/logs
|
|
||||||
extra_hosts:
|
|
||||||
- "otel-collector:host-gateway"
|
|
||||||
- "pyroscope:host-gateway"
|
|
||||||
volumes:
|
|
||||||
- ./deploy/data/issue-2715/rustfs0:/data/rustfs0
|
|
||||||
- ./deploy/data/issue-2715/rustfs1:/data/rustfs1
|
|
||||||
- ./deploy/data/issue-2715/rustfs2:/data/rustfs2
|
|
||||||
- ./deploy/data/issue-2715/rustfs3:/data/rustfs3
|
|
||||||
- ./deploy/data/issue-2715/rustfs4:/data/rustfs4
|
|
||||||
- ./deploy/data/issue-2715/rustfs5:/data/rustfs5
|
|
||||||
- ./deploy/data/issue-2715/rustfs6:/data/rustfs6
|
|
||||||
- ./deploy/data/issue-2715/rustfs7:/data/rustfs7
|
|
||||||
- ./deploy/data/issue-2715/rustfs8:/data/rustfs8
|
|
||||||
- ./deploy/logs/issue-2715:/opt/rustfs/logs
|
|
||||||
restart: unless-stopped
|
|
||||||
healthcheck:
|
|
||||||
test:
|
|
||||||
[
|
|
||||||
"CMD",
|
|
||||||
"sh",
|
|
||||||
"-c",
|
|
||||||
"curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
|
|
||||||
]
|
|
||||||
interval: 30s
|
|
||||||
timeout: 10s
|
|
||||||
retries: 3
|
|
||||||
start_period: 40s
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user