mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-30 10:08:58 +00:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 copilot-swe-agent[bot]
|
fe637d2087 |
@@ -1,51 +0,0 @@
|
||||
---
|
||||
name: arch-checks
|
||||
description: Resolve failures from the repository's architecture guard scripts — check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_unsafe_code_allowances.sh, check_logging_guardrails.sh, check_doc_paths.sh. Use when make pre-commit / pre-pr or CI fails on one of these checks.
|
||||
---
|
||||
|
||||
# Architecture Guard Checks
|
||||
|
||||
All five run in `make pre-commit` / `make pre-pr` and in CI. Fix the cause;
|
||||
never weaken a check to get green.
|
||||
|
||||
## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
|
||||
|
||||
Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no
|
||||
upward imports. Known legacy violations live in
|
||||
`scripts/layer-dependency-baseline.txt`.
|
||||
|
||||
- **New violation**: restructure your change so the dependency points
|
||||
downward (move the shared type/function to the lower layer).
|
||||
- **You legitimately removed a baseline entry**: run
|
||||
`./scripts/check_layer_dependencies.sh --update-baseline` and commit the
|
||||
shrunken baseline. Never add new entries to the baseline to make a new
|
||||
violation pass.
|
||||
|
||||
## `check_architecture_migration_rules.sh` — required doc sections
|
||||
|
||||
Asserts that the core docs under `docs/architecture/` (overview,
|
||||
crate-boundaries, runtime-lifecycle, readiness-matrix,
|
||||
storage-control-data-plane, global-state-crate-split-plan,
|
||||
ecstore-module-split-plan, …) still contain specific headings and exact
|
||||
source lines. If it fails after a doc edit, you reworded or removed a
|
||||
guarded line — restore the wording or update the script deliberately in the
|
||||
same PR, with rationale.
|
||||
|
||||
## `check_unsafe_code_allowances.sh`
|
||||
|
||||
Every `#[allow(unsafe_code)]` needs a `SAFETY:` comment within a few lines.
|
||||
Write the actual safety argument; don't add a placeholder.
|
||||
|
||||
## `check_logging_guardrails.sh`
|
||||
|
||||
A fixed list of security-sensitive files (auth, IAM, KMS, admin handlers…)
|
||||
is scanned for logging violations. If you created a new sensitive file,
|
||||
consider adding it to the script's `checked_files` list.
|
||||
|
||||
## `check_doc_paths.sh`
|
||||
|
||||
Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
|
||||
`docs/architecture/*.md`) must not reference repo file paths that no longer
|
||||
exist. If your refactor moved code, update the docs that point at it — the
|
||||
error message lists `doc -> stale-path` pairs. Historical plans under
|
||||
`docs/superpowers/plans/` are exempt.
|
||||
@@ -1,91 +0,0 @@
|
||||
---
|
||||
name: code-change-verification
|
||||
description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
|
||||
---
|
||||
|
||||
# Code Change Verification
|
||||
|
||||
Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. Read the scope: commit, PR, patch, or file list.
|
||||
2. Map each changed area by risk and user impact.
|
||||
3. Inspect each risky change in context.
|
||||
4. Report findings first, ordered by severity.
|
||||
5. Close with residual risks and verification recommendations.
|
||||
|
||||
## Core Workflow
|
||||
|
||||
### 1) Scope and assumptions
|
||||
- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
|
||||
- If context is missing, state assumptions before deeper analysis.
|
||||
- Focus only on requested scope; avoid reviewing unrelated files.
|
||||
|
||||
### 2) Risk map
|
||||
- Prioritize in this order:
|
||||
- Data correctness and user-visible behavior
|
||||
- API/contract compatibility
|
||||
- Security and authz/authn boundaries
|
||||
- Concurrency and lifecycle correctness
|
||||
- Performance and resource usage
|
||||
- Give higher priority to stateful paths, migration logic, defaults, and error handling.
|
||||
|
||||
### 3) Evidence-based inspection
|
||||
- Read each modified hunk with neighboring context.
|
||||
- Trace call paths and call-site expectations.
|
||||
- Check for:
|
||||
- invariant breaks and missing guards
|
||||
- unchecked assumptions and null/empty/error-path handling
|
||||
- stale tests, fixtures, and configs
|
||||
- hidden coupling to shared helpers/constants/features
|
||||
- If a point is uncertain, mark it as an open question instead of guessing.
|
||||
|
||||
#### Rust-specific checks (apply to all Rust changes)
|
||||
|
||||
- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
|
||||
- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
|
||||
- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
|
||||
- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
|
||||
- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
|
||||
- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
|
||||
- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
|
||||
- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
|
||||
- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
|
||||
- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
|
||||
|
||||
### 4) Findings-first output
|
||||
- Order findings by severity:
|
||||
- P0: critical failure, security breach, or data loss risk
|
||||
- P1: high-impact regression
|
||||
- P2: medium risk correctness gap
|
||||
- P3: low risk/quality debt
|
||||
- For each finding include:
|
||||
- Severity
|
||||
- `path:line` reference
|
||||
- concise issue statement
|
||||
- impact and likely failure mode
|
||||
- specific fix or mitigation
|
||||
- validation step to confirm
|
||||
- If no issues exist, explicitly state `No findings` and why.
|
||||
|
||||
### 5) Close
|
||||
- Report assumptions and unknowns.
|
||||
- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
|
||||
|
||||
## Output Template
|
||||
|
||||
1. Findings
|
||||
2. No findings (if applicable)
|
||||
3. Assumptions / Unknowns
|
||||
4. Recommended verification steps
|
||||
|
||||
## Finding Template
|
||||
|
||||
- `[P1] Missing timeout for downstream call`
|
||||
- Location: `path/to/file.rs:123`
|
||||
- Issue: ...
|
||||
- Impact: ...
|
||||
- Fix suggestion: ...
|
||||
- Validation: ...
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Code Change Verification"
|
||||
short_description: "Prioritize risks and verify code changes before merge."
|
||||
default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."
|
||||
@@ -1,83 +0,0 @@
|
||||
---
|
||||
name: plugin-contract-guard
|
||||
description: Invariants and change procedure for the target-plugin / extension system — plugin manifests, admin plugin/extension catalog and instance APIs, secret redaction, external-plugin install policy. Use when editing crates/targets (manifest, plugin, control_plane, catalog, runtime), crates/extension-schema, or rustfs/src/admin plugin_contract.rs / plugins_*.rs / extensions.rs / target_descriptor.rs.
|
||||
---
|
||||
|
||||
# Plugin & Extension Contract Guard
|
||||
|
||||
The "plugin system" spans four surfaces that must stay consistent:
|
||||
|
||||
| Surface | Location |
|
||||
|---|---|
|
||||
| Manifests & registry | `crates/targets/src/{manifest,plugin}.rs` |
|
||||
| Install/enable planning (control plane) | `crates/targets/src/control_plane.rs` |
|
||||
| Extension schemas | `crates/extension-schema/src/lib.rs`, `crates/targets/src/catalog/extension.rs` |
|
||||
| Admin API contract | `rustfs/src/admin/plugin_contract.rs`, `handlers/{plugins_catalog,plugins_instances,extensions,target_descriptor}.rs` |
|
||||
|
||||
## Hard invariants (verify before merging)
|
||||
|
||||
1. **Secrets have one source of truth.** Secret config keys are declared only
|
||||
in the plugin manifest (`TargetPluginManifest.secret_fields`,
|
||||
`crates/targets/src/manifest.rs`) and flow to admin via
|
||||
`AdminTargetSpec.secret_fields`. Never add a hand-maintained per-service
|
||||
secret table in a handler; if redaction misses a field, fix the manifest.
|
||||
|
||||
2. **Redaction must round-trip.** Instance GET responses replace secret values
|
||||
with `***redacted***` (`REDACTED_SECRET_VALUE` in `plugins_instances.rs`).
|
||||
Instance PUT restores the stored secret when it receives that placeholder
|
||||
back (`restore_redacted_secret_values`). Any new read or write path for
|
||||
target config must keep both halves: redact on the way out, restore the
|
||||
placeholder on the way in. The placeholder literal must never be persisted.
|
||||
|
||||
3. **Fixtures never reach production responses.**
|
||||
`example_external_webhook_plugin()` (`crates/targets/src/catalog/mod.rs`)
|
||||
is a test/demo fixture for control-plane planning tests. Production
|
||||
catalog/extension handlers must not include it; regression tests
|
||||
(`plugin_catalog_never_exposes_example_or_external_fixtures`,
|
||||
`extension_catalog_never_exposes_example_or_external_fixtures`) enforce it.
|
||||
|
||||
4. **External plugin flow is planning-only and deny-by-default.**
|
||||
`plan_external_target_plugin_action` returns decisions, it executes
|
||||
nothing. `TargetPluginExternalFlowGate::default()` is fully closed and
|
||||
`TargetPluginInstallPolicy::default().allowed_download_hosts` is empty —
|
||||
keep it that way; tests opt in via explicit policies. Install validation
|
||||
requires https, an allowlisted host, a full 64-hex-char sha256 digest,
|
||||
signature and provenance URIs, and an artifact matching the host
|
||||
`target_triple`.
|
||||
|
||||
5. **Custom target types must not collide.** Unknown target types get an
|
||||
interned unique `custom:<type>` plugin id (`custom_plugin_id` in
|
||||
`manifest.rs`). Custom plugins with secrets must register via
|
||||
`TargetPluginDescriptor::with_manifest` and declare `secret_fields`;
|
||||
`::new` derives a manifest with no secrets.
|
||||
|
||||
## Changing the admin JSON contract
|
||||
|
||||
- Shapes are locked twice in `plugin_contract.rs` tests: insta snapshots
|
||||
(`rustfs/src/admin/snapshots/`) plus literal `json!` assertions. Update
|
||||
both deliberately; a shape change is a console-facing API change.
|
||||
- Field naming is `snake_case`, except discovery blocks
|
||||
(`runtimeCapabilities`, `clusterSnapshot`, `extensionsCatalog`) which are
|
||||
camelCase **by cross-endpoint convention** (same shape in `system.rs`,
|
||||
`console.rs`, `pools.rs`). Do not "fix" that inconsistency locally.
|
||||
- Contract types deliberately duplicate `rustfs_targets` types
|
||||
(anti-corruption layer). Add a `From` impl; do not serialize internal
|
||||
types directly.
|
||||
|
||||
## Handler conventions
|
||||
|
||||
- Every new admin plugin/extension route needs authorization at the top of
|
||||
`call` and an `include_str!` guard test asserting it (repo-wide pattern —
|
||||
see `plugin_instance_handlers_require_admin_authorization_contract`).
|
||||
- Reads use `GetBucketTargetAction` (instances) or `ServerInfoAdminAction`
|
||||
(catalogs); writes use `SetBucketTargetAction`.
|
||||
- Refresh persisted module switches once per request
|
||||
(`refresh_persisted_module_switches`), then evaluate the sync
|
||||
`module_disabled_block_reason` per domain — do not re-read the store per
|
||||
domain or per instance.
|
||||
|
||||
## Generic bounds
|
||||
|
||||
Event-payload generics use the `PluginEvent` blanket trait
|
||||
(`crates/targets/src/plugin.rs`). Do not respell
|
||||
`Send + Sync + 'static + Clone + Serialize + DeserializeOwned`.
|
||||
@@ -1,94 +0,0 @@
|
||||
---
|
||||
name: pr-creation-checker
|
||||
description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
|
||||
---
|
||||
|
||||
# PR Creation Checker
|
||||
|
||||
Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
|
||||
|
||||
## Read sources of truth first
|
||||
|
||||
- Read `AGENTS.md`.
|
||||
- Read `.github/pull_request_template.md`.
|
||||
- Use `Makefile` and `.config/make/` for local quality commands.
|
||||
- Use `.github/workflows/ci.yml` for CI expectations.
|
||||
- Do not restate long command matrices or template sections from memory when the files exist.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Collect PR context
|
||||
- Confirm base branch, current branch, change goal, and scope.
|
||||
- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
|
||||
- Confirm whether the branch includes only intended changes.
|
||||
|
||||
2. Inspect change scope
|
||||
- Review the diff and summarize what changed.
|
||||
- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
|
||||
- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
|
||||
- Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
|
||||
- Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
|
||||
|
||||
3. Verify readiness requirements
|
||||
- Require `make pre-commit` before marking PRs ready when the diff changes Rust code, product behavior, CI behavior, runtime configuration, security-sensitive logic, migrations, storage, auth, networking, or other high-risk paths.
|
||||
- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, allow focused verification instead of `make pre-commit` when it directly validates the changed surface.
|
||||
- For focused verification, explain why the full gate was not run and list the scope-specific commands in the PR body.
|
||||
- If `make` is unavailable, use the equivalent commands from `.config/make/`.
|
||||
- Add scope-specific verification commands when the changed area needs more than the baseline.
|
||||
- If required checks fail, stop and return `BLOCKED`.
|
||||
|
||||
4. Draft PR metadata
|
||||
- Write the PR title in English using Conventional Commits and keep it within 72 characters.
|
||||
- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
|
||||
- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
|
||||
- Keep the PR body in English.
|
||||
- Use the exact section headings from `.github/pull_request_template.md`.
|
||||
- Fill non-applicable sections with `N/A`.
|
||||
- Include verification commands in the PR description.
|
||||
- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
|
||||
- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
|
||||
|
||||
5. Prepare reviewer context
|
||||
- Summarize why the change exists.
|
||||
- Summarize what was verified.
|
||||
- Call out risks, rollout notes, config impact, and rollback notes when applicable.
|
||||
- Mention assumptions or missing context instead of guessing.
|
||||
|
||||
6. Prepare CLI-safe output
|
||||
- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
|
||||
- Return a ready-to-save PR body plus a short title.
|
||||
- If not ready, return blockers first and list the minimum steps needed to unblock.
|
||||
|
||||
## Output format
|
||||
|
||||
### Status
|
||||
- `READY` or `BLOCKED`
|
||||
|
||||
### Title
|
||||
- `<type>(<scope>): <summary>`
|
||||
|
||||
### PR Body
|
||||
- Reproduce the repository template headings exactly.
|
||||
- Fill every section.
|
||||
- Omit local absolute paths unless explicitly required.
|
||||
|
||||
### Verification
|
||||
- List each command run.
|
||||
- State pass/fail.
|
||||
|
||||
### Risks
|
||||
- List breaking changes, config changes, migration impact, or `N/A`.
|
||||
|
||||
## Blocker rules
|
||||
|
||||
- Return `BLOCKED` if a code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk change has not passed `make pre-commit`.
|
||||
- Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface.
|
||||
- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
|
||||
- Return `BLOCKED` if required template sections are missing.
|
||||
- Return `BLOCKED` if the title/body is not in English.
|
||||
- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
|
||||
- Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
|
||||
|
||||
## Reference
|
||||
|
||||
- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "PR Creation Checker"
|
||||
short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
|
||||
default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."
|
||||
@@ -1,16 +0,0 @@
|
||||
# PR Readiness Checklist
|
||||
|
||||
- Confirm the branch is based on current `main`.
|
||||
- Confirm the diff matches the stated scope.
|
||||
- Confirm no secrets, logs, temp files, or unrelated refactors are included.
|
||||
- Confirm `make pre-commit` passed for code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk changes.
|
||||
- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, confirm focused verification covered the changed surface and the PR body explains why the full gate was not run.
|
||||
- Confirm extra verification commands are listed for risky changes.
|
||||
- Confirm the PR title uses Conventional Commits and stays within 72 characters.
|
||||
- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
|
||||
- Confirm the PR body is in English.
|
||||
- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
|
||||
- Confirm non-applicable sections are filled with `N/A`.
|
||||
- Confirm the PR body does not include local absolute paths unless explicitly required.
|
||||
- Confirm multiline GitHub CLI commands use `--body-file`.
|
||||
- Confirm new hardcoded string literals were not introduced for values already represented by existing constants/enums (including protocol labels, error identifiers, headers, and metric names), or record a justified exception.
|
||||
@@ -1,112 +0,0 @@
|
||||
---
|
||||
name: rust-code-quality
|
||||
description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
|
||||
---
|
||||
|
||||
# Rust Code Quality Gate
|
||||
|
||||
Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch.
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. Identify changed `.rs` files.
|
||||
2. Run automated checks on changed files.
|
||||
3. Run manual review checklist on the diff.
|
||||
4. Report findings; block merge if P0/P1 issues exist.
|
||||
|
||||
## Automated Checks
|
||||
|
||||
Run these on every changed `.rs` file (excluding test modules):
|
||||
|
||||
```bash
|
||||
# 1. unwrap/expect in production code
|
||||
rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
|
||||
|
||||
# 2. Silent type truncation via `as` cast
|
||||
rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
|
||||
|
||||
# 3. String as error type
|
||||
rg -n 'Result<.*String>' <changed-files> | grep -v test
|
||||
|
||||
# 4. Box<dyn Error> in public APIs
|
||||
rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
|
||||
|
||||
# 5. println/eprintln in production
|
||||
rg -n 'println!\|eprintln!' <changed-files> | grep -v test
|
||||
|
||||
# 6. Ordering::Relaxed usage (verify each is intentional)
|
||||
rg -n 'Ordering::Relaxed' <changed-files>
|
||||
```
|
||||
|
||||
## Manual Review Checklist
|
||||
|
||||
For every Rust code change, verify:
|
||||
|
||||
### Error Handling
|
||||
- [ ] No `unwrap()` or `expect()` in production code without justification comment
|
||||
- [ ] No `Result<_, String>` in public API signatures
|
||||
- [ ] No `Box<dyn Error>` in public trait/struct methods
|
||||
- [ ] `Error::source()` is overridden when inner error is stored
|
||||
- [ ] Error messages are actionable (what failed, with what input)
|
||||
|
||||
### Type Safety
|
||||
- [ ] No silent `as` truncation (negative→unsigned, large→small)
|
||||
- [ ] `try_into()` or explicit clamping used for numeric conversions
|
||||
- [ ] No `f64 as usize` without prior clamping
|
||||
|
||||
### Concurrency
|
||||
- [ ] Lock acquisition order is documented when multiple locks are used
|
||||
- [ ] No `tokio::sync` write guards held across `.await` without bounded hold time
|
||||
- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
|
||||
- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
|
||||
|
||||
### Memory and Performance
|
||||
- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
|
||||
- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
|
||||
- [ ] Large buffers wrapped in `Arc` rather than cloned
|
||||
- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
|
||||
|
||||
### Recursion Safety
|
||||
- [ ] Recursive functions have a depth limit or use iterative traversal
|
||||
- [ ] Tree/cache traversals handle corrupted/cyclic input safely
|
||||
|
||||
### Testing
|
||||
- [ ] Every test function has at least one `assert!`
|
||||
- [ ] Tests use `.expect("context")` not bare `.unwrap()`
|
||||
- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
|
||||
|
||||
### Serde
|
||||
- [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
|
||||
- [ ] `#[serde(default)]` not used on security-critical fields without validation
|
||||
|
||||
### Code Hygiene
|
||||
- [ ] No `#![allow(dead_code)]` at crate root
|
||||
- [ ] No camelCase statics or Hungarian notation
|
||||
- [ ] New string literals don't duplicate existing constants
|
||||
|
||||
## Severity Classification
|
||||
|
||||
- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
|
||||
- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
|
||||
- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
|
||||
- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
|
||||
|
||||
## Output Template
|
||||
|
||||
```
|
||||
## Rust Code Quality Report
|
||||
|
||||
### Automated Scan
|
||||
- unwrap/expect in production: N found
|
||||
- as casts: N found
|
||||
- String errors: N found
|
||||
- println/eprintln: N found
|
||||
|
||||
### Findings
|
||||
- [P1] `path:line` — description
|
||||
- Fix: ...
|
||||
- Validation: ...
|
||||
|
||||
### Verdict
|
||||
PASS / BLOCKED (list blocking findings)
|
||||
```
|
||||
@@ -1,52 +0,0 @@
|
||||
# Rust Code Quality Checklist
|
||||
|
||||
Use this as a quick pre-merge checklist for every Rust code change.
|
||||
|
||||
## Critical (P0 — block merge)
|
||||
|
||||
| Check | Command |
|
||||
|-------|---------|
|
||||
| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
|
||||
| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
|
||||
| Lock order consistent across call sites | Manual: trace all lock acquisitions |
|
||||
| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
|
||||
| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
|
||||
|
||||
## High (P1 — must fix)
|
||||
|
||||
| Check | Command |
|
||||
|-------|---------|
|
||||
| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
|
||||
| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
|
||||
| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
|
||||
| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
|
||||
| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
|
||||
|
||||
## Medium (P2 — should fix)
|
||||
|
||||
| Check | Command |
|
||||
|-------|---------|
|
||||
| Tests have assertions | Manual: check for `assert` in test functions |
|
||||
| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
|
||||
| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
|
||||
| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
|
||||
|
||||
## Low (P3 — nice to fix)
|
||||
|
||||
| Check | Command |
|
||||
|-------|---------|
|
||||
| No camelCase statics | `rg 'static ref [a-z]' <files>` |
|
||||
| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
|
||||
| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
|
||||
|
||||
## Quick One-Liner
|
||||
|
||||
```bash
|
||||
# Run all automated checks on changed files
|
||||
CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
|
||||
echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
|
||||
echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
|
||||
echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
|
||||
echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
|
||||
echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
|
||||
```
|
||||
@@ -1,107 +0,0 @@
|
||||
---
|
||||
name: rustfs-logging-governance
|
||||
description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use when editing or reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
|
||||
---
|
||||
|
||||
# RustFS Logging Governance
|
||||
|
||||
Use this skill when RustFS logging needs to be added, cleaned up, reviewed, or protected against regressions.
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. Identify the files whose logs are changing.
|
||||
2. Scan current `tracing` or `log` macros before editing.
|
||||
3. Convert sentence-style logs to short event-style logs.
|
||||
4. Demote hot-path success logs unless operators truly need them at `info`.
|
||||
5. Preserve failure, fallback, and security-relevant diagnostics.
|
||||
6. Update `scripts/check_logging_guardrails.sh` when a broad cleanup removes a legacy pattern class.
|
||||
7. Validate with formatting, targeted checks/tests, and the logging guardrail script.
|
||||
|
||||
## Core Workflow
|
||||
|
||||
### 1. Scope the logging surface
|
||||
|
||||
- Read the changed module in full before touching log lines.
|
||||
- Classify the log site:
|
||||
- lifecycle/startup
|
||||
- request or validation path
|
||||
- background loop or hot path
|
||||
- fallback/degraded behavior
|
||||
- cloud metadata or external fetch path
|
||||
- metrics/config summary
|
||||
- Do not rewrite business logic to make logging easier.
|
||||
|
||||
### 2. Use the RustFS event shape
|
||||
|
||||
- Prefer fields first, message second.
|
||||
- Use short labels, not prose paragraphs.
|
||||
- Default field shape:
|
||||
- `event`
|
||||
- `component`
|
||||
- `subsystem`
|
||||
- `state` or `result`
|
||||
- key context fields
|
||||
- Reuse stable field names and avoid inventing near-duplicates.
|
||||
|
||||
See `references/logging-governance.md` for the event model, level policy, and anti-pattern list.
|
||||
|
||||
### 3. Choose the right level
|
||||
|
||||
- `error`: operation failure that affects behavior or security guarantees.
|
||||
- `warn`: degraded path, fallback, suspicious input, or operator-actionable misconfiguration.
|
||||
- `info`: low-frequency lifecycle or mode selection.
|
||||
- `debug`: targeted diagnostics and low-volume detail.
|
||||
- `trace`: hot-path and repetitive success-path events.
|
||||
|
||||
When in doubt, lower the verbosity of normal success paths and keep structured detail in fields.
|
||||
|
||||
### 4. Preserve security and privacy boundaries
|
||||
|
||||
- Do not log secrets, tokens, auth headers, raw credential payloads, or merged config dumps.
|
||||
- Avoid logging raw forwarded headers or full trusted network inventories above `debug`.
|
||||
- Keep warning/error logs useful without echoing attacker-controlled payloads unnecessarily.
|
||||
|
||||
### 5. Keep summaries aggregated
|
||||
|
||||
- Replace multi-line startup banners or checklist logs with one structured event.
|
||||
- If metrics already express a concept, avoid duplicating it with many `info!` lines.
|
||||
- Prefer counts, modes, and sources over inventories unless debug detail is truly needed.
|
||||
|
||||
### 6. Update guardrails when needed
|
||||
|
||||
- Broad logging cleanup should usually extend `scripts/check_logging_guardrails.sh`.
|
||||
- Add forbidden patterns only for styles the repo has intentionally retired:
|
||||
- sentence-style lifecycle logs
|
||||
- noisy hot-path `info!`
|
||||
- checklist-style summary logs
|
||||
- legacy fallback wording that has been replaced by structured fields
|
||||
- Keep guardrails concrete and grep-friendly.
|
||||
|
||||
### 7. Validate manually
|
||||
|
||||
Use the smallest relevant set:
|
||||
|
||||
```bash
|
||||
cargo fmt --all --check
|
||||
./scripts/check_logging_guardrails.sh
|
||||
cargo check -p <affected-crate>
|
||||
cargo test -p <affected-crate>
|
||||
```
|
||||
|
||||
For broader Rust changes, add:
|
||||
|
||||
```bash
|
||||
./scripts/check_unsafe_code_allowances.sh
|
||||
./scripts/check_architecture_migration_rules.sh
|
||||
cargo clippy -p <affected-crates> --all-targets -- -D warnings
|
||||
```
|
||||
|
||||
## RustFS-Specific Notes
|
||||
|
||||
- The durable RustFS logging direction is `event + component + subsystem + state/result + key context fields`.
|
||||
- `crates/concurrency` and `crates/trusted-proxies` are examples of this style for lifecycle, fallback, and cloud metadata logs.
|
||||
- `scripts/check_logging_guardrails.sh` is the enforcement point for preventing removed log styles from returning.
|
||||
|
||||
## References
|
||||
|
||||
- Read `references/logging-governance.md` when you need the detailed field set, anti-pattern examples, or guardrail update checklist.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "RustFS Logging Governance"
|
||||
short_description: "Standardize RustFS logs with structured events and guardrails."
|
||||
default_prompt: "Use $rustfs-logging-governance to standardize or review RustFS logging, reduce noise, and update guardrails."
|
||||
@@ -1,285 +0,0 @@
|
||||
# RustFS Logging Governance Reference
|
||||
|
||||
## Workspace Scope Map
|
||||
|
||||
Use `Cargo.toml` `[workspace].members` as the source of truth for crate membership. When doing a broad logging sweep, classify crates by operational role so logs stay consistent within each role.
|
||||
|
||||
### Core Server And Request Handling
|
||||
|
||||
- `rustfs`
|
||||
- Role: top-level server, startup, auth, admin wiring, S3 request handling.
|
||||
- Logging focus: startup lifecycle, config summaries, authn/authz failures, protocol entrypoints, degraded subsystems.
|
||||
- `crates/protocols`
|
||||
- Role: protocol integrations such as FTP, SFTP, WebDAV, and related server-side protocol layers.
|
||||
- Logging focus: listener lifecycle, per-protocol enablement/disablement, request bridge failures.
|
||||
- `crates/madmin`
|
||||
- Role: admin API contracts and management interfaces.
|
||||
- Logging focus: admin action boundaries, validation failures, compatibility warnings.
|
||||
- `crates/trusted-proxies`
|
||||
- Role: forwarded IP trust, proxy chain validation, cloud metadata sources.
|
||||
- Logging focus: direct/trusted/fallback decisions, degraded metadata fetches, aggregated config summaries.
|
||||
- `crates/keystone`
|
||||
- Role: Keystone auth integration.
|
||||
- Logging focus: integration enablement, upstream auth failures, config safety without credential leakage.
|
||||
|
||||
### Storage, Healing, And Data Plane
|
||||
|
||||
- `crates/ecstore`
|
||||
- Role: erasure-coded storage implementation and peer/store initialization.
|
||||
- Logging focus: disk/peer lifecycle, storage fallback, object I/O failures, avoid per-object noise.
|
||||
- `crates/heal`
|
||||
- Role: healing orchestration and repair workflows.
|
||||
- Logging focus: scheduler lifecycle, repair decisions, backlog or skipped work summaries, avoid repetitive task spam at `info`.
|
||||
- `crates/scanner`
|
||||
- Role: data integrity scanning and health monitoring.
|
||||
- Logging focus: scan lifecycle, compaction/deep-heal transitions, lag/backlog, noisy folder iteration should stay at `debug/trace`.
|
||||
- `crates/object-capacity`
|
||||
- Role: capacity scan and refresh core.
|
||||
- Logging focus: refresh lifecycle, degraded capacity sources, aggregate stats rather than per-object chatter.
|
||||
- `crates/filemeta`
|
||||
- Role: file metadata parsing and helpers.
|
||||
- Logging focus: parse failures, schema/format mismatch, avoid dumping raw metadata payloads.
|
||||
- `crates/storage-api`
|
||||
- Role: storage contracts and shared data plane interfaces.
|
||||
- Logging focus: contract mismatch and boundary diagnostics, usually low-volume.
|
||||
- `crates/checksums`
|
||||
- Role: checksum helpers and validation.
|
||||
- Logging focus: integrity failures and compatibility mismatches, not per-chunk success logs.
|
||||
- `crates/zip`
|
||||
- Role: ZIP handling and compression helpers.
|
||||
- Logging focus: parse/extract failures, archive path safety issues, avoid verbose file-by-file success logs.
|
||||
|
||||
### Security, Identity, And Policy
|
||||
|
||||
- `crates/iam`
|
||||
- Role: identity and access management.
|
||||
- Logging focus: authz decision boundaries, imported payload safety, do not leak principals, secrets, or claims.
|
||||
- `crates/policy`
|
||||
- Role: policy modeling and evaluation.
|
||||
- Logging focus: deny/allow decision context, parser/validation failures, no raw secret-bearing request dumps.
|
||||
- `crates/credentials`
|
||||
- Role: credential handling.
|
||||
- Logging focus: never log secrets or tokens; only safe identifiers and redacted states.
|
||||
- `crates/kms`
|
||||
- Role: key management service integration.
|
||||
- Logging focus: init/health/fallback, key-source availability, never log key material.
|
||||
- `crates/crypto`
|
||||
- Role: cryptographic helpers and security primitives.
|
||||
- Logging focus: only algorithm or mode state, not plaintext, ciphertext, or secret-derived material.
|
||||
- `crates/security-governance`
|
||||
- Role: security governance contracts.
|
||||
- Logging focus: policy/state transitions and enforcement diagnostics.
|
||||
- `crates/signer`
|
||||
- Role: request signing helpers.
|
||||
- Logging focus: signature validation failures without expected-signature leakage.
|
||||
|
||||
### Notifications, Audit, And Targets
|
||||
|
||||
- `crates/notify`
|
||||
- Role: notification dispatch, runtime facade, notifier implementations.
|
||||
- Logging focus: target lifecycle, dispatch summaries, stream lag/backpressure, avoid per-event success spam.
|
||||
- `crates/audit`
|
||||
- Role: audit target fan-out and audit pipeline management.
|
||||
- Logging focus: pipeline lifecycle, target availability, batch dispatch summaries, avoid noisy "started successfully" prose.
|
||||
- `crates/targets`
|
||||
- Role: target-specific configuration and utilities used by fan-out style systems.
|
||||
- Logging focus: target selection, config validation, per-target degraded state.
|
||||
- `crates/s3-types`
|
||||
- Role: S3 event and type definitions.
|
||||
- Logging focus: usually minimal; keep logging at integration boundaries rather than low-level type crates.
|
||||
- `crates/s3-ops`
|
||||
- Role: S3 operation definitions and mapping.
|
||||
- Logging focus: mapping/contract failures, unsupported combinations, not normal-path request spam.
|
||||
|
||||
### Concurrency, Locking, And Runtime Foundations
|
||||
|
||||
- `crates/concurrency`
|
||||
- Role: timeout, locking, backpressure, and I/O scheduling facade.
|
||||
- Logging focus: lifecycle transitions and degraded states, not high-frequency worker/permit churn at `info`.
|
||||
- `crates/lock`
|
||||
- Role: distributed locking implementation.
|
||||
- Logging focus: lock lifecycle, contention anomalies, lock ordering or timeout diagnostics.
|
||||
- `crates/tls-runtime`
|
||||
- Role: shared TLS runtime foundation.
|
||||
- Logging focus: certificate lifecycle, reload/fallback, validation failures without sensitive dumps.
|
||||
- `crates/obs`
|
||||
- Role: observability helpers.
|
||||
- Logging focus: this crate shapes other crates' telemetry conventions; avoid recursive or redundant summaries.
|
||||
- `crates/io-core`
|
||||
- Role: zero-copy I/O core primitives.
|
||||
- Logging focus: keep very sparse; prefer metrics unless failures are actionable.
|
||||
- `crates/io-metrics`
|
||||
- Role: I/O metrics collection.
|
||||
- Logging focus: typically minimal; metrics should carry the hot-path signal.
|
||||
- `crates/rio`
|
||||
- Role: Rust I/O utility layer.
|
||||
- Logging focus: compatibility or runtime boundary failures, not fast-path internals.
|
||||
- `crates/rio-v2`
|
||||
- Role: next-generation I/O compatibility layer.
|
||||
- Logging focus: migration/feature-mode differences and degraded fallback between I/O paths.
|
||||
- `crates/utils`
|
||||
- Role: shared helpers.
|
||||
- Logging focus: usually avoid direct logging in generic helpers unless the helper is itself an operational boundary.
|
||||
- `crates/common`
|
||||
- Role: shared data structures and helpers.
|
||||
- Logging focus: same principle as `utils`; prefer callers to log context-rich events.
|
||||
- `crates/config`
|
||||
- Role: configuration management.
|
||||
- Logging focus: config source, fallback, validation, and summary aggregation; avoid dumping merged configs.
|
||||
- `crates/data-usage`
|
||||
- Role: shared data usage models and algorithms.
|
||||
- Logging focus: refresh lifecycle, summary stats, and degraded reads.
|
||||
|
||||
### Schema, Contracts, And API Support
|
||||
|
||||
- `crates/protos`
|
||||
- Role: protobuf definitions.
|
||||
- Logging focus: usually none inside the crate; emit logs at decode/use boundaries.
|
||||
- `crates/extension-schema`
|
||||
- Role: extension schema contracts.
|
||||
- Logging focus: schema validation and compatibility mismatches.
|
||||
- `crates/s3select-api`
|
||||
- Role: S3 Select API interfaces.
|
||||
- Logging focus: request validation and unsupported feature boundaries.
|
||||
- `crates/s3select-query`
|
||||
- Role: S3 Select query engine.
|
||||
- Logging focus: query parse/planning/execution failures, avoid row-level spam.
|
||||
- `crates/protocols`
|
||||
- Role: non-S3 protocol support.
|
||||
- Logging focus: see core server section; keep per-request verbosity below `info`.
|
||||
|
||||
### Testing And Non-Production Crates
|
||||
|
||||
- `crates/e2e_test`
|
||||
- Role: end-to-end tests.
|
||||
- Logging focus: test clarity matters more than production governance, but avoid copying test-only logging style into production crates.
|
||||
|
||||
## Current Guardrail Coverage Map
|
||||
|
||||
`scripts/check_logging_guardrails.sh` currently enforces retired patterns in these high-signal areas:
|
||||
|
||||
- `rustfs/src/main.rs`
|
||||
- `rustfs/src/startup_iam.rs`
|
||||
- `rustfs/src/auth.rs`
|
||||
- `rustfs/src/protocols/client.rs`
|
||||
- `crates/audit/src/pipeline.rs`
|
||||
- `crates/audit/src/system.rs`
|
||||
- `crates/audit/src/global.rs`
|
||||
- `crates/notify/src/config_manager.rs`
|
||||
- `crates/notify/src/runtime_facade.rs`
|
||||
- `crates/notify/src/notifier.rs`
|
||||
- `crates/ecstore/src/store/peer.rs`
|
||||
- `crates/ecstore/src/store/init.rs`
|
||||
- `crates/ecstore/src/tier/tier.rs`
|
||||
- `crates/concurrency/src/workers.rs`
|
||||
- `crates/concurrency/src/manager.rs`
|
||||
- `crates/concurrency/src/lock.rs`
|
||||
- `crates/concurrency/src/deadlock.rs`
|
||||
- `crates/trusted-proxies/src/global.rs`
|
||||
- `crates/trusted-proxies/src/config/loader.rs`
|
||||
- `crates/trusted-proxies/src/proxy/metrics.rs`
|
||||
- `crates/trusted-proxies/src/proxy/validator.rs`
|
||||
- `crates/trusted-proxies/src/proxy/chain.rs`
|
||||
- `crates/trusted-proxies/src/middleware/service.rs`
|
||||
- `crates/trusted-proxies/src/cloud/detector.rs`
|
||||
- `crates/trusted-proxies/src/cloud/ranges.rs`
|
||||
- `crates/trusted-proxies/src/cloud/metadata/aws.rs`
|
||||
- `crates/trusted-proxies/src/cloud/metadata/azure.rs`
|
||||
- `crates/trusted-proxies/src/cloud/metadata/gcp.rs`
|
||||
|
||||
When expanding coverage, prefer crates with:
|
||||
|
||||
- repeated sentence-style lifecycle logs
|
||||
- high-frequency success-path `info!`
|
||||
- startup/config checklist banners
|
||||
- security-sensitive fallback wording
|
||||
- external fetch/retry/fallback flows
|
||||
|
||||
That typically means the next broad candidates are `rustfs`, `crates/notify`, `crates/audit`, `crates/targets`, `crates/heal`, and `crates/scanner`.
|
||||
|
||||
## Event Model
|
||||
|
||||
Prefer this structure when the fields are available:
|
||||
|
||||
- `event`
|
||||
- `component`
|
||||
- `subsystem`
|
||||
- `state` or `result`
|
||||
- stable context fields such as:
|
||||
- `enabled`
|
||||
- `implementation`
|
||||
- `validation_mode`
|
||||
- `peer_ip`
|
||||
- `client_ip`
|
||||
- `proxy_hops`
|
||||
- `duration_ms`
|
||||
- `fallback`
|
||||
- `reason`
|
||||
- `range_count`
|
||||
- `hold_time_ms`
|
||||
- `available_slots`
|
||||
- `total_slots`
|
||||
- `permits_in_use`
|
||||
|
||||
## Level Policy
|
||||
|
||||
- `error`: the operation fails and callers or security guarantees are affected.
|
||||
- `warn`: a degraded path, fallback, suspicious request, or operator-actionable config issue occurs.
|
||||
- `info`: a low-frequency lifecycle or mode transition occurs.
|
||||
- `debug`: useful diagnostics exist but normal operators do not need them all the time.
|
||||
- `trace`: hot-path and repetitive success-path details occur.
|
||||
|
||||
## Preferred Patterns
|
||||
|
||||
- Use a short message label:
|
||||
- `"trusted proxy validation failed"`
|
||||
- `"concurrency manager state changed"`
|
||||
- `"trusted proxy cloud metadata loaded"`
|
||||
- Put key meaning into fields, not only the message text.
|
||||
- Aggregate config or metrics summaries into one log event.
|
||||
|
||||
## Retired Patterns
|
||||
|
||||
These should usually be removed or replaced:
|
||||
|
||||
- Sentence-style lifecycle logs:
|
||||
- `info!("Concurrency manager stopped")`
|
||||
- `info!("Trusted Proxies module initialized")`
|
||||
- Checklist or banner logs:
|
||||
- `info!("=== Application Configuration ===")`
|
||||
- `info!("Available metrics:")`
|
||||
- Hot-path noise:
|
||||
- `info!("worker take, {}", *available)`
|
||||
- `debug!("Proxy validation successful in {:?}", duration)`
|
||||
- Legacy fallback prose:
|
||||
- `"Request from private network but not trusted: ..."`
|
||||
- `"Cloud metadata fetching is disabled"`
|
||||
|
||||
## Guardrail Update Checklist
|
||||
|
||||
When extending `scripts/check_logging_guardrails.sh`:
|
||||
|
||||
1. Add the touched files to `checked_files`.
|
||||
2. Add only legacy patterns that have been intentionally retired.
|
||||
3. Keep patterns literal and grep-friendly.
|
||||
4. Run the guardrail script after changes.
|
||||
5. Avoid adding patterns for logs that are still valid elsewhere in the repo.
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
For logging-only changes:
|
||||
|
||||
```bash
|
||||
cargo fmt --all --check
|
||||
./scripts/check_logging_guardrails.sh
|
||||
cargo check -p <affected-crate>
|
||||
cargo test -p <affected-crate>
|
||||
```
|
||||
|
||||
For broader Rust changes:
|
||||
|
||||
```bash
|
||||
./scripts/check_unsafe_code_allowances.sh
|
||||
./scripts/check_architecture_migration_rules.sh
|
||||
cargo clippy -p <affected-crates> --all-targets -- -D warnings
|
||||
```
|
||||
@@ -1,124 +0,0 @@
|
||||
---
|
||||
name: rustfs-release-version-bump
|
||||
description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation."
|
||||
---
|
||||
# RustFS Release Version Bump
|
||||
|
||||
Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).
|
||||
|
||||
Validated baseline: release pattern used in PR `#2957`.
|
||||
|
||||
## Required inputs
|
||||
|
||||
- Exact target version, for example `1.0.0-beta.4`.
|
||||
- Delivery scope:
|
||||
- Local only (`edit/verify`).
|
||||
- Local + git (`commit/push`).
|
||||
- Full GitHub flow (`commit/push/PR`).
|
||||
|
||||
If target version is missing or ambiguous, stop and ask before editing.
|
||||
|
||||
## Read before editing
|
||||
|
||||
- `AGENTS.md` (root and nearest path-specific files).
|
||||
- `.github/pull_request_template.md`.
|
||||
- Current branch status and diff against `origin/main`.
|
||||
|
||||
## Default release file scope
|
||||
|
||||
Treat the following file list as the default checklist for each release bump:
|
||||
|
||||
- `Cargo.toml`
|
||||
- `Cargo.lock`
|
||||
- `README.md`
|
||||
- `README_ZH.md`
|
||||
- `flake.nix`
|
||||
- `helm/rustfs/Chart.yaml`
|
||||
- `rustfs.spec`
|
||||
|
||||
Only drop a file when the current repository release process clearly no longer requires it.
|
||||
|
||||
## Hard release policy
|
||||
|
||||
- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
|
||||
- Helm chart version mapping follows `beta.N -> 0.N.0`.
|
||||
- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
|
||||
- Do not change these rules without explicit confirmation.
|
||||
|
||||
## Step-by-step workflow
|
||||
|
||||
1. Confirm intent and isolate scope
|
||||
- Confirm target version string exactly.
|
||||
- Confirm whether user requested local-only or full GitHub flow.
|
||||
- Inspect current branch and ensure only release-related files are touched for this task.
|
||||
|
||||
2. Update workspace versions
|
||||
- Bump `[workspace.package].version` in `Cargo.toml`.
|
||||
- Bump internal workspace crate dependency versions in `Cargo.toml`.
|
||||
- Update `Cargo.lock` so workspace package versions match target version.
|
||||
- Re-scan for partial leftovers.
|
||||
|
||||
3. Update release assets
|
||||
- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
|
||||
- `flake.nix`: update package version to target version.
|
||||
- `helm/rustfs/Chart.yaml`:
|
||||
- `appVersion` = target version.
|
||||
- `version` follows chart mapping rule, for example:
|
||||
- `1.0.0-beta.3` -> `0.3.0`
|
||||
- `1.0.0-beta.4` -> `0.4.0`
|
||||
- `rustfs.spec`:
|
||||
- Set `Release` to prerelease suffix (example `beta.4`).
|
||||
- Add/update top changelog entry with exact format:
|
||||
- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
|
||||
- `- Update RPM package to RustFS 1.0.0-beta.4`
|
||||
- Changelog identity and time must come from current environment:
|
||||
- `git config --get user.name`
|
||||
- `git config --get user.email`
|
||||
- `date '+%a %b %d %Y'`
|
||||
- Changelog version text must match target release version exactly.
|
||||
|
||||
4. Verify before shipping
|
||||
- Run:
|
||||
- `cargo fmt --all`
|
||||
- `cargo fmt --all --check`
|
||||
- `make pre-commit`
|
||||
- If verification passes, run `cargo clean`.
|
||||
- If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
|
||||
|
||||
5. Commit strategy
|
||||
- Preferred split when both parts changed:
|
||||
- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
|
||||
- `chore(release): align release assets for <version>` for docs and packaging files.
|
||||
- If user asks for one commit, use one commit.
|
||||
- Stage only intended release files; do not include unrelated working tree changes.
|
||||
|
||||
6. Push and PR
|
||||
- Push branch:
|
||||
- `git push -u origin <branch>` (first push), or `git push` (tracking already exists).
|
||||
- Create PR with template headings unchanged:
|
||||
- `gh pr create --base main --head <branch> --title ... --body-file ...`
|
||||
- PR title/body must be English.
|
||||
- Use `N/A` for non-applicable template sections.
|
||||
- Include verification commands and any `BLOCKED` reason clearly.
|
||||
|
||||
## Recommended check commands
|
||||
|
||||
- `git status --short --branch`
|
||||
- `git diff --name-only origin/main...HEAD`
|
||||
- `git diff --stat origin/main...HEAD`
|
||||
- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
|
||||
- `cargo fmt --all`
|
||||
- `cargo fmt --all --check`
|
||||
- `make pre-commit`
|
||||
- `cargo clean`
|
||||
|
||||
## Output contract
|
||||
|
||||
When using this skill, always report:
|
||||
|
||||
- Target version.
|
||||
- Files changed.
|
||||
- Any assumptions or uncertainties requiring confirmation.
|
||||
- Verification result (`PASSED` or `BLOCKED`) with key evidence.
|
||||
- Commit message(s) used.
|
||||
- Push status and PR URL when GitHub flow is requested.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "RustFS Release Bump"
|
||||
short_description: "Prepare RustFS release branches like PR #2957."
|
||||
default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."
|
||||
@@ -1,136 +0,0 @@
|
||||
---
|
||||
name: security-advisory-lessons
|
||||
description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
|
||||
---
|
||||
|
||||
# RustFS Security Advisory Lessons
|
||||
|
||||
Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
|
||||
|
||||
When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
|
||||
|
||||
```bash
|
||||
gh api repos/rustfs/rustfs/security-advisories --paginate \
|
||||
--jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
|
||||
```
|
||||
|
||||
Fetch full advisory details only when the live summary suggests a new or changed lesson:
|
||||
|
||||
```bash
|
||||
gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
|
||||
```
|
||||
|
||||
For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Scope the change
|
||||
- Identify touched routes, protocol frontends, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
|
||||
- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, and console preview/auth code.
|
||||
|
||||
### 2. Map to advisory classes
|
||||
- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
|
||||
- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
|
||||
|
||||
### 3. Verify fail-closed behavior
|
||||
- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
|
||||
- Prefer exact action/permission checks over broad helper calls or inferred ownership.
|
||||
- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
|
||||
|
||||
### 4. Require regression evidence
|
||||
- For behavior changes, add focused negative tests that reproduce the advisory class.
|
||||
- For sensitive fixes, include tests for the bypass form, not only the happy path.
|
||||
- If a test is impractical, explain the residual risk and provide a manual verification command.
|
||||
|
||||
### 5. Report clearly
|
||||
- Lead with concrete findings and file/line evidence.
|
||||
- Separate proven vulnerabilities from hardening risks.
|
||||
- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
|
||||
|
||||
## Advisory-Derived Guardrails
|
||||
|
||||
### Auth and admin authorization
|
||||
- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
|
||||
- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
|
||||
- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
|
||||
- Read-only admin APIs such as metrics, server info, and diagnostics still require admin authorization; checking only that credentials exist is not enough.
|
||||
- Replication admin reads can expose remote target credentials; list/get target endpoints require replication/admin authorization and must not return secrets to low-privilege callers.
|
||||
- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
|
||||
|
||||
### IAM and service accounts
|
||||
- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
|
||||
- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims.
|
||||
- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
|
||||
- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
|
||||
|
||||
### S3 copy, multipart, and presigned POST
|
||||
- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
|
||||
- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
|
||||
- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
|
||||
|
||||
### Protocol frontends and IAM parity
|
||||
- FTP/FTPS, SFTP, gateway, and other protocol drivers must enforce IAM per operation before calling storage backends; authentication to a protocol listener is not authorization.
|
||||
- Match protocol commands to the same S3 actions as HTTP, such as `RETR` to `GetObject`, `SIZE`/`MDTM` to `HeadObject`, `MKD` to `CreateBucket`, and bucket probes to `ListBucket` or `HeadBucket`.
|
||||
- Review every handler in a protocol driver, not only the changed handler, because RustFS advisories show mixed guarded and unguarded siblings in the same driver.
|
||||
- Regression tests for protocol frontends should deny the shared authorization hook and prove the backend is not reached for the denied command.
|
||||
- Compare protocol secrets in constant time, normalize invalid-user and invalid-secret failures where practical, and add rate limiting before exposing password-style protocol endpoints.
|
||||
|
||||
### Paths, object keys, and filesystem access
|
||||
- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
|
||||
- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
|
||||
- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
|
||||
- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries.
|
||||
|
||||
### Secrets, default credentials, and crypto
|
||||
- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
|
||||
- Defaults for root credentials and internode/RPC auth must fail closed for network-reachable deployments or generate per-install random secrets; warnings alone are not a security boundary.
|
||||
- Keep cryptographic roles separated: root S3 credentials, RPC HMAC keys, and STS/JWT signing keys must not be reused or deterministically derived from each other.
|
||||
- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
|
||||
- Plan key rotation and key IDs when removing exposed keys.
|
||||
|
||||
### Logging and debug output
|
||||
- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
|
||||
- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
|
||||
- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
|
||||
|
||||
### RPC, parsing, and panic safety
|
||||
- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
|
||||
- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
|
||||
- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
|
||||
- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call.
|
||||
|
||||
### Browser, CORS, and console surfaces
|
||||
- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
|
||||
- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
|
||||
- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
|
||||
- Preview safety must be based on trusted content type and sandboxing, not object names or extensions such as `.pdf`.
|
||||
- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps.
|
||||
|
||||
### Profiling, debug, and health endpoints
|
||||
- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
|
||||
- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
|
||||
- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
|
||||
|
||||
### Trusted proxy and network identity
|
||||
- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
|
||||
- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
|
||||
- Add tests for direct spoofed headers and trusted-proxy headers.
|
||||
|
||||
### SSE and storage invariants
|
||||
- Encryption metadata is not proof that bytes were encrypted on disk.
|
||||
- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
|
||||
- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
|
||||
|
||||
## Review Prompts
|
||||
|
||||
Use these prompts while reviewing a diff:
|
||||
|
||||
- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
|
||||
- Does a non-HTTP protocol path call the same authorization boundary as the S3 API before touching storage?
|
||||
- Does a public/default/empty config change security behavior from fail-closed to fail-open?
|
||||
- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
|
||||
- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
|
||||
- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
|
||||
- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
|
||||
- Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
|
||||
- Does the test prove the exploit form is denied, or only that the intended form still works?
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Security Advisory Lessons"
|
||||
short_description: "Apply advisory lessons in reviews."
|
||||
default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes."
|
||||
@@ -1,119 +0,0 @@
|
||||
# RustFS Advisory Pattern Map
|
||||
|
||||
This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
|
||||
|
||||
When current advisory state, severity, URLs, or full text matters, fetch it live:
|
||||
|
||||
```bash
|
||||
gh api repos/rustfs/rustfs/security-advisories --paginate \
|
||||
--jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
|
||||
gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
|
||||
```
|
||||
|
||||
Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
|
||||
|
||||
## Pattern Index
|
||||
|
||||
### Admin authorization and route exposure
|
||||
|
||||
- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
|
||||
- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
|
||||
- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
|
||||
- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
|
||||
- `GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
|
||||
- `GHSA-796f-j7xp-hwf4`: `/rustfs/admin/v3/list-remote-targets` checked only that credentials existed and leaked replication target credentials. Lesson: replication target reads are privileged admin operations, and stored remote credentials require strict authz plus response redaction review.
|
||||
- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
|
||||
|
||||
### IAM import, service accounts, and privilege boundaries
|
||||
|
||||
- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
|
||||
- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
|
||||
- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
|
||||
|
||||
### S3 copy, multipart, and upload policy validation
|
||||
|
||||
- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
|
||||
- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
|
||||
- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
|
||||
|
||||
### Protocol frontends and IAM parity
|
||||
|
||||
- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
|
||||
- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
|
||||
- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
|
||||
|
||||
### Filesystem paths and object key traversal
|
||||
|
||||
- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
|
||||
- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
|
||||
- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
|
||||
|
||||
### Secrets, defaults, and cryptographic misuse
|
||||
|
||||
- `GHSA-j59h-h7q5-q348`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, and console surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
|
||||
- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
|
||||
- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
|
||||
- `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
|
||||
- `GHSA-m77q-r63m-pj89`: STS JWTs used the root secret key as the shared token signing key, allowing token forgery when the root secret was known. Lesson: STS signing keys need key separation, rotation, and key IDs; do not reuse root credentials for JWT/HMAC signing.
|
||||
- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
|
||||
|
||||
### Sensitive logging and debug output
|
||||
|
||||
- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
|
||||
- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
|
||||
- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
|
||||
|
||||
### RPC input validation and panic safety
|
||||
|
||||
- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
|
||||
- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
|
||||
- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
|
||||
|
||||
### Browser, CORS, and console isolation
|
||||
|
||||
- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
|
||||
- `GHSA-7gcx-wg4x-q9x6`: an incomplete preview fix reintroduced extension-based PDF detection and bypassed the sandboxed fallback for attacker-controlled content. Lesson: browser-surface fixes need regression tests for alternate viewers and file-type branches, and preview trust must come from validated content type plus sandboxing rather than object names.
|
||||
- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
|
||||
|
||||
### Trusted proxy and source IP conditions
|
||||
|
||||
- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
|
||||
|
||||
### SSE and on-disk storage invariants
|
||||
|
||||
- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
|
||||
|
||||
### Serde deserialization and input validation
|
||||
|
||||
- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
|
||||
- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
|
||||
- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` or clamp.
|
||||
- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
|
||||
|
||||
## Useful Search Seeds
|
||||
|
||||
Use these targeted searches when a diff touches security-sensitive code:
|
||||
|
||||
```bash
|
||||
rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
|
||||
rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
|
||||
rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
|
||||
rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
|
||||
rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
|
||||
rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
|
||||
rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
|
||||
rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
|
||||
rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
|
||||
```
|
||||
|
||||
## Minimum Regression Test Expectations
|
||||
|
||||
- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
|
||||
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
|
||||
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
|
||||
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
|
||||
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
|
||||
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
|
||||
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
|
||||
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
|
||||
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
name: test-coverage-improver
|
||||
description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
|
||||
---
|
||||
|
||||
# Test Coverage Improver
|
||||
|
||||
Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
|
||||
|
||||
## Usage assumptions
|
||||
- Focus scope is either changed lines/files, a module, or the whole repository.
|
||||
- Coverage artifact must be generated or provided in a supported format.
|
||||
- If required context is missing, call out assumptions explicitly before proposing work.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Define scope and baseline
|
||||
- Confirm target language, framework, and branch.
|
||||
- Confirm whether the scope is changed files only or full-repo.
|
||||
|
||||
2. Produce coverage snapshot
|
||||
- Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
|
||||
- JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
|
||||
- Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
|
||||
- Collect total, per-file, and changed-line coverage.
|
||||
|
||||
3. Rank highest-risk gaps
|
||||
- Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
|
||||
- Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
|
||||
- Keep shortlist to 5–8 gaps.
|
||||
- For each gap, capture: file, lines, uncovered branches, and estimated risk score.
|
||||
|
||||
4. Propose high-impact tests
|
||||
- For each shortlisted gap, output:
|
||||
- Intent and expected behavior.
|
||||
- Normal, edge, and failure scenarios.
|
||||
- Assertions and side effects to verify.
|
||||
- Setup needs (fixtures, mocks, integration dependencies).
|
||||
- Estimated effort (`S/M/L`).
|
||||
|
||||
5. Close with validation plan
|
||||
- State which gaps remain after proposals.
|
||||
- Provide concrete verification command and acceptance threshold.
|
||||
- List assumptions or blockers (environment, fixtures, flaky dependencies).
|
||||
|
||||
## Output template
|
||||
|
||||
### Coverage Snapshot
|
||||
- total / branch coverage
|
||||
- changed-file coverage
|
||||
- top missing regions by size
|
||||
|
||||
### Top Gaps (ranked)
|
||||
- `path:line-range` | risk score | why critical
|
||||
|
||||
### Test Proposals
|
||||
- `path:line-range`
|
||||
- Test name
|
||||
- scenarios
|
||||
- assertions
|
||||
- effort
|
||||
|
||||
### Validation Plan
|
||||
- command
|
||||
- pass criteria
|
||||
- remaining risk
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Test Coverage Improver"
|
||||
short_description: "Find top uncovered risk areas and propose high-impact tests."
|
||||
default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."
|
||||
@@ -1,25 +0,0 @@
|
||||
# Coverage Gap Prioritization Guide
|
||||
|
||||
Use this rubric for each uncovered area.
|
||||
|
||||
Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
|
||||
|
||||
- Criticality:
|
||||
- 5: authz/authn, data-loss, payment/consistency path
|
||||
- 4: state mutation, cache invalidation, scheduling
|
||||
- 3: error handling + fallbacks in user-visible flows
|
||||
- 2: parsing/format conversion paths
|
||||
- 1: logging-only or low-impact utilities
|
||||
|
||||
- CoverageDebt:
|
||||
- 0: 0–5 uncovered lines
|
||||
- 1: 6–20 uncovered lines
|
||||
- 2: 21–40 uncovered lines
|
||||
- 3: 41+ uncovered lines
|
||||
|
||||
- Volatility:
|
||||
- 1: stable legacy code with few recent edits
|
||||
- 2: changed in last 2 releases
|
||||
- 3: touched in last 30 days or currently in active PR
|
||||
|
||||
Sort by score descending, then by business impact.
|
||||
@@ -1,34 +0,0 @@
|
||||
---
|
||||
name: tier-debug
|
||||
description: Debug ILM tiering / lifecycle transition issues — NoSuchVersion on tier GET, restore failures, xl.meta inspection, remote-tier versionId tracing. Use when investigating tiered/transitioned objects, warm backends, or transition metadata.
|
||||
---
|
||||
|
||||
# Tier / ILM Debugging
|
||||
|
||||
Full playbook: [docs/operations/tier-ilm-debugging.md](../../../docs/operations/tier-ilm-debugging.md)
|
||||
— read it before changing tier code.
|
||||
|
||||
Quick moves:
|
||||
|
||||
```bash
|
||||
# Inspect transition metadata on disk (one xl.meta per erasure shard disk)
|
||||
cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/{bucket}/{object}/xl.meta"
|
||||
|
||||
# Trace what versionId is sent to the remote tier
|
||||
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./target/debug/rustfs …
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- `transition_ver_id: <none>` → correct for an unversioned tier bucket; no
|
||||
`versionId` must be sent on tier GET/DELETE.
|
||||
- `transition_ver_id: 00000000-…` (nil) → corrupt legacy write-back; readers
|
||||
must filter it out, never send it.
|
||||
- Empty-string `transitioned-versionID` metadata under both
|
||||
`x-rustfs-internal-*` and `x-minio-internal-*` keys → object went to an
|
||||
unversioned tier bucket.
|
||||
|
||||
Code entry points: `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs`
|
||||
(ILM actions), `crates/ecstore/src/services/tier/` (warm backends),
|
||||
`crates/filemeta/src/filemeta/version.rs` (metadata read/write + regression
|
||||
tests).
|
||||
@@ -1,26 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# RustFS Cargo configuration
|
||||
|
||||
# Enable tokio_unstable cfg for dial9-tokio-telemetry support
|
||||
# This allows dial9 to hook into Tokio's internal runtime events
|
||||
[build]
|
||||
# Enable Tokio unstable features required by dial9-tokio-telemetry for runtime tracing.
|
||||
# See: https://docs.rs/tokio/latest/tokio/#unstable-features
|
||||
rustflags = ["--cfg", "tokio_unstable"]
|
||||
|
||||
# Enable frame pointers for CPU profiling (Linux only, optional but recommended)
|
||||
# Uncomment the following line for better CPU profiling data
|
||||
# rustflags = ["--cfg", "tokio_unstable", "-C", "force-frame-pointers=yes"]
|
||||
@@ -1 +0,0 @@
|
||||
../.agents/skills
|
||||
@@ -1,64 +0,0 @@
|
||||
## —— Development/Source builds using direct buildx commands ---------------------------------------
|
||||
|
||||
.PHONY: docker-dev
|
||||
docker-dev: ## Build dev multi-arch image (cannot load locally)
|
||||
@echo "🏗️ Building multi-architecture development Docker images with buildx..."
|
||||
@echo "💡 This builds from source code and is intended for local development and testing"
|
||||
@echo "⚠️ Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
|
||||
$(DOCKER_CLI) buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
--file $(DOCKERFILE_SOURCE) \
|
||||
--tag rustfs:source-latest \
|
||||
--tag rustfs:dev-latest \
|
||||
.
|
||||
|
||||
.PHONY: docker-dev-local
|
||||
docker-dev-local: ## Build dev single-arch image (local load)
|
||||
@echo "🏗️ Building single-architecture development Docker image for local use..."
|
||||
@echo "💡 This builds from source code for the current platform and loads locally"
|
||||
$(DOCKER_CLI) buildx build \
|
||||
--file $(DOCKERFILE_SOURCE) \
|
||||
--tag rustfs:source-latest \
|
||||
--tag rustfs:dev-latest \
|
||||
--load \
|
||||
.
|
||||
|
||||
.PHONY: docker-dev-push
|
||||
docker-dev-push: ## Build and push multi-arch development image # e.g (make docker-dev-push REGISTRY=xxx)
|
||||
@if [ -z "$(REGISTRY)" ]; then \
|
||||
echo "❌ Error: Please specify registry, example: make docker-dev-push REGISTRY=ghcr.io/username"; \
|
||||
exit 1; \
|
||||
fi
|
||||
@echo "🚀 Building and pushing multi-architecture development Docker images..."
|
||||
@echo "💡 Pushing to registry: $(REGISTRY)"
|
||||
$(DOCKER_CLI) buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
--file $(DOCKERFILE_SOURCE) \
|
||||
--tag $(REGISTRY)/rustfs:source-latest \
|
||||
--tag $(REGISTRY)/rustfs:dev-latest \
|
||||
--push \
|
||||
.
|
||||
|
||||
.PHONY: dev-env-start
|
||||
dev-env-start: ## Start development container environment
|
||||
@echo "🚀 Starting development environment..."
|
||||
$(DOCKER_CLI) buildx build \
|
||||
--file $(DOCKERFILE_SOURCE) \
|
||||
--tag rustfs:dev \
|
||||
--load \
|
||||
.
|
||||
$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
|
||||
$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
|
||||
$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) \
|
||||
-p 9010:9010 -p 9000:9000 \
|
||||
-v $(shell pwd):/workspace \
|
||||
-it rustfs:dev
|
||||
|
||||
.PHONY: dev-env-stop
|
||||
dev-env-stop: ## Stop development container environment
|
||||
@echo "🛑 Stopping development environment..."
|
||||
$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
|
||||
$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
|
||||
|
||||
.PHONY: dev-env-restart
|
||||
dev-env-restart: dev-env-stop dev-env-start ## Restart development container environment
|
||||
@@ -1,41 +0,0 @@
|
||||
## —— Production builds using docker buildx (for CI/CD and production) -----------------------------
|
||||
|
||||
.PHONY: docker-buildx
|
||||
docker-buildx: ## Build production multi-arch image (no push)
|
||||
@echo "🏗️ Building multi-architecture production Docker images with buildx..."
|
||||
./docker-buildx.sh
|
||||
|
||||
.PHONY: docker-buildx-push
|
||||
docker-buildx-push: ## Build and push production multi-arch image
|
||||
@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
|
||||
./docker-buildx.sh --push
|
||||
|
||||
.PHONY: docker-buildx-version
|
||||
docker-buildx-version: ## Build and version production multi-arch image # e.g (make docker-buildx-version VERSION=v1.0.0)
|
||||
@if [ -z "$(VERSION)" ]; then \
|
||||
echo "❌ Error: Please specify version, example: make docker-buildx-version VERSION=v1.0.0"; \
|
||||
exit 1; \
|
||||
fi
|
||||
@echo "🏗️ Building multi-architecture production Docker images (version: $(VERSION))..."
|
||||
./docker-buildx.sh --release $(VERSION)
|
||||
|
||||
.PHONY: docker-buildx-push-version
|
||||
docker-buildx-push-version: ## Build and version and push production multi-arch image # e.g (make docker-buildx-push-version VERSION=v1.0.0)
|
||||
@if [ -z "$(VERSION)" ]; then \
|
||||
echo "❌ Error: Please specify version, example: make docker-buildx-push-version VERSION=v1.0.0"; \
|
||||
exit 1; \
|
||||
fi
|
||||
@echo "🚀 Building and pushing multi-architecture production Docker images (version: $(VERSION))..."
|
||||
./docker-buildx.sh --release $(VERSION) --push
|
||||
|
||||
.PHONY: docker-buildx-production-local
|
||||
docker-buildx-production-local: ## Build production single-arch image locally
|
||||
@echo "🏗️ Building single-architecture production Docker image locally..."
|
||||
@echo "💡 Alternative to docker-buildx.sh for local testing"
|
||||
$(DOCKER_CLI) buildx build \
|
||||
--file $(DOCKERFILE_PRODUCTION) \
|
||||
--tag rustfs:production-latest \
|
||||
--tag rustfs:latest \
|
||||
--load \
|
||||
--build-arg RELEASE=latest \
|
||||
.
|
||||
@@ -1,16 +0,0 @@
|
||||
## —— Single Architecture Docker Builds (Traditional) ----------------------------------------------
|
||||
|
||||
.PHONY: docker-build-production
|
||||
docker-build-production: ## Build single-arch production image
|
||||
@echo "🏗️ Building single-architecture production Docker image..."
|
||||
@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
|
||||
$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
|
||||
|
||||
.PHONY: docker-build-source
|
||||
docker-build-source: ## Build single-arch source image
|
||||
@echo "🏗️ Building single-architecture source Docker image..."
|
||||
@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
|
||||
DOCKER_BUILDKIT=1 $(DOCKER_CLI) build \
|
||||
--build-arg BUILDKIT_INLINE_CACHE=1 \
|
||||
-f $(DOCKERFILE_SOURCE) -t rustfs:source .
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
## —— Docker-based build (alternative approach) ----------------------------------------------------
|
||||
|
||||
# Usage: make BUILD_OS=ubuntu22.04 build-docker
|
||||
# Output: target/ubuntu22.04/release/rustfs
|
||||
|
||||
.PHONY: build-docker
|
||||
build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
|
||||
build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
|
||||
build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
|
||||
build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
|
||||
@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
|
||||
$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) --load .
|
||||
$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
|
||||
|
||||
.PHONY: docker-inspect-multiarch
|
||||
docker-inspect-multiarch: ## Check image architecture support
|
||||
@if [ -z "$(IMAGE)" ]; then \
|
||||
echo "❌ Error: Please specify image, example: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
|
||||
exit 1; \
|
||||
fi
|
||||
@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
|
||||
docker buildx imagetools inspect $(IMAGE)
|
||||
@@ -1,55 +0,0 @@
|
||||
## —— Local Native Build using build-rustfs.sh script (Recommended) --------------------------------
|
||||
|
||||
.PHONY: build
|
||||
build: ## Build RustFS binary (includes console by default)
|
||||
@echo "🔨 Building RustFS using build-rustfs.sh script..."
|
||||
./build-rustfs.sh
|
||||
|
||||
.PHONY: build-dev
|
||||
build-dev: ## Build RustFS in Development mode
|
||||
@echo "🔨 Building RustFS in development mode..."
|
||||
./build-rustfs.sh --dev
|
||||
|
||||
.PHONY: build-musl
|
||||
build-musl: ## Build x86_64 musl version
|
||||
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
|
||||
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
|
||||
./build-rustfs.sh --platform x86_64-unknown-linux-musl
|
||||
|
||||
.PHONY: build-gnu
|
||||
build-gnu: ## Build x86_64 GNU version
|
||||
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
|
||||
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
|
||||
./build-rustfs.sh --platform x86_64-unknown-linux-gnu
|
||||
|
||||
.PHONY: build-musl-arm64
|
||||
build-musl-arm64: ## Build aarch64 musl version
|
||||
@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
|
||||
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
|
||||
./build-rustfs.sh --platform aarch64-unknown-linux-musl
|
||||
|
||||
.PHONY: build-gnu-arm64
|
||||
build-gnu-arm64: ## Build aarch64 GNU version
|
||||
@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
|
||||
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
|
||||
./build-rustfs.sh --platform aarch64-unknown-linux-gnu
|
||||
|
||||
|
||||
.PHONY: build-cross-all
|
||||
build-cross-all: core-deps ## Build binaries for all architectures
|
||||
@echo "🔧 Building all target architectures..."
|
||||
@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
|
||||
@echo "🔨 Generating protobuf code..."
|
||||
cargo run --bin gproto || true
|
||||
|
||||
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
|
||||
./build-rustfs.sh --platform x86_64-unknown-linux-musl
|
||||
|
||||
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
|
||||
./build-rustfs.sh --platform x86_64-unknown-linux-gnu
|
||||
|
||||
@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
|
||||
./build-rustfs.sh --platform aarch64-unknown-linux-musl
|
||||
|
||||
@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
|
||||
./build-rustfs.sh --platform aarch64-unknown-linux-gnu
|
||||
@@ -1,24 +0,0 @@
|
||||
## —— Check and Inform Dependencies ----------------------------------------------------------------
|
||||
|
||||
# Fatal check
|
||||
# Checks all required dependencies and exits with error if not found
|
||||
# (e.g., cargo, rustfmt)
|
||||
check-%:
|
||||
@command -v $* >/dev/null 2>&1 || { \
|
||||
echo >&2 "❌ '$*' is not installed."; \
|
||||
exit 1; \
|
||||
}
|
||||
|
||||
# Warning-only check
|
||||
# Checks for optional dependencies and issues a warning if not found
|
||||
# (e.g., cargo-nextest for enhanced testing)
|
||||
warn-%:
|
||||
@command -v $* >/dev/null 2>&1 || { \
|
||||
echo >&2 "⚠️ '$*' is not installed."; \
|
||||
}
|
||||
|
||||
# For checking dependencies use check-<dep-name> or warn-<dep-name>
|
||||
.PHONY: core-deps fmt-deps test-deps
|
||||
core-deps: check-cargo ## Check core dependencies
|
||||
fmt-deps: check-rustfmt ## Check lint and formatting dependencies
|
||||
test-deps: warn-cargo-nextest ## Check tests dependencies
|
||||
@@ -1,6 +0,0 @@
|
||||
## —— Deploy using dev_deploy.sh script ------------------------------------------------------------
|
||||
|
||||
.PHONY: deploy-dev
|
||||
deploy-dev: build-musl ## Deploy to dev server
|
||||
@echo "🚀 Deploying to dev server: $${IP}"
|
||||
./scripts/dev_deploy.sh $${IP}
|
||||
@@ -1,38 +0,0 @@
|
||||
## —— Help, Help Build and Help Docker -------------------------------------------------------------
|
||||
|
||||
|
||||
.PHONY: help
|
||||
help: ## Shows This Help Menu
|
||||
echo -e "$$HEADER"
|
||||
grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} /^## / {printf "\n${green}%s${reset}\n", $$0; next} {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}'
|
||||
|
||||
.PHONY: help-build
|
||||
help-build: ## Shows RustFS build help
|
||||
@echo ""
|
||||
@echo "💡 build-rustfs.sh script provides more options, smart detection and binary verification"
|
||||
@echo ""
|
||||
@echo "🔧 Direct usage of build-rustfs.sh script:"
|
||||
@echo ""
|
||||
@echo " ./build-rustfs.sh --help # View script help"
|
||||
@echo " ./build-rustfs.sh --no-console # Build without console resources"
|
||||
@echo " ./build-rustfs.sh --force-console-update # Force update console resources"
|
||||
@echo " ./build-rustfs.sh --dev # Development mode build"
|
||||
@echo " ./build-rustfs.sh --sign # Sign binary files"
|
||||
@echo " ./build-rustfs.sh --platform x86_64-unknown-linux-gnu # Specify target platform"
|
||||
@echo " ./build-rustfs.sh --skip-verification # Skip binary verification"
|
||||
@echo ""
|
||||
|
||||
.PHONY: help-docker
|
||||
help-docker: ## Shows docker environment and suggestion help
|
||||
@echo ""
|
||||
@echo "📋 Environment Variables:"
|
||||
@echo " REGISTRY Image registry address (required for push)"
|
||||
@echo " DOCKERHUB_USERNAME Docker Hub username"
|
||||
@echo " DOCKERHUB_TOKEN Docker Hub access token"
|
||||
@echo " GITHUB_TOKEN GitHub access token"
|
||||
@echo ""
|
||||
@echo "💡 Suggestions:"
|
||||
@echo " Production use: Use docker-buildx* commands (based on precompiled binaries)"
|
||||
@echo " Local development: Use docker-dev* commands (build from source)"
|
||||
@echo " Development environment: Use dev-env-* commands to manage dev containers"
|
||||
@echo ""
|
||||
@@ -1,56 +0,0 @@
|
||||
## —— Code quality and Formatting ------------------------------------------------------------------
|
||||
|
||||
.NOTPARALLEL: fix
|
||||
|
||||
.PHONY: fmt
|
||||
fmt: core-deps fmt-deps ## Format code
|
||||
@echo "🔧 Formatting code..."
|
||||
cargo fmt --all
|
||||
|
||||
.PHONY: fmt-check
|
||||
fmt-check: core-deps fmt-deps ## Check code formatting
|
||||
@echo "📝 Checking code formatting..."
|
||||
cargo fmt --all --check
|
||||
|
||||
.PHONY: clippy-check
|
||||
clippy-check: core-deps ## Run clippy checks
|
||||
@echo "🔍 Running clippy checks..."
|
||||
cargo clippy --all-targets --all-features -- -D warnings
|
||||
|
||||
.PHONY: clippy-fix
|
||||
clippy-fix: core-deps ## Apply clippy fixes
|
||||
@echo "🔧 Applying clippy fixes..."
|
||||
cargo clippy --fix --allow-dirty
|
||||
|
||||
.PHONY: fix
|
||||
fix: fmt clippy-fix ## Format code and apply clippy fixes
|
||||
|
||||
.PHONY: quick-check
|
||||
quick-check: core-deps ## Run fast workspace compilation check
|
||||
@echo "🔨 Running fast compilation check..."
|
||||
cargo check --workspace --exclude e2e_test
|
||||
|
||||
.PHONY: unsafe-code-check
|
||||
unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
|
||||
@echo "🔒 Checking unsafe_code allowances..."
|
||||
./scripts/check_unsafe_code_allowances.sh
|
||||
|
||||
.PHONY: architecture-migration-check
|
||||
architecture-migration-check: ## Check architecture migration guardrails
|
||||
@echo "🏗️ Checking architecture migration guardrails..."
|
||||
./scripts/check_architecture_migration_rules.sh
|
||||
|
||||
.PHONY: logging-guardrails-check
|
||||
logging-guardrails-check: ## Check logging guardrails for redaction and noise regressions
|
||||
@echo "🪵 Checking logging guardrails..."
|
||||
./scripts/check_logging_guardrails.sh
|
||||
|
||||
.PHONY: tokio-io-uring-check
|
||||
tokio-io-uring-check: ## Check tokio io-uring runtime feature stays removed
|
||||
@echo "🚫 Checking tokio io-uring feature guard..."
|
||||
./scripts/check_no_tokio_io_uring.sh
|
||||
|
||||
.PHONY: compilation-check
|
||||
compilation-check: core-deps ## Run compilation check
|
||||
@echo "🔨 Running compilation check..."
|
||||
cargo check --all-targets
|
||||
@@ -1,26 +0,0 @@
|
||||
## —— Pre Commit Checks ----------------------------------------------------------------------------
|
||||
|
||||
.NOTPARALLEL: pre-commit pre-pr dev-check
|
||||
|
||||
.PHONY: setup-hooks
|
||||
setup-hooks: ## Set up git hooks
|
||||
@echo "🔧 Setting up git hooks..."
|
||||
chmod +x .git/hooks/pre-commit
|
||||
@echo "✅ Git hooks setup complete!"
|
||||
|
||||
.PHONY: doc-paths-check
|
||||
doc-paths-check: ## Check that instruction/architecture docs reference existing file paths
|
||||
@echo "📄 Checking doc path references..."
|
||||
./scripts/check_doc_paths.sh
|
||||
|
||||
.PHONY: pre-commit
|
||||
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check doc-paths-check quick-check ## Run fast pre-commit checks without clippy/full tests
|
||||
@echo "✅ All pre-commit checks passed!"
|
||||
|
||||
.PHONY: pre-pr
|
||||
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check doc-paths-check clippy-check test ## Run full pre-PR checks with clippy and tests
|
||||
@echo "✅ All pre-PR checks passed!"
|
||||
|
||||
.PHONY: dev-check
|
||||
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check doc-paths-check quick-check ## Run fast local development checks
|
||||
@echo "✅ Fast development checks passed!"
|
||||
@@ -1,28 +0,0 @@
|
||||
## —— Tests and e2e test ---------------------------------------------------------------------------
|
||||
|
||||
TEST_THREADS ?= 1
|
||||
|
||||
.PHONY: script-tests
|
||||
script-tests: ## Run shell script tests
|
||||
@echo "Running script tests..."
|
||||
./scripts/test_build_rustfs_options.sh
|
||||
./scripts/test_entrypoint_credentials.sh
|
||||
|
||||
.PHONY: test
|
||||
test: core-deps test-deps script-tests ## Run all tests
|
||||
@echo "🧪 Running tests..."
|
||||
@if command -v cargo-nextest >/dev/null 2>&1; then \
|
||||
cargo nextest run --all --exclude e2e_test; \
|
||||
else \
|
||||
echo "ℹ️ cargo-nextest not found; falling back to 'cargo test'"; \
|
||||
cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
|
||||
fi
|
||||
cargo test --all --doc
|
||||
|
||||
.PHONY: e2e-server
|
||||
e2e-server: ## Run e2e-server tests
|
||||
sh $(shell pwd)/scripts/run.sh
|
||||
|
||||
.PHONY: probe-e2e
|
||||
probe-e2e: ## Probe e2e tests
|
||||
sh $(shell pwd)/scripts/probe.sh
|
||||
+229
-99
@@ -1,131 +1,261 @@
|
||||
# RustFS Docker Infrastructure
|
||||
# RustFS Docker Images
|
||||
|
||||
This directory contains the complete Docker infrastructure for building, deploying, and monitoring RustFS. It provides ready-to-use configurations for development, testing, and production-grade observability.
|
||||
This directory contains Docker configuration files and supporting infrastructure for building and running RustFS container images.
|
||||
|
||||
## 📂 Directory Structure
|
||||
## 📁 Directory Structure
|
||||
|
||||
| Directory | Description | Status |
|
||||
| :--- | :--- | :--- |
|
||||
| **[`observability/`](observability/README.md)** | **[RECOMMENDED]** Full-stack observability (Prometheus, Grafana, Tempo, Loki). | ✅ Production-Ready |
|
||||
| **[`compose/`](compose/README.md)** | Specialized setups (e.g., 4-node distributed cluster testing). | ⚠️ Testing Only |
|
||||
| **[`mqtt/`](mqtt/README.md)** | EMQX Broker configuration for MQTT integration testing. | 🧪 Development |
|
||||
| **[`openobserve-otel/`](openobserve-otel/README.md)** | Alternative lightweight observability stack using OpenObserve. | 🔄 Alternative |
|
||||
|
||||
---
|
||||
|
||||
## 📄 Root Directory Files
|
||||
|
||||
The following files in the project root are essential for Docker operations:
|
||||
|
||||
### Build Scripts & Dockerfiles
|
||||
|
||||
| File | Description | Usage |
|
||||
| :--- | :--- | :--- |
|
||||
| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
|
||||
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
|
||||
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
|
||||
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
|
||||
|
||||
### Docker Compose Configurations
|
||||
|
||||
| File | Description | Usage |
|
||||
| :--- | :--- | :--- |
|
||||
| **`docker-compose.yml`** | **Main Development Setup**<br>Comprehensive setup with profiles for development, observability, and proxying. | `docker compose up -d`<br>`docker compose --profile observability up -d` |
|
||||
| **`docker-compose-simple.yml`** | **Quick Start Setup**<br>Minimal configuration running a single RustFS instance with 4 volumes. Perfect for first-time users. | `docker compose -f docker-compose-simple.yml up -d` |
|
||||
|
||||
---
|
||||
|
||||
## 🌟 Observability Stack (Recommended)
|
||||
|
||||
Located in: [`.docker/observability/`](observability/README.md)
|
||||
|
||||
We provide a comprehensive, industry-standard observability stack designed for deep insights into RustFS performance. This is the recommended setup for both development and production monitoring.
|
||||
|
||||
### Components
|
||||
- **Metrics**: Prometheus (Collection) + Grafana (Visualization)
|
||||
- **Traces**: Tempo (Storage) + Jaeger (UI)
|
||||
- **Logs**: Loki
|
||||
- **Ingestion**: OpenTelemetry Collector
|
||||
|
||||
### Key Features
|
||||
- **Full Persistence**: All metrics, logs, and traces are saved to Docker volumes, ensuring no data loss on restarts.
|
||||
- **Correlation**: Seamlessly jump between Logs, Traces, and Metrics in Grafana.
|
||||
- **High Performance**: Optimized configurations for batching, compression, and memory management.
|
||||
|
||||
### Quick Start
|
||||
```bash
|
||||
cd .docker/observability
|
||||
docker compose up -d
|
||||
```
|
||||
rustfs/
|
||||
├── Dockerfile # Production image (Alpine + pre-built binaries)
|
||||
├── Dockerfile.source # Development image (Debian + source build)
|
||||
├── docker-buildx.sh # Multi-architecture build script
|
||||
├── Makefile # Build automation with simplified commands
|
||||
└── .docker/ # Supporting infrastructure
|
||||
├── observability/ # Monitoring and observability configs
|
||||
├── compose/ # Docker Compose configurations
|
||||
├── mqtt/ # MQTT broker configs
|
||||
└── openobserve-otel/ # OpenObserve + OpenTelemetry configs
|
||||
```
|
||||
|
||||
---
|
||||
## 🎯 Image Variants
|
||||
|
||||
## 🧪 Specialized Environments
|
||||
### Core Images
|
||||
|
||||
Located in: [`.docker/compose/`](compose/README.md)
|
||||
| Image | Base OS | Build Method | Size | Use Case |
|
||||
|-------|---------|--------------|------|----------|
|
||||
| `production` (default) | Alpine 3.18 | GitHub Releases | Smallest | Production deployment |
|
||||
| `source` | Debian Bookworm | Source build | Medium | Custom builds with cross-compilation |
|
||||
| `dev` | Debian Bookworm | Development tools | Large | Interactive development |
|
||||
|
||||
These configurations are tailored for specific testing scenarios that require complex topologies.
|
||||
## 🚀 Usage Examples
|
||||
|
||||
### Quick Start (Production)
|
||||
|
||||
### Distributed Cluster (4-Nodes)
|
||||
Simulates a real-world distributed environment with 4 RustFS nodes running locally.
|
||||
```bash
|
||||
docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
|
||||
# Default production image (Alpine + GitHub Releases)
|
||||
docker run -p 9000:9000 rustfs/rustfs:latest
|
||||
|
||||
# Specific version
|
||||
docker run -p 9000:9000 rustfs/rustfs:1.2.3
|
||||
```
|
||||
|
||||
### Integrated Observability Test
|
||||
A self-contained environment running 4 RustFS nodes alongside the full observability stack. Useful for end-to-end testing of telemetry.
|
||||
### Complete Tag Strategy Examples
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/compose/docker-compose.observability.yaml up -d
|
||||
# Stable Releases
|
||||
docker run rustfs/rustfs:1.2.3 # Main version (production)
|
||||
docker run rustfs/rustfs:1.2.3-production # Explicit production variant
|
||||
docker run rustfs/rustfs:1.2.3-source # Source build variant
|
||||
docker run rustfs/rustfs:latest # Latest stable
|
||||
|
||||
# Prerelease Versions
|
||||
docker run rustfs/rustfs:1.3.0-alpha.2 # Specific alpha version
|
||||
docker run rustfs/rustfs:alpha # Latest alpha
|
||||
docker run rustfs/rustfs:beta # Latest beta
|
||||
docker run rustfs/rustfs:rc # Latest release candidate
|
||||
|
||||
# Development Versions
|
||||
docker run rustfs/rustfs:dev # Latest main branch development
|
||||
docker run rustfs/rustfs:dev-13e4a0b # Specific commit
|
||||
docker run rustfs/rustfs:dev-latest # Latest development
|
||||
docker run rustfs/rustfs:main-latest # Main branch latest
|
||||
```
|
||||
|
||||
---
|
||||
### Development Environment
|
||||
|
||||
## 📡 MQTT Integration
|
||||
|
||||
Located in: [`.docker/mqtt/`](mqtt/README.md)
|
||||
|
||||
Provides an EMQX broker for testing RustFS MQTT features.
|
||||
|
||||
### Quick Start
|
||||
```bash
|
||||
cd .docker/mqtt
|
||||
docker compose up -d
|
||||
```
|
||||
- **Dashboard**: [http://localhost:18083](http://localhost:18083) (Default: `admin` / `public`)
|
||||
- **MQTT Port**: `1883`
|
||||
# Quick setup using Makefile (recommended)
|
||||
make docker-dev-local # Build development image locally
|
||||
make dev-env-start # Start development container
|
||||
|
||||
---
|
||||
# Manual Docker commands
|
||||
docker run -it -v $(pwd):/workspace -p 9000:9000 rustfs/rustfs:latest-dev
|
||||
|
||||
## 👁️ Alternative: OpenObserve
|
||||
# Build from source locally
|
||||
docker build -f Dockerfile.source -t rustfs:custom .
|
||||
|
||||
Located in: [`.docker/openobserve-otel/`](openobserve-otel/README.md)
|
||||
|
||||
For users preferring a lightweight, all-in-one solution, we support OpenObserve. It combines logs, metrics, and traces into a single binary and UI.
|
||||
|
||||
### Quick Start
|
||||
```bash
|
||||
cd .docker/openobserve-otel
|
||||
docker compose up -d
|
||||
# Development with hot reload
|
||||
docker-compose up rustfs-dev
|
||||
```
|
||||
|
||||
---
|
||||
## 🏗️ Build Arguments and Scripts
|
||||
|
||||
## 🔧 Common Operations
|
||||
### Using Makefile Commands (Recommended)
|
||||
|
||||
The easiest way to build images using simplified commands:
|
||||
|
||||
### Cleaning Up
|
||||
To stop all containers and remove volumes (**WARNING**: deletes all persisted data):
|
||||
```bash
|
||||
docker compose down -v
|
||||
# Development images (build from source)
|
||||
make docker-dev-local # Build for local use (single arch)
|
||||
make docker-dev # Build multi-arch (for CI/CD)
|
||||
make docker-dev-push REGISTRY=xxx # Build and push to registry
|
||||
|
||||
# Production images (using pre-built binaries)
|
||||
make docker-buildx # Build multi-arch production images
|
||||
make docker-buildx-push # Build and push production images
|
||||
make docker-buildx-version VERSION=v1.0.0 # Build specific version
|
||||
|
||||
# Development environment
|
||||
make dev-env-start # Start development container
|
||||
make dev-env-stop # Stop development container
|
||||
make dev-env-restart # Restart development container
|
||||
|
||||
# Help
|
||||
make help-docker # Show all Docker-related commands
|
||||
```
|
||||
|
||||
### Viewing Logs
|
||||
To follow logs for a specific service:
|
||||
### Using docker-buildx.sh (Advanced)
|
||||
|
||||
For direct script usage and advanced scenarios:
|
||||
|
||||
```bash
|
||||
docker compose logs -f [service_name]
|
||||
# Build latest version for all architectures
|
||||
./docker-buildx.sh
|
||||
|
||||
# Build and push to registry
|
||||
./docker-buildx.sh --push
|
||||
|
||||
# Build specific version
|
||||
./docker-buildx.sh --release v1.2.3
|
||||
|
||||
# Build and push specific version
|
||||
./docker-buildx.sh --release v1.2.3 --push
|
||||
```
|
||||
|
||||
### Checking Status
|
||||
To see the status of all running containers:
|
||||
### Manual Docker Builds
|
||||
|
||||
All images support dynamic version selection:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
# Build production image with latest release
|
||||
docker build --build-arg RELEASE="latest" -t rustfs:latest .
|
||||
|
||||
# Build from source with specific target
|
||||
docker build -f Dockerfile.source \
|
||||
--build-arg TARGETPLATFORM="linux/amd64" \
|
||||
-t rustfs:source .
|
||||
|
||||
# Development build
|
||||
docker build -f Dockerfile.source -t rustfs:dev .
|
||||
```
|
||||
|
||||
## 🔧 Binary Download Sources
|
||||
|
||||
### Unified GitHub Releases
|
||||
|
||||
The production image downloads from GitHub Releases for reliability and transparency:
|
||||
|
||||
- ✅ **production** → GitHub Releases API with automatic latest detection
|
||||
- ✅ **Checksum verification** → SHA256SUMS validation when available
|
||||
- ✅ **Multi-architecture** → Supports amd64 and arm64
|
||||
|
||||
### Source Build
|
||||
|
||||
The source variant compiles from source code with advanced features:
|
||||
|
||||
- 🔧 **Cross-compilation** → Supports multiple target platforms via `TARGETPLATFORM`
|
||||
- ⚡ **Build caching** → sccache for faster compilation
|
||||
- 🎯 **Optimized builds** → Release optimizations with LTO and symbol stripping
|
||||
|
||||
## 📋 Architecture Support
|
||||
|
||||
All variants support multi-architecture builds:
|
||||
|
||||
- **linux/amd64** (x86_64)
|
||||
- **linux/arm64** (aarch64)
|
||||
|
||||
Architecture is automatically detected during build using Docker's `TARGETARCH` build argument.
|
||||
|
||||
## 🔐 Security Features
|
||||
|
||||
- **Checksum Verification**: Production image verifies SHA256SUMS when available
|
||||
- **Non-root User**: All images run as user `rustfs` (UID 1000)
|
||||
- **Minimal Runtime**: Production image only includes necessary dependencies
|
||||
- **Secure Defaults**: No hardcoded credentials or keys
|
||||
|
||||
## 🛠️ Development Workflow
|
||||
|
||||
### Quick Start with Makefile (Recommended)
|
||||
|
||||
```bash
|
||||
# 1. Start development environment
|
||||
make dev-env-start
|
||||
|
||||
# 2. Your development container is now running with:
|
||||
# - Port 9000 exposed for RustFS
|
||||
# - Port 9010 exposed for admin console
|
||||
# - Current directory mounted as /workspace
|
||||
|
||||
# 3. Stop when done
|
||||
make dev-env-stop
|
||||
```
|
||||
|
||||
### Manual Development Setup
|
||||
|
||||
```bash
|
||||
# Build development image from source
|
||||
make docker-dev-local
|
||||
|
||||
# Or use traditional Docker commands
|
||||
docker build -f Dockerfile.source -t rustfs:dev .
|
||||
|
||||
# Run with development tools
|
||||
docker run -it -v $(pwd):/workspace -p 9000:9000 rustfs:dev bash
|
||||
|
||||
# Or use docker-compose for complex setups
|
||||
docker-compose up rustfs-dev
|
||||
```
|
||||
|
||||
### Common Development Tasks
|
||||
|
||||
```bash
|
||||
# Build and test locally
|
||||
make build # Build binary natively
|
||||
make docker-dev-local # Build development Docker image
|
||||
make test # Run tests
|
||||
make fmt # Format code
|
||||
make clippy # Run linter
|
||||
|
||||
# Get help
|
||||
make help # General help
|
||||
make help-docker # Docker-specific help
|
||||
make help-build # Build-specific help
|
||||
```
|
||||
|
||||
## 🚀 CI/CD Integration
|
||||
|
||||
The project uses GitHub Actions for automated multi-architecture Docker builds:
|
||||
|
||||
### Automated Builds
|
||||
|
||||
- **Tags**: Automatic builds triggered on version tags (e.g., `v1.2.3`)
|
||||
- **Main Branch**: Development builds with `dev-latest` and `main-latest` tags
|
||||
- **Pull Requests**: Test builds without registry push
|
||||
|
||||
### Build Variants
|
||||
|
||||
Each build creates three image variants:
|
||||
|
||||
- `rustfs/rustfs:v1.2.3` (production - Alpine-based)
|
||||
- `rustfs/rustfs:v1.2.3-source` (source build - Debian-based)
|
||||
- `rustfs/rustfs:v1.2.3-dev` (development - Debian-based with tools)
|
||||
|
||||
### Manual Builds
|
||||
|
||||
Trigger custom builds via GitHub Actions:
|
||||
|
||||
```bash
|
||||
# Use workflow_dispatch to build specific versions
|
||||
# Available options: latest, main-latest, dev-latest, v1.2.3, dev-abc123
|
||||
```
|
||||
|
||||
## 📦 Supporting Infrastructure
|
||||
|
||||
The `.docker/` directory contains supporting configuration files:
|
||||
|
||||
- **observability/** - Prometheus, Grafana, OpenTelemetry configs
|
||||
- **compose/** - Multi-service Docker Compose setups
|
||||
- **mqtt/** - MQTT broker configurations
|
||||
- **openobserve-otel/** - Log aggregation and tracing setup
|
||||
|
||||
See individual README files in each subdirectory for specific usage instructions.
|
||||
|
||||
+58
-80
@@ -1,102 +1,80 @@
|
||||
# Specialized Docker Compose Configurations
|
||||
# Docker Compose Configurations
|
||||
|
||||
This directory contains specialized Docker Compose configurations for specific testing scenarios.
|
||||
|
||||
## ⚠️ Important Note
|
||||
|
||||
**For Observability:**
|
||||
We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
|
||||
|
||||
The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.
|
||||
This directory contains specialized Docker Compose configurations for different use cases.
|
||||
|
||||
## 📁 Configuration Files
|
||||
|
||||
### Cluster Testing
|
||||
This directory contains specialized Docker Compose configurations and their associated Dockerfiles, keeping related files organized together.
|
||||
|
||||
- **`docker-compose.cluster.yaml`**
|
||||
- **Purpose**: Simulates a 4-node RustFS distributed cluster.
|
||||
- **Use Case**: Testing distributed storage logic, consensus, and failover.
|
||||
- **Nodes**: 4 RustFS instances.
|
||||
- **Storage**: Uses local HTTP endpoints.
|
||||
### Main Configuration (Root Directory)
|
||||
|
||||
### Legacy / Minimal Observability
|
||||
- **`../../docker-compose.yml`** - **Default Production Setup**
|
||||
- Complete production-ready configuration
|
||||
- Includes RustFS server + full observability stack
|
||||
- Supports multiple profiles: `dev`, `observability`, `cache`, `proxy`
|
||||
- Recommended for most users
|
||||
|
||||
- **`docker-compose.observability.yaml`**
|
||||
- **Purpose**: A minimal observability setup.
|
||||
- **Status**: **Deprecated**. Please use `../observability/docker-compose.yml` instead.
|
||||
### Specialized Configurations
|
||||
|
||||
- **`docker-compose.cluster.yaml`** - **Distributed Testing**
|
||||
- 4-node cluster setup for testing distributed storage
|
||||
- Uses local compiled binaries
|
||||
- Simulates multi-node environment
|
||||
- Ideal for development and cluster testing
|
||||
|
||||
- **`docker-compose.observability.yaml`** - **Observability Focus**
|
||||
- Specialized setup for testing observability features
|
||||
- Includes OpenTelemetry, Jaeger, Prometheus, Loki, Grafana
|
||||
- Uses `../../Dockerfile.source` for builds
|
||||
- Perfect for observability development
|
||||
|
||||
## 🚀 Usage Examples
|
||||
|
||||
### Production Setup
|
||||
|
||||
```bash
|
||||
# Start main service
|
||||
docker-compose up -d
|
||||
|
||||
# Start with development profile
|
||||
docker-compose --profile dev up -d
|
||||
|
||||
# Start with full observability
|
||||
docker-compose --profile observability up -d
|
||||
```
|
||||
|
||||
### Cluster Testing
|
||||
|
||||
To start a 4-node cluster for distributed testing:
|
||||
|
||||
```bash
|
||||
# From project root
|
||||
docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
|
||||
# Build and start 4-node cluster (run from project root)
|
||||
cd .docker/compose
|
||||
docker-compose -f docker-compose.cluster.yaml up -d
|
||||
|
||||
# Or run directly from project root
|
||||
docker-compose -f .docker/compose/docker-compose.cluster.yaml up -d
|
||||
```
|
||||
|
||||
### Script-Based 4-Node Validation (Recommended)
|
||||
|
||||
Use the local validation script when you need local-source image build, failover checks,
|
||||
and benchmark workflow in one command:
|
||||
### Observability Testing
|
||||
|
||||
```bash
|
||||
# Default mode: WAIT_PROBE_MODE=service
|
||||
# This avoids false negatives where /health/ready remains 503 locally
|
||||
# while the service path is already available.
|
||||
./scripts/run_four_node_cluster_failover_bench.sh
|
||||
# Start observability-focused environment (run from project root)
|
||||
cd .docker/compose
|
||||
docker-compose -f docker-compose.observability.yaml up -d
|
||||
|
||||
# Or run directly from project root
|
||||
docker-compose -f .docker/compose/docker-compose.observability.yaml up -d
|
||||
```
|
||||
|
||||
Strict mode is available when you explicitly want `/health/ready == 200` as the gate:
|
||||
## 🔧 Configuration Overview
|
||||
|
||||
```bash
|
||||
WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
|
||||
```
|
||||
| Configuration | Nodes | Storage | Observability | Use Case |
|
||||
|---------------|-------|---------|---------------|----------|
|
||||
| **Main** | 1 | Volume mounts | Full stack | Production |
|
||||
| **Cluster** | 4 | HTTP endpoints | Basic | Testing |
|
||||
| **Observability** | 4 | Local data | Advanced | Development |
|
||||
|
||||
### Profiling + Trace Validation
|
||||
## 📝 Notes
|
||||
|
||||
The profiling-focused 4-node compose keeps profiling enabled and points RustFS
|
||||
to an OTLP/HTTP collector endpoint:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
|
||||
```
|
||||
|
||||
Important behavior notes:
|
||||
|
||||
- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
|
||||
traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
|
||||
- Startup usually produces logs and metrics first. That does not guarantee
|
||||
visible traces yet.
|
||||
- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
|
||||
- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
|
||||
many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
|
||||
`RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.
|
||||
|
||||
Minimal trace verification flow:
|
||||
|
||||
```bash
|
||||
# 1. Start the profiling compose with richer span visibility.
|
||||
RUSTFS_OBS_LOGGER_LEVEL=debug \
|
||||
docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
|
||||
|
||||
# 2. Generate real request traffic after startup.
|
||||
curl -I http://127.0.0.1:9000/health
|
||||
curl -I http://127.0.0.1:9000/health/ready
|
||||
|
||||
# 3. Then inspect Tempo or Jaeger.
|
||||
# Grafana: http://localhost:3000
|
||||
# Jaeger: http://localhost:16686
|
||||
```
|
||||
|
||||
If logs and metrics are present but traces are sparse, the most common cause is
|
||||
"no real request traffic yet" or "`info` level filtered nested spans", not an
|
||||
OTLP routing failure.
|
||||
|
||||
### (Deprecated) Minimal Observability
|
||||
|
||||
```bash
|
||||
# From project root
|
||||
docker compose -f .docker/compose/docker-compose.observability.yaml up -d
|
||||
```
|
||||
- Always ensure you have built the required binaries before starting cluster tests
|
||||
- The main configuration is sufficient for most use cases
|
||||
- Specialized configurations are for specific testing scenarios
|
||||
|
||||
@@ -1,236 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Profiling-first 4-node local-build compose.
|
||||
#
|
||||
# Goals:
|
||||
# - force linux/amd64 runtime/build on Apple Silicon hosts;
|
||||
# - enable RustFS built-in CPU profiling;
|
||||
# - keep all tuning knobs host-overridable via env.
|
||||
#
|
||||
# Observability notes:
|
||||
# - `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS appends
|
||||
# `/v1/traces`, `/v1/metrics`, and `/v1/logs` automatically.
|
||||
# - Logs and metrics usually appear during startup. Traces mainly appear after
|
||||
# real HTTP/S3/gRPC requests create spans.
|
||||
# - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request trace span but
|
||||
# filters many `debug`-level nested spans. Use `debug` when validating trace
|
||||
# richness rather than collector reachability.
|
||||
|
||||
services:
|
||||
node1:
|
||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
|
||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile.source
|
||||
hostname: node1
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
|
||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
||||
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
|
||||
# should show richer nested spans during request-path verification.
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
||||
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
|
||||
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
|
||||
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
|
||||
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
|
||||
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
|
||||
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
volumes:
|
||||
- node1_data_0:/data/rustfs0
|
||||
- node1_data_1:/data/rustfs1
|
||||
- node1_data_2:/data/rustfs2
|
||||
- node1_data_3:/data/rustfs3
|
||||
ports:
|
||||
- "9000:9000"
|
||||
networks:
|
||||
- rustfs-cluster-net
|
||||
|
||||
node2:
|
||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
|
||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile.source
|
||||
hostname: node2
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
|
||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
||||
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
|
||||
# should show richer nested spans during request-path verification.
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
||||
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
|
||||
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
|
||||
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
|
||||
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
|
||||
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
|
||||
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
volumes:
|
||||
- node2_data_0:/data/rustfs0
|
||||
- node2_data_1:/data/rustfs1
|
||||
- node2_data_2:/data/rustfs2
|
||||
- node2_data_3:/data/rustfs3
|
||||
ports:
|
||||
- "9001:9000"
|
||||
networks:
|
||||
- rustfs-cluster-net
|
||||
|
||||
node3:
|
||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
|
||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile.source
|
||||
hostname: node3
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
|
||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
||||
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
|
||||
# should show richer nested spans during request-path verification.
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
||||
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
|
||||
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
|
||||
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
|
||||
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
|
||||
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
|
||||
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
volumes:
|
||||
- node3_data_0:/data/rustfs0
|
||||
- node3_data_1:/data/rustfs1
|
||||
- node3_data_2:/data/rustfs2
|
||||
- node3_data_3:/data/rustfs3
|
||||
ports:
|
||||
- "9002:9000"
|
||||
networks:
|
||||
- rustfs-cluster-net
|
||||
|
||||
node4:
|
||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
|
||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile.source
|
||||
hostname: node4
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
|
||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
||||
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
|
||||
# should show richer nested spans during request-path verification.
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
||||
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
|
||||
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
|
||||
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
|
||||
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
|
||||
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
|
||||
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
volumes:
|
||||
- node4_data_0:/data/rustfs0
|
||||
- node4_data_1:/data/rustfs1
|
||||
- node4_data_2:/data/rustfs2
|
||||
- node4_data_3:/data/rustfs3
|
||||
ports:
|
||||
- "9003:9000"
|
||||
networks:
|
||||
- rustfs-cluster-net
|
||||
|
||||
volumes:
|
||||
node1_data_0:
|
||||
node1_data_1:
|
||||
node1_data_2:
|
||||
node1_data_3:
|
||||
node2_data_0:
|
||||
node2_data_1:
|
||||
node2_data_2:
|
||||
node2_data_3:
|
||||
node3_data_0:
|
||||
node3_data_1:
|
||||
node3_data_2:
|
||||
node3_data_3:
|
||||
node4_data_0:
|
||||
node4_data_1:
|
||||
node4_data_2:
|
||||
node4_data_3:
|
||||
|
||||
networks:
|
||||
rustfs-cluster-net:
|
||||
driver: bridge
|
||||
@@ -1,220 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
services:
|
||||
node1:
|
||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile.source
|
||||
hostname: node1
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
||||
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
|
||||
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
|
||||
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
|
||||
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
|
||||
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
|
||||
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
|
||||
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
|
||||
- RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
|
||||
- RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
|
||||
- RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
|
||||
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
|
||||
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
|
||||
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
|
||||
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
|
||||
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
|
||||
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
|
||||
- RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
|
||||
- RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-0}
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
volumes:
|
||||
- node1_data_0:/data/rustfs0
|
||||
- node1_data_1:/data/rustfs1
|
||||
- node1_data_2:/data/rustfs2
|
||||
- node1_data_3:/data/rustfs3
|
||||
ports:
|
||||
- "9000:9000"
|
||||
networks:
|
||||
- rustfs-cluster-net
|
||||
|
||||
node2:
|
||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile.source
|
||||
hostname: node2
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
||||
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
|
||||
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
|
||||
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
|
||||
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
|
||||
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
|
||||
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
|
||||
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
|
||||
- RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
|
||||
- RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
|
||||
- RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
|
||||
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
|
||||
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
|
||||
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
|
||||
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
|
||||
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
|
||||
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
|
||||
- RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
|
||||
- RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-0}
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
volumes:
|
||||
- node2_data_0:/data/rustfs0
|
||||
- node2_data_1:/data/rustfs1
|
||||
- node2_data_2:/data/rustfs2
|
||||
- node2_data_3:/data/rustfs3
|
||||
ports:
|
||||
- "9001:9000"
|
||||
networks:
|
||||
- rustfs-cluster-net
|
||||
|
||||
node3:
|
||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile.source
|
||||
hostname: node3
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
||||
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
|
||||
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
|
||||
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
|
||||
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
|
||||
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
|
||||
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
|
||||
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
|
||||
- RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
|
||||
- RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
|
||||
- RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
|
||||
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
|
||||
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
|
||||
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
|
||||
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
|
||||
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
|
||||
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
|
||||
- RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
|
||||
- RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-0}
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
volumes:
|
||||
- node3_data_0:/data/rustfs0
|
||||
- node3_data_1:/data/rustfs1
|
||||
- node3_data_2:/data/rustfs2
|
||||
- node3_data_3:/data/rustfs3
|
||||
ports:
|
||||
- "9002:9000"
|
||||
networks:
|
||||
- rustfs-cluster-net
|
||||
|
||||
node4:
|
||||
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile.source
|
||||
hostname: node4
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
|
||||
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
|
||||
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
|
||||
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
|
||||
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
|
||||
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
|
||||
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
|
||||
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
|
||||
- RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
|
||||
- RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
|
||||
- RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
|
||||
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
|
||||
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
|
||||
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
|
||||
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
|
||||
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
|
||||
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
|
||||
- RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
|
||||
- RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-0}
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
volumes:
|
||||
- node4_data_0:/data/rustfs0
|
||||
- node4_data_1:/data/rustfs1
|
||||
- node4_data_2:/data/rustfs2
|
||||
- node4_data_3:/data/rustfs3
|
||||
ports:
|
||||
- "9003:9000"
|
||||
networks:
|
||||
- rustfs-cluster-net
|
||||
|
||||
volumes:
|
||||
node1_data_0:
|
||||
node1_data_1:
|
||||
node1_data_2:
|
||||
node1_data_3:
|
||||
node2_data_0:
|
||||
node2_data_1:
|
||||
node2_data_2:
|
||||
node2_data_3:
|
||||
node3_data_0:
|
||||
node3_data_1:
|
||||
node3_data_2:
|
||||
node3_data_3:
|
||||
node4_data_0:
|
||||
node4_data_1:
|
||||
node4_data_2:
|
||||
node4_data_3:
|
||||
|
||||
networks:
|
||||
rustfs-cluster-net:
|
||||
driver: bridge
|
||||
-56
@@ -1,56 +0,0 @@
|
||||
services:
|
||||
node1:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=16
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
|
||||
|
||||
node2:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=16
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
|
||||
|
||||
node3:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=16
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
|
||||
|
||||
node4:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=16
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
|
||||
-56
@@ -1,56 +0,0 @@
|
||||
services:
|
||||
node1:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=24
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
|
||||
|
||||
node2:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=24
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
|
||||
|
||||
node3:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=24
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
|
||||
|
||||
node4:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=24
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
|
||||
@@ -1,56 +0,0 @@
|
||||
services:
|
||||
node1:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=20
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
|
||||
|
||||
node2:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=20
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
|
||||
|
||||
node3:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=20
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
|
||||
|
||||
node4:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=20
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
|
||||
@@ -1,56 +0,0 @@
|
||||
services:
|
||||
node1:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=12
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
|
||||
|
||||
node2:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=12
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
|
||||
|
||||
node3:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=12
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
|
||||
|
||||
node4:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=12
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
|
||||
@@ -1,56 +0,0 @@
|
||||
services:
|
||||
node1:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=6
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
|
||||
|
||||
node2:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=6
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
|
||||
|
||||
node3:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=6
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
|
||||
|
||||
node4:
|
||||
environment:
|
||||
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
|
||||
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
|
||||
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
|
||||
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
|
||||
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
|
||||
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
|
||||
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
|
||||
- RUSTFS_RUNTIME_WORKER_THREADS=6
|
||||
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
|
||||
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
|
||||
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
|
||||
@@ -13,183 +13,62 @@
|
||||
# limitations under the License.
|
||||
|
||||
services:
|
||||
# --- Observability Stack ---
|
||||
|
||||
tempo-init:
|
||||
image: busybox:latest
|
||||
command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
|
||||
volumes:
|
||||
- tempo-data:/var/tempo
|
||||
user: root
|
||||
networks:
|
||||
- rustfs-network
|
||||
restart: "no"
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:2.10.5
|
||||
user: "10001"
|
||||
command: [ "-config.file=/etc/tempo.yaml" ]
|
||||
volumes:
|
||||
- ../../.docker/observability/tempo.yaml:/etc/tempo.yaml:ro
|
||||
- tempo-data:/var/tempo
|
||||
ports:
|
||||
- "3200:3200" # tempo
|
||||
- "4317" # otlp grpc
|
||||
- "4318" # otlp http
|
||||
- "7946" # memberlist
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- rustfs-network
|
||||
depends_on:
|
||||
tempo-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: [ "CMD", "/tempo", "-version" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:latest
|
||||
image: otel/opentelemetry-collector-contrib:0.129.1
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
volumes:
|
||||
- ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
|
||||
- ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
|
||||
ports:
|
||||
- "1888:1888" # pprof
|
||||
- "8888:8888" # Prometheus metrics for Collector
|
||||
- "8889:8889" # Prometheus metrics for application indicators
|
||||
- "13133:13133" # health check
|
||||
- "4317:4317" # OTLP gRPC
|
||||
- "4318:4318" # OTLP HTTP
|
||||
- "55679:55679" # zpages
|
||||
- 1888:1888
|
||||
- 8888:8888
|
||||
- 8889:8889
|
||||
- 13133:13133
|
||||
- 4317:4317
|
||||
- 4318:4318
|
||||
- 55679:55679
|
||||
networks:
|
||||
- rustfs-network
|
||||
depends_on:
|
||||
- tempo
|
||||
- jaeger
|
||||
- prometheus
|
||||
- loki
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "/otelcol-contrib", "--version" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
jaeger:
|
||||
image: jaegertracing/jaeger:latest
|
||||
image: jaegertracing/jaeger:2.8.0
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
- SPAN_STORAGE_TYPE=badger
|
||||
- BADGER_EPHEMERAL=false
|
||||
- BADGER_DIRECTORY_VALUE=/badger/data
|
||||
- BADGER_DIRECTORY_KEY=/badger/key
|
||||
- COLLECTOR_OTLP_ENABLED=true
|
||||
volumes:
|
||||
- ../../.docker/observability/jaeger.yaml:/etc/jaeger/config.yml:ro
|
||||
- jaeger-data:/badger
|
||||
ports:
|
||||
- "16686:16686" # Web UI
|
||||
- "14269:14269" # Admin/Metrics
|
||||
- "4317" # otlp grpc
|
||||
- "4318" # otlp http
|
||||
command: [ "--config", "/etc/jaeger/config.yml" ]
|
||||
- "16686:16686"
|
||||
- "14317:4317"
|
||||
- "14318:4318"
|
||||
networks:
|
||||
- rustfs-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:latest
|
||||
image: prom/prometheus:v3.4.2
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
volumes:
|
||||
- ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- ../../.docker/observability/prometheus-rules:/etc/prometheus/rules:ro
|
||||
- prometheus-data:/prometheus
|
||||
- ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml
|
||||
ports:
|
||||
- "9090:9090"
|
||||
command:
|
||||
- '--config.file=/etc/prometheus/prometheus.yml'
|
||||
- '--web.enable-otlp-receiver'
|
||||
- '--web.enable-remote-write-receiver'
|
||||
- '--enable-feature=promql-experimental-functions'
|
||||
- '--storage.tsdb.path=/prometheus'
|
||||
- '--storage.tsdb.retention.time=30d'
|
||||
networks:
|
||||
- rustfs-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
loki:
|
||||
image: grafana/loki:latest
|
||||
image: grafana/loki:3.5.1
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
volumes:
|
||||
- ../../.docker/observability/loki.yaml:/etc/loki/loki.yaml:ro
|
||||
- loki-data:/loki
|
||||
- ../../.docker/observability/loki-config.yaml:/etc/loki/local-config.yaml
|
||||
ports:
|
||||
- "3100:3100"
|
||||
command: -config.file=/etc/loki/loki.yaml
|
||||
command: -config.file=/etc/loki/local-config.yaml
|
||||
networks:
|
||||
- rustfs-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
|
||||
interval: 15s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 60s
|
||||
|
||||
pyroscope:
|
||||
image: grafana/pyroscope:latest
|
||||
ports:
|
||||
- "4040:4040"
|
||||
command:
|
||||
- -self-profiling.disable-push=true
|
||||
networks:
|
||||
- rustfs-network
|
||||
restart: unless-stopped
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
image: grafana/grafana:12.0.2
|
||||
ports:
|
||||
- "3000:3000" # Web UI
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_PASSWORD=admin
|
||||
- GF_SECURITY_ADMIN_USER=admin
|
||||
- TZ=Asia/Shanghai
|
||||
- GF_INSTALL_PLUGINS=grafana-pyroscope-datasource
|
||||
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
|
||||
networks:
|
||||
- rustfs-network
|
||||
volumes:
|
||||
- ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
- grafana-data:/var/lib/grafana
|
||||
depends_on:
|
||||
- prometheus
|
||||
- tempo
|
||||
- loki
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
# --- RustFS Cluster ---
|
||||
|
||||
node1:
|
||||
build:
|
||||
@@ -200,15 +79,13 @@ services:
|
||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
||||
platform: linux/amd64
|
||||
ports:
|
||||
- "9001:9000"
|
||||
- "9001:9000" # Map port 9001 of the host to port 9000 of the container
|
||||
networks:
|
||||
- rustfs-network
|
||||
depends_on:
|
||||
- otel-collector
|
||||
|
||||
node2:
|
||||
build:
|
||||
@@ -219,15 +96,13 @@ services:
|
||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
||||
platform: linux/amd64
|
||||
ports:
|
||||
- "9002:9000"
|
||||
- "9002:9000" # Map port 9002 of the host to port 9000 of the container
|
||||
networks:
|
||||
- rustfs-network
|
||||
depends_on:
|
||||
- otel-collector
|
||||
|
||||
node3:
|
||||
build:
|
||||
@@ -238,15 +113,13 @@ services:
|
||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
||||
platform: linux/amd64
|
||||
ports:
|
||||
- "9003:9000"
|
||||
- "9003:9000" # Map port 9003 of the host to port 9000 of the container
|
||||
networks:
|
||||
- rustfs-network
|
||||
depends_on:
|
||||
- otel-collector
|
||||
|
||||
node4:
|
||||
build:
|
||||
@@ -257,22 +130,13 @@ services:
|
||||
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
|
||||
- RUSTFS_ADDRESS=:9000
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=debug
|
||||
platform: linux/amd64
|
||||
ports:
|
||||
- "9004:9000"
|
||||
- "9004:9000" # Map port 9004 of the host to port 9000 of the container
|
||||
networks:
|
||||
- rustfs-network
|
||||
depends_on:
|
||||
- otel-collector
|
||||
|
||||
volumes:
|
||||
prometheus-data:
|
||||
tempo-data:
|
||||
loki-data:
|
||||
jaeger-data:
|
||||
grafana-data:
|
||||
|
||||
networks:
|
||||
rustfs-network:
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
# MQTT Broker (EMQX)
|
||||
|
||||
This directory contains the configuration for running an EMQX MQTT broker, which can be used for testing RustFS's MQTT integration.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
To start the EMQX broker:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## 📊 Access
|
||||
|
||||
- **Dashboard**: [http://localhost:18083](http://localhost:18083)
|
||||
- **Default Credentials**: `admin` / `public`
|
||||
- **MQTT Port**: `1883`
|
||||
- **WebSocket Port**: `8083`
|
||||
|
||||
## 🛠️ Configuration
|
||||
|
||||
The `docker-compose.yml` file sets up a single-node EMQX instance.
|
||||
|
||||
- **Persistence**: Data is not persisted by default (for testing).
|
||||
- **Network**: Uses the default bridge network.
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
- This setup is intended for development and testing purposes.
|
||||
- For production deployments, please refer to the official [EMQX Documentation](https://www.emqx.io/docs/en/latest/).
|
||||
@@ -1,93 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
worker_processes auto;
|
||||
pid /var/run/nginx.pid;
|
||||
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
|
||||
'$status $body_bytes_sent "$http_referer" '
|
||||
'"$http_user_agent" "$http_x_forwarded_for"';
|
||||
|
||||
access_log /var/log/nginx/access.log main;
|
||||
error_log /var/log/nginx/error.log warn;
|
||||
|
||||
sendfile on;
|
||||
keepalive_timeout 65;
|
||||
|
||||
# RustFS Server Block
|
||||
server {
|
||||
listen 80;
|
||||
server_name localhost;
|
||||
|
||||
# Redirect HTTP to HTTPS (optional, uncomment if SSL is configured)
|
||||
# return 301 https://$host$request_uri;
|
||||
|
||||
location / {
|
||||
proxy_pass http://rustfs:9000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# S3 specific headers
|
||||
proxy_set_header X-Amz-Date $http_x_amz_date;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
|
||||
# Disable buffering for large uploads
|
||||
proxy_request_buffering off;
|
||||
client_max_body_size 0;
|
||||
}
|
||||
|
||||
location /rustfs/console {
|
||||
proxy_pass http://rustfs:9001;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
}
|
||||
|
||||
# SSL Configuration (Example)
|
||||
# server {
|
||||
# listen 443 ssl;
|
||||
# server_name localhost;
|
||||
#
|
||||
# ssl_certificate /etc/nginx/ssl/server.crt;
|
||||
# ssl_certificate_key /etc/nginx/ssl/server.key;
|
||||
#
|
||||
# # Restrict to modern TLS versions and ciphers. Operators copying this
|
||||
# # example must keep at least these directives — without them, nginx
|
||||
# # may negotiate older protocol versions that have known weaknesses.
|
||||
# ssl_protocols TLSv1.2 TLSv1.3;
|
||||
# ssl_prefer_server_ciphers on;
|
||||
# ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
|
||||
# ssl_session_timeout 1d;
|
||||
# ssl_session_cache shared:SSL:10m;
|
||||
# ssl_session_tickets off;
|
||||
# # add_header Strict-Transport-Security "max-age=63072000" always;
|
||||
#
|
||||
# location / {
|
||||
# proxy_pass http://rustfs:9000;
|
||||
# ...
|
||||
# }
|
||||
# }
|
||||
}
|
||||
@@ -1,5 +0,0 @@
|
||||
jaeger-data/*
|
||||
loki-data/*
|
||||
prometheus-data/*
|
||||
tempo-data/*
|
||||
grafana-data/*
|
||||
+83
-169
@@ -1,195 +1,109 @@
|
||||
# RustFS Observability Stack
|
||||
# Observability
|
||||
|
||||
This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.
|
||||
This directory contains the observability stack for the application. The stack is composed of the following components:
|
||||
|
||||
## Components
|
||||
- Prometheus v3.2.1
|
||||
- Grafana 11.6.0
|
||||
- Loki 3.4.2
|
||||
- Jaeger 2.4.0
|
||||
- Otel Collector 0.120.0 # 0.121.0 remove loki
|
||||
|
||||
The stack is composed of the following best-in-class open-source components:
|
||||
## Prometheus
|
||||
|
||||
- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
|
||||
- **Grafana** (v11.1.0): The leading platform for observability visualization.
|
||||
- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
|
||||
- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
|
||||
- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
|
||||
- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
|
||||
Prometheus is a monitoring and alerting toolkit. It scrapes metrics from instrumented jobs, either directly or via an
|
||||
intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to
|
||||
either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be
|
||||
used to visualize the collected data.
|
||||
|
||||
By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
|
||||
If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
|
||||
## Grafana
|
||||
|
||||
## Architecture
|
||||
Grafana is a multi-platform open-source analytics and interactive visualization web application. It provides charts,
|
||||
graphs, and alerts for the web when connected to supported data sources.
|
||||
|
||||
1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
|
||||
2. **Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
|
||||
- **Traces** -> **Tempo** (Primary) & **Jaeger** (Secondary/Optional)
|
||||
- **Metrics** -> **Prometheus** (via scraping the Collector's exporter)
|
||||
- **Logs** -> **Loki**
|
||||
3. **Visualization**: **Grafana** connects to all backends (Prometheus, Tempo, Loki, Jaeger) to provide a unified dashboard experience.
|
||||
## Loki
|
||||
|
||||
## Features
|
||||
Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is
|
||||
designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of
|
||||
labels for each log stream.
|
||||
|
||||
- **Full Persistence**: All data (Metrics, Logs, Traces) is persisted to Docker volumes, ensuring no data loss on restart.
|
||||
- **Correlation**: Seamless navigation between Metrics, Logs, and Traces in Grafana.
|
||||
- Jump from a Metric spike to relevant Traces.
|
||||
- Jump from a Trace to relevant Logs.
|
||||
- **High Performance**: Optimized configurations for batching, compression, and memory management.
|
||||
- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
|
||||
## Jaeger
|
||||
|
||||
## GET Performance Optimization Dashboards
|
||||
Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and
|
||||
troubleshooting microservices-based distributed systems, including:
|
||||
|
||||
Three pre-built Grafana dashboards are included for monitoring RustFS GET performance optimization rollout:
|
||||
- Distributed context propagation
|
||||
- Distributed transaction monitoring
|
||||
- Root cause analysis
|
||||
- Service dependency analysis
|
||||
- Performance / latency optimization
|
||||
|
||||
### Available Dashboards
|
||||
## Otel Collector
|
||||
|
||||
| Dashboard | File | Description |
|
||||
|-----------|------|-------------|
|
||||
| **GET Rollout Health** | `grafana-get-rollout-health.json` | Monitors optimization rollout: latency by reader path, early-stop hit rate, codec streaming usage, pipeline failures |
|
||||
| **GET Data Integrity** | `grafana-get-data-integrity.json` | Monitors data safety: bitrot verify failures, decode errors, short reads, shard read outcomes |
|
||||
| **GET Resource Impact** | `grafana-get-resource-impact.json` | Monitors resource usage: concurrent requests, IO queue utilization, disk permit wait, RSS trend |
|
||||
The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry
|
||||
data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source
|
||||
observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends.
|
||||
|
||||
### Prometheus Alert Rules
|
||||
## How to use
|
||||
|
||||
The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-configured alerting rules:
|
||||
To deploy the observability stack, run the following command:
|
||||
|
||||
| Alert | Severity | Condition |
|
||||
|-------|----------|-----------|
|
||||
| `GetP99Regression` | Critical | GET p99 latency > 2x baseline for 10m |
|
||||
| `PipelineFailureSpike` | Critical | Pipeline failure rate > 5x baseline for 5m |
|
||||
| `BitrotMismatchSpike` | Critical | Bitrot mismatch rate > 3x baseline for 5m |
|
||||
| `EarlyStopInsufficientQuorum` | Warning | Early-stop insufficient quorum rate > 0.1/s for 5m |
|
||||
| `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
|
||||
| `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
|
||||
- docker latest version
|
||||
|
||||
### Enabling Alert Rules
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.override.yml up -d
|
||||
```
|
||||
|
||||
Add the alert rules file to your Prometheus configuration:
|
||||
- docker compose v2.0.0 or before
|
||||
|
||||
```bash
|
||||
docke-compose -f docker-compose.yml -f docker-compose.override.yml up -d
|
||||
```
|
||||
|
||||
To access the Grafana dashboard, navigate to `http://localhost:3000` in your browser. The default username and password
|
||||
are `admin` and `admin`, respectively.
|
||||
|
||||
To access the Jaeger dashboard, navigate to `http://localhost:16686` in your browser.
|
||||
|
||||
To access the Prometheus dashboard, navigate to `http://localhost:9090` in your browser.
|
||||
|
||||
## How to stop
|
||||
|
||||
To stop the observability stack, run the following command:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.override.yml down
|
||||
```
|
||||
|
||||
## How to remove data
|
||||
|
||||
To remove the data generated by the observability stack, run the following command:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.override.yml down -v
|
||||
```
|
||||
|
||||
## How to configure
|
||||
|
||||
To configure the observability stack, modify the `docker-compose.override.yml` file. The file contains the following
|
||||
|
||||
```yaml
|
||||
# prometheus.yml
|
||||
rule_files:
|
||||
- "/etc/prometheus/rules/*.yml"
|
||||
services:
|
||||
prometheus:
|
||||
environment:
|
||||
- PROMETHEUS_CONFIG_FILE=/etc/prometheus/prometheus.yml
|
||||
volumes:
|
||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml
|
||||
|
||||
# Or mount the file in docker-compose.yml:
|
||||
# volumes:
|
||||
# - ./prometheus-rules:/etc/prometheus/rules
|
||||
grafana:
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_PASSWORD=admin
|
||||
volumes:
|
||||
- ./grafana/provisioning:/etc/grafana/provisioning
|
||||
```
|
||||
|
||||
### Dashboard Usage
|
||||
The `prometheus` service mounts the `prometheus.yml` file to `/etc/prometheus/prometheus.yml`. The `grafana` service
|
||||
mounts the `grafana/provisioning` directory to `/etc/grafana/provisioning`. You can modify these files to configure the
|
||||
observability stack.
|
||||
|
||||
The dashboards are automatically provisioned when Grafana starts. They use the `${DS_PROMETHEUS}` datasource variable, so you need a Prometheus datasource configured in Grafana.
|
||||
|
||||
Key panels to monitor during optimization rollout:
|
||||
|
||||
1. **GET Latency by Reader Path** - Compare `codec_streaming` vs `legacy_duplex` latency
|
||||
2. **Early-Stop Hit Rate** - Verify early-stop is triggering effectively
|
||||
3. **Pipeline Failure Rate** - Detect any new failure modes introduced by optimizations
|
||||
4. **Bitrot Verify Failures** - Ensure data integrity is maintained
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Docker
|
||||
- Docker Compose
|
||||
|
||||
### Deploy
|
||||
|
||||
Run the following command to start the entire stack:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### High Availability Tempo
|
||||
|
||||
The default `docker-compose.yml` is the single-node stack.
|
||||
If you need the Kafka-backed HA Tempo configuration, start it with:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
|
||||
```
|
||||
|
||||
### Access Dashboards
|
||||
|
||||
| Service | URL | Credentials | Description |
|
||||
| :------------- | :----------------------------------------------- | :---------------- | :----------------------------- |
|
||||
| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | Main visualization hub. |
|
||||
| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | Metric queries and status. |
|
||||
| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | Secondary trace visualization. |
|
||||
| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo status/metrics. |
|
||||
|
||||
## Configuration
|
||||
|
||||
### Data Persistence
|
||||
|
||||
Data is stored in the following Docker volumes:
|
||||
|
||||
- `prometheus-data`: Prometheus metrics
|
||||
- `tempo-data`: Tempo traces (WAL and Blocks)
|
||||
- `loki-data`: Loki logs (Chunks and Rules)
|
||||
- `jaeger-data`: Jaeger traces (Badger DB)
|
||||
|
||||
To clear all data:
|
||||
|
||||
```bash
|
||||
docker compose down -v
|
||||
```
|
||||
|
||||
### Customization
|
||||
|
||||
- **Prometheus**: Edit `prometheus.yml` to add scrape targets or alerting rules.
|
||||
- **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
|
||||
- **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
|
||||
|
||||
### Verifying RustFS Traces
|
||||
|
||||
When RustFS points `RUSTFS_OBS_ENDPOINT` at this stack, treat the value as the
|
||||
OTLP/HTTP base URL, for example:
|
||||
|
||||
```bash
|
||||
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
|
||||
```
|
||||
|
||||
RustFS automatically expands that base URL to:
|
||||
|
||||
- `/v1/traces`
|
||||
- `/v1/metrics`
|
||||
- `/v1/logs`
|
||||
|
||||
Important behavior notes:
|
||||
|
||||
- Logs and metrics usually appear during startup, so seeing those two signals
|
||||
first is expected.
|
||||
- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
|
||||
startup, because request-path spans are created on demand.
|
||||
- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
|
||||
many nested `debug` spans. If Tempo or Jaeger looks sparse, retry with
|
||||
`RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting collector or Tempo issues.
|
||||
|
||||
Minimal validation flow:
|
||||
|
||||
```bash
|
||||
# 1. Start this observability stack.
|
||||
docker compose up -d
|
||||
|
||||
# 2. Start RustFS with OTLP/HTTP export and richer span visibility.
|
||||
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
|
||||
export RUSTFS_OBS_LOGGER_LEVEL=debug
|
||||
|
||||
# 3. Generate real request traffic.
|
||||
curl -I http://127.0.0.1:9000/health
|
||||
curl -I http://127.0.0.1:9000/health/ready
|
||||
|
||||
# 4. Inspect Grafana or Jaeger.
|
||||
# Grafana: http://localhost:3000
|
||||
# Jaeger: http://localhost:16686
|
||||
```
|
||||
|
||||
If logs and metrics are present but traces are sparse, the most common cause is
|
||||
"no real request traffic yet" or "`info` level filtered nested spans", not an
|
||||
OTLP routing failure.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Service Health**: Check the health of services using `docker compose ps`.
|
||||
- **Logs**: View logs for a specific service using `docker compose logs -f <service_name>`.
|
||||
- **Otel Collector**: Check `http://localhost:13133` for health status and `http://localhost:1888/debug/pprof/` for profiling.
|
||||
|
||||
@@ -1,191 +1,27 @@
|
||||
# RustFS 可观测性技术栈
|
||||
## 部署可观测性系统
|
||||
|
||||
本目录包含 RustFS 的全面可观测性技术栈，旨在提供对应用程序性能、日志和追踪的深入洞察。
|
||||
OpenTelemetry Collector 提供了一个厂商中立的遥测数据处理方案，用于接收、处理和导出遥测数据。它消除了为支持多种开源可观测性数据格式（如
|
||||
Jaeger、Prometheus 等）而需要运行和维护多个代理/收集器的必要性。
|
||||
|
||||
## 组件
|
||||
### 快速部署
|
||||
|
||||
该技术栈由以下一流的开源组件组成：
|
||||
|
||||
- **Prometheus** (v2.53.1): 行业标准的指标收集和告警工具。
|
||||
- **Grafana** (v11.1.0): 领先的可观测性可视化平台。
|
||||
- **Loki** (v3.1.0): 水平可扩展、高可用、多租户的日志聚合系统。
|
||||
- **Tempo** (v2.5.0): 高吞吐量、最小依赖的分布式追踪后端。
|
||||
- **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
|
||||
- **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
|
||||
|
||||
默认情况下，这套技术栈使用 Tempo 单二进制模式，不依赖 Kafka/Redpanda。
|
||||
如果需要基于 Kafka 的 HA Tempo 路径，请使用 `docker-compose-example-for-rustfs.yml` 配合 `docker-compose-tempo-ha-override.yml`。
|
||||
|
||||
## 架构
|
||||
|
||||
1. **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
|
||||
2. **处理与导出**: Collector 处理数据（批处理、内存限制）并将其导出到相应的后端：
|
||||
- **追踪** -> **Tempo** (主要) & **Jaeger** (辅助/可选)
|
||||
- **指标** -> **Prometheus** (通过抓取 Collector 的导出器)
|
||||
- **日志** -> **Loki**
|
||||
3. **可视化**: **Grafana** 连接到所有后端（Prometheus, Tempo, Loki, Jaeger），提供统一的仪表盘体验。
|
||||
|
||||
## 特性
|
||||
|
||||
- **完全持久化**: 所有数据（指标、日志、追踪）都持久化到 Docker 卷，确保重启后无数据丢失。
|
||||
- **关联性**: 在 Grafana 中实现指标、日志和追踪之间的无缝导航。
|
||||
- 从指标峰值跳转到相关追踪。
|
||||
- 从追踪跳转到相关日志。
|
||||
- **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
|
||||
- **标准化协议**: 完全基于 OpenTelemetry 标准构建。
|
||||
|
||||
## GET 性能优化仪表盘
|
||||
|
||||
包含三个预构建的 Grafana 仪表盘，用于监控 RustFS GET 性能优化发布：
|
||||
|
||||
### 可用仪表盘
|
||||
|
||||
| 仪表盘 | 文件 | 描述 |
|
||||
|--------|------|------|
|
||||
| **GET 发布健康度** | `grafana-get-rollout-health.json` | 监控优化发布：按 reader path 的延迟、early-stop 命中率、codec streaming 使用率、pipeline 失败率 |
|
||||
| **GET 数据完整性** | `grafana-get-data-integrity.json` | 监控数据安全：bitrot 校验失败、decode 错误、short read、shard 读取结果 |
|
||||
| **GET 资源影响** | `grafana-get-resource-impact.json` | 监控资源使用：并发请求数、IO 队列利用率、disk permit 等待、RSS 趋势 |
|
||||
|
||||
### Prometheus 告警规则
|
||||
|
||||
文件 `prometheus-rules/rustfs-get-optimization-alerts.yaml` 包含预配置的告警规则：
|
||||
|
||||
| 告警 | 级别 | 条件 |
|
||||
|------|------|------|
|
||||
| `GetP99Regression` | 严重 | GET p99 延迟 > 2x 基线，持续 10 分钟 |
|
||||
| `PipelineFailureSpike` | 严重 | Pipeline 失败率 > 5x 基线，持续 5 分钟 |
|
||||
| `BitrotMismatchSpike` | 严重 | Bitrot 不匹配率 > 3x 基线，持续 5 分钟 |
|
||||
| `EarlyStopInsufficientQuorum` | 警告 | Early-stop quorum 不足率 > 0.1/s，持续 5 分钟 |
|
||||
| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线，持续 10 分钟 |
|
||||
| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%，持续 5 分钟 |
|
||||
|
||||
### 启用告警规则
|
||||
|
||||
在 Prometheus 配置中添加告警规则文件：
|
||||
|
||||
```yaml
|
||||
# prometheus.yml
|
||||
rule_files:
|
||||
- "/etc/prometheus/rules/*.yml"
|
||||
|
||||
# 或在 docker-compose.yml 中挂载文件：
|
||||
# volumes:
|
||||
# - ./prometheus-rules:/etc/prometheus/rules
|
||||
```
|
||||
|
||||
### 仪表盘使用
|
||||
|
||||
仪表盘在 Grafana 启动时自动预置。它们使用 `${DS_PROMETHEUS}` 数据源变量，因此需要在 Grafana 中配置 Prometheus 数据源。
|
||||
|
||||
优化发布期间需要关注的关键面板：
|
||||
|
||||
1. **GET 延迟按 Reader Path** - 对比 `codec_streaming` vs `legacy_duplex` 延迟
|
||||
2. **Early-Stop 命中率** - 验证 early-stop 是否有效触发
|
||||
3. **Pipeline 失败率** - 检测优化引入的新故障模式
|
||||
4. **Bitrot 校验失败** - 确保数据完整性
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 前置条件
|
||||
|
||||
- Docker
|
||||
- Docker Compose
|
||||
|
||||
### 部署
|
||||
|
||||
运行以下命令启动整个技术栈：
|
||||
1. 进入 `.docker/observability` 目录
|
||||
2. 执行以下命令启动服务：
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose -f docker-compose.yml up -d
|
||||
```
|
||||
|
||||
### Tempo 高可用模式
|
||||
### 访问监控面板
|
||||
|
||||
默认的 `docker-compose.yml` 对应单机栈。
|
||||
如果需要基于 Kafka 的 HA Tempo 配置，请使用：
|
||||
服务启动后，可通过以下地址访问各个监控面板：
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
|
||||
- Grafana: `http://localhost:3000` (默认账号/密码：`admin`/`admin`)
|
||||
- Jaeger: `http://localhost:16686`
|
||||
- Prometheus: `http://localhost:9090`
|
||||
|
||||
## 配置可观测性
|
||||
|
||||
```shell
|
||||
export RUSTFS_OBS_ENDPOINT="http://localhost:4317" # OpenTelemetry Collector 地址
|
||||
```
|
||||
|
||||
### 访问仪表盘
|
||||
|
||||
| 服务 | URL | 凭据 | 描述 |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | 主要可视化中心。 |
|
||||
| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | 指标查询和状态。 |
|
||||
| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | 辅助追踪可视化。 |
|
||||
| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo 状态/指标。 |
|
||||
|
||||
## 配置
|
||||
|
||||
### 数据持久化
|
||||
|
||||
数据存储在以下 Docker 卷中：
|
||||
|
||||
- `prometheus-data`: Prometheus 指标
|
||||
- `tempo-data`: Tempo 追踪 (WAL 和 Blocks)
|
||||
- `loki-data`: Loki 日志 (Chunks 和 Rules)
|
||||
- `jaeger-data`: Jaeger 追踪 (Badger DB)
|
||||
|
||||
要清除所有数据：
|
||||
|
||||
```bash
|
||||
docker compose down -v
|
||||
```
|
||||
|
||||
### 自定义
|
||||
|
||||
- **Prometheus**: 编辑 `prometheus.yml` 以添加抓取目标或告警规则。
|
||||
- **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
|
||||
- **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
|
||||
|
||||
### 验证 RustFS Trace
|
||||
|
||||
当 RustFS 将 `RUSTFS_OBS_ENDPOINT` 指向这套技术栈时，应将该值视为
|
||||
OTLP/HTTP 的基础 URL，例如：
|
||||
|
||||
```bash
|
||||
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
|
||||
```
|
||||
|
||||
RustFS 会自动在该基础 URL 后补全：
|
||||
|
||||
- `/v1/traces`
|
||||
- `/v1/metrics`
|
||||
- `/v1/logs`
|
||||
|
||||
需要注意：
|
||||
|
||||
- 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
|
||||
- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
|
||||
- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
|
||||
如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
|
||||
|
||||
最小验证流程：
|
||||
|
||||
```bash
|
||||
# 1. 启动本目录下的可观测性技术栈。
|
||||
docker compose up -d
|
||||
|
||||
# 2. 以 OTLP/HTTP 导出方式启动 RustFS，并提高 span 可见性。
|
||||
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
|
||||
export RUSTFS_OBS_LOGGER_LEVEL=debug
|
||||
|
||||
# 3. 产生真实请求流量。
|
||||
curl -I http://127.0.0.1:9000/health
|
||||
curl -I http://127.0.0.1:9000/health/ready
|
||||
|
||||
# 4. 到 Grafana 或 Jaeger 中检查。
|
||||
# Grafana: http://localhost:3000
|
||||
# Jaeger: http://localhost:16686
|
||||
```
|
||||
|
||||
如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
|
||||
“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
|
||||
|
||||
## 故障排除
|
||||
|
||||
- **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
|
||||
- **日志**: 使用 `docker compose logs -f <service_name>` 查看特定服务的日志。
|
||||
- **Otel Collector**: 检查 `http://localhost:13133` 获取健康状态，检查 `http://localhost:1888/debug/pprof/` 进行性能分析。
|
||||
|
||||
@@ -1,268 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
services:
|
||||
rustfs:
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
image: rustfs/rustfs:latest
|
||||
container_name: rustfs-server
|
||||
ports:
|
||||
- "9000:9000" # S3 API port
|
||||
- "9001:9001" # Console port
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=rustfsadmin
|
||||
- RUSTFS_SECRET_KEY=rustfsadmin
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=info
|
||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
||||
- RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
|
||||
volumes:
|
||||
- rustfs-data:/data/rustfs
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"sh",
|
||||
"-c",
|
||||
"curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
|
||||
]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
depends_on:
|
||||
otel-collector:
|
||||
condition: service_started
|
||||
|
||||
rustfs-init:
|
||||
image: alpine
|
||||
container_name: rustfs-init
|
||||
volumes:
|
||||
- rustfs-data:/data
|
||||
networks:
|
||||
- otel-network
|
||||
command: >
|
||||
sh -c "
|
||||
chown -R 10001:10001 /data &&
|
||||
echo 'Volume Permissions fixed' &&
|
||||
exit 0
|
||||
"
|
||||
restart: no
|
||||
|
||||
# --- Tracing ---
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:latest
|
||||
container_name: tempo
|
||||
command: [ "-config.file=/etc/tempo.yaml" ]
|
||||
volumes:
|
||||
- ./tempo.yaml:/etc/tempo.yaml:ro
|
||||
- tempo-data:/var/tempo
|
||||
ports:
|
||||
- "3200:3200" # tempo
|
||||
- "4317" # otlp grpc
|
||||
- "4318" # otlp http
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "/tempo", "-version" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
|
||||
redpanda:
|
||||
image: redpandadata/redpanda:latest # for tempo ingest
|
||||
container_name: redpanda
|
||||
ports:
|
||||
- "9092:9092"
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
command: >
|
||||
redpanda start --overprovisioned
|
||||
--mode=dev-container
|
||||
--kafka-addr=PLAINTEXT://0.0.0.0:9092
|
||||
--advertise-kafka-addr=PLAINTEXT://redpanda:9092
|
||||
|
||||
jaeger:
|
||||
image: jaegertracing/jaeger:latest
|
||||
container_name: jaeger
|
||||
environment:
|
||||
- SPAN_STORAGE_TYPE=badger
|
||||
- BADGER_EPHEMERAL=false
|
||||
- BADGER_DIRECTORY_VALUE=/badger/data
|
||||
- BADGER_DIRECTORY_KEY=/badger/key
|
||||
- COLLECTOR_OTLP_ENABLED=true
|
||||
volumes:
|
||||
- ./jaeger.yaml:/etc/jaeger/config.yml
|
||||
- jaeger-data:/badger
|
||||
ports:
|
||||
- "16686:16686" # Web UI
|
||||
- "14269:14269" # Admin/Metrics
|
||||
- "4317" # otlp grpc
|
||||
- "4318" # otlp http
|
||||
command: [ "--config", "/etc/jaeger/config.yml" ]
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
|
||||
# --- Metrics ---
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:latest
|
||||
container_name: prometheus
|
||||
volumes:
|
||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- prometheus-data:/prometheus
|
||||
ports:
|
||||
- "9090:9090"
|
||||
command:
|
||||
- "--config.file=/etc/prometheus/prometheus.yml"
|
||||
- "--web.enable-otlp-receiver" # Enable OTLP
|
||||
- "--web.enable-remote-write-receiver" # Enable remote write
|
||||
- "--enable-feature=promql-experimental-functions" # Enable info()
|
||||
- "--storage.tsdb.retention.time=30d"
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- otel-network
|
||||
healthcheck:
|
||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
# --- Logging ---
|
||||
|
||||
loki:
|
||||
image: grafana/loki:latest
|
||||
container_name: loki
|
||||
volumes:
|
||||
- ./loki.yaml:/etc/loki/loki.yaml:ro
|
||||
- loki-data:/loki
|
||||
ports:
|
||||
- "3100:3100"
|
||||
command: -config.file=/etc/loki/loki.yaml
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
|
||||
interval: 15s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 60s
|
||||
|
||||
# --- Collection ---
|
||||
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:latest
|
||||
volumes:
|
||||
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
|
||||
ports:
|
||||
- "1888:1888" # pprof
|
||||
- "8888:8888" # Prometheus metrics for Collector
|
||||
- "8889:8889" # Prometheus metrics for application indicators
|
||||
- "13133:13133" # health check
|
||||
- "4317:4317" # OTLP gRPC
|
||||
- "4318:4318" # OTLP HTTP
|
||||
- "55679:55679" # zpages
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- tempo
|
||||
- jaeger
|
||||
- prometheus
|
||||
- loki
|
||||
healthcheck:
|
||||
test: [ "CMD", "/otelcol-contrib", "--version" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
# --- Profiles ---
|
||||
|
||||
pyroscope:
|
||||
image: grafana/pyroscope:latest
|
||||
container_name: pyroscope
|
||||
ports:
|
||||
- "4040:4040"
|
||||
command:
|
||||
- -self-profiling.disable-push=true
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
|
||||
# --- Visualization ---
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
container_name: grafana
|
||||
ports:
|
||||
- "3000:3000"
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_PASSWORD=admin
|
||||
- GF_SECURITY_ADMIN_USER=admin
|
||||
volumes:
|
||||
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./grafana/dashboards:/etc/grafana/dashboards:ro
|
||||
- grafana-data:/var/lib/grafana
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- prometheus
|
||||
- tempo
|
||||
- loki
|
||||
healthcheck:
|
||||
test:
|
||||
[ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
volumes:
|
||||
rustfs-data:
|
||||
tempo-data:
|
||||
jaeger-data:
|
||||
prometheus-data:
|
||||
loki-data:
|
||||
grafana-data:
|
||||
|
||||
networks:
|
||||
otel-network:
|
||||
driver: bridge
|
||||
name: "network_otel"
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.28.0.0/16
|
||||
driver_opts:
|
||||
com.docker.network.enable_ipv6: "true"
|
||||
@@ -1,62 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Docker Compose override file for High Availability Tempo setup
|
||||
#
|
||||
# Usage:
|
||||
# docker-compose -f docker-compose-example-for-rustfs.yml \
|
||||
# -f docker-compose-tempo-ha-override.yml up
|
||||
|
||||
services:
|
||||
# Override Tempo to use high-availability configuration
|
||||
tempo:
|
||||
volumes:
|
||||
- ./tempo-ha.yaml:/etc/tempo.yaml:ro
|
||||
- tempo-data:/var/tempo
|
||||
ports:
|
||||
- "3200:3200" # Tempo HTTP
|
||||
- "4317:4317" # OTLP gRPC
|
||||
- "4318:4318" # OTLP HTTP
|
||||
- "7946:7946" # Memberlist
|
||||
- "14250:14250" # Jaeger gRPC
|
||||
- "14268:14268" # Jaeger Thrift HTTP
|
||||
- "9411:9411" # Zipkin
|
||||
environment:
|
||||
- TEMPO_MEMBERLIST_BIND_PORT=7946
|
||||
healthcheck:
|
||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
depends_on:
|
||||
- redpanda
|
||||
|
||||
volumes:
|
||||
tempo-data:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: tmpfs
|
||||
device: tmpfs
|
||||
o: "size=4g" # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
|
||||
|
||||
# Network configuration remains the same
|
||||
# networks:
|
||||
# otel-network:
|
||||
# driver: bridge
|
||||
# name: "network_otel"
|
||||
# ipam:
|
||||
# config:
|
||||
# - subnet: 172.28.0.0/16
|
||||
|
||||
@@ -14,198 +14,93 @@
|
||||
|
||||
services:
|
||||
|
||||
# --- Tracing ---
|
||||
tempo-init:
|
||||
image: busybox:latest
|
||||
command: ["sh", "-c", "chown -R 10001:10001 /var/tempo"]
|
||||
volumes:
|
||||
- ./tempo-data:/var/tempo
|
||||
user: root
|
||||
networks:
|
||||
- otel-network
|
||||
restart: "no"
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:2.10.5
|
||||
container_name: tempo
|
||||
command: [ "-config.file=/etc/tempo.yaml" ]
|
||||
image: grafana/tempo:latest
|
||||
user: "10001" # The container must be started with root to execute chown in the script
|
||||
command: [ "-config.file=/etc/tempo.yaml" ] # This is passed as a parameter to the entry point script
|
||||
volumes:
|
||||
- ./tempo.yaml:/etc/tempo.yaml:ro
|
||||
- tempo-data:/var/tempo
|
||||
- ./tempo-data:/var/tempo
|
||||
ports:
|
||||
- "3200:3200" # tempo
|
||||
- "4317" # otlp grpc
|
||||
- "4318" # otlp http
|
||||
- "7946" # memberlist
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "/tempo", "-version" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
|
||||
vulture:
|
||||
image: grafana/tempo-vulture:latest
|
||||
restart: always
|
||||
command:
|
||||
[
|
||||
"-prometheus-listen-address=:8080",
|
||||
"-tempo-query-url=http://tempo:3200",
|
||||
"-tempo-push-url=http://tempo:4317",
|
||||
]
|
||||
depends_on:
|
||||
- tempo
|
||||
|
||||
jaeger:
|
||||
image: jaegertracing/jaeger:latest
|
||||
container_name: jaeger
|
||||
environment:
|
||||
- SPAN_STORAGE_TYPE=badger
|
||||
- BADGER_EPHEMERAL=false
|
||||
- BADGER_DIRECTORY_VALUE=/badger/data
|
||||
- BADGER_DIRECTORY_KEY=/badger/key
|
||||
- COLLECTOR_OTLP_ENABLED=true
|
||||
volumes:
|
||||
- ./jaeger.yaml:/etc/jaeger/config.yml
|
||||
- jaeger-data:/badger
|
||||
ports:
|
||||
- "16686:16686" # Web UI
|
||||
- "18888:8888" # Metrics
|
||||
- "4317" # otlp grpc
|
||||
- "4318" # otlp http
|
||||
command: [ "--config", "/etc/jaeger/config.yml" ]
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:8888/metrics" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
|
||||
# --- Metrics ---
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:latest
|
||||
container_name: prometheus
|
||||
volumes:
|
||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- ./prometheus-rules:/etc/prometheus/rules:ro
|
||||
- prometheus-data:/prometheus
|
||||
ports:
|
||||
- "9090:9090"
|
||||
command:
|
||||
- "--config.file=/etc/prometheus/prometheus.yml"
|
||||
- "--web.enable-otlp-receiver" # Enable OTLP
|
||||
- "--web.enable-remote-write-receiver" # Enable remote write
|
||||
- "--enable-feature=promql-experimental-functions" # Enable info()
|
||||
- "--storage.tsdb.retention.time=30d"
|
||||
- "24317:4317" # otlp grpc
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- otel-network
|
||||
healthcheck:
|
||||
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
# --- Logging ---
|
||||
|
||||
loki:
|
||||
image: grafana/loki:latest
|
||||
container_name: loki
|
||||
volumes:
|
||||
- ./loki.yaml:/etc/loki/loki.yaml:ro
|
||||
- loki-data:/loki
|
||||
ports:
|
||||
- "3100:3100"
|
||||
command: -config.file=/etc/loki/loki.yaml
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: [ "CMD", "/usr/bin/loki", "--version" ]
|
||||
interval: 15s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 60s
|
||||
|
||||
# --- Collection ---
|
||||
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:latest
|
||||
image: otel/opentelemetry-collector-contrib:0.129.1
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
volumes:
|
||||
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
|
||||
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
|
||||
ports:
|
||||
- "1888:1888" # pprof
|
||||
- "8888:8888" # Prometheus metrics for Collector
|
||||
- "8889:8889" # Prometheus metrics for application indicators
|
||||
- "13133:13133" # health check
|
||||
- "4317:4317" # OTLP gRPC
|
||||
- "4318:4318" # OTLP HTTP
|
||||
- "55679:55679" # zpages
|
||||
- "1888:1888"
|
||||
- "8888:8888"
|
||||
- "8889:8889"
|
||||
- "13133:13133"
|
||||
- "4317:4317"
|
||||
- "4318:4318"
|
||||
- "55679:55679"
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- tempo
|
||||
- jaeger
|
||||
- prometheus
|
||||
- loki
|
||||
healthcheck:
|
||||
test: [ "CMD", "/otelcol-contrib", "--version" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
# --- Profiles ---
|
||||
|
||||
pyroscope:
|
||||
image: grafana/pyroscope:latest
|
||||
container_name: pyroscope
|
||||
jaeger:
|
||||
image: jaegertracing/jaeger:2.8.0
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
ports:
|
||||
- "4040:4040"
|
||||
command:
|
||||
- -self-profiling.disable-push=true
|
||||
- "16686:16686"
|
||||
- "14317:4317"
|
||||
- "14318:4318"
|
||||
networks:
|
||||
- otel-network
|
||||
prometheus:
|
||||
image: prom/prometheus:v3.4.2
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
volumes:
|
||||
- ./prometheus.yml:/etc/prometheus/prometheus.yml
|
||||
ports:
|
||||
- "9090:9090"
|
||||
networks:
|
||||
- otel-network
|
||||
loki:
|
||||
image: grafana/loki:3.5.1
|
||||
environment:
|
||||
- TZ=Asia/Shanghai
|
||||
volumes:
|
||||
- ./loki-config.yaml:/etc/loki/local-config.yaml
|
||||
ports:
|
||||
- "3100:3100"
|
||||
command: -config.file=/etc/loki/local-config.yaml
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
|
||||
# --- Visualization ---
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
container_name: grafana
|
||||
image: grafana/grafana:12.0.2
|
||||
ports:
|
||||
- "3000:3000"
|
||||
- "3000:3000" # Web UI
|
||||
volumes:
|
||||
- ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_PASSWORD=admin
|
||||
- GF_SECURITY_ADMIN_USER=admin
|
||||
volumes:
|
||||
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./grafana/dashboards:/etc/grafana/dashboards:ro
|
||||
- grafana-data:/var/lib/grafana
|
||||
- TZ=Asia/Shanghai
|
||||
networks:
|
||||
- otel-network
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- prometheus
|
||||
- tempo
|
||||
- loki
|
||||
healthcheck:
|
||||
test:
|
||||
[ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
volumes:
|
||||
tempo-data:
|
||||
jaeger-data:
|
||||
prometheus-data:
|
||||
loki-data:
|
||||
grafana-data:
|
||||
|
||||
networks:
|
||||
otel-network:
|
||||
driver: bridge
|
||||
name: "network_otel"
|
||||
ipam:
|
||||
config:
|
||||
- subnet: 172.28.0.0/16
|
||||
name: "network_otel_config"
|
||||
driver_opts:
|
||||
com.docker.network.enable_ipv6: "true"
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
uid: prometheus
|
||||
access: proxy
|
||||
orgId: 1
|
||||
url: http://prometheus:9090
|
||||
basicAuth: false
|
||||
isDefault: false
|
||||
version: 1
|
||||
editable: false
|
||||
jsonData:
|
||||
httpMethod: GET
|
||||
- name: Tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
orgId: 1
|
||||
url: http://tempo:3200
|
||||
basicAuth: false
|
||||
isDefault: true
|
||||
version: 1
|
||||
editable: false
|
||||
apiVersion: 1
|
||||
uid: tempo
|
||||
jsonData:
|
||||
httpMethod: GET
|
||||
serviceMap:
|
||||
datasourceUid: prometheus
|
||||
streamingEnabled:
|
||||
search: true
|
||||
@@ -1,500 +0,0 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"liveNow": false,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 20,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "line+area"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "transparent",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max", "sum"]
|
||||
}
|
||||
},
|
||||
"title": "Bitrot Verify Failures",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{stage=\"bitrot_verify\"}[$__rate_interval])) by (path, reason)",
|
||||
"legendFormat": "bitrot_fail {{path}} / {{reason}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"bitrot_verify_failed\"}[$__rate_interval])) by (path, stage)",
|
||||
"legendFormat": "verify_failed {{path}} / {{stage}}",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Bitrot Verify Duration (p50 / p95 / p99)",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p50",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p95",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p99",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 20,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "line+area"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "transparent",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max", "sum"]
|
||||
}
|
||||
},
|
||||
"title": "Decode Errors",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"decode_error\"}[$__rate_interval])) by (path, stage)",
|
||||
"legendFormat": "decode_error {{path}} / {{stage}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 20,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "line+area"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "transparent",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max", "sum"]
|
||||
}
|
||||
},
|
||||
"title": "Short Reads",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"short_read\"}[$__rate_interval])) by (path, stage)",
|
||||
"legendFormat": "short_read {{path}} / {{stage}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Shard Read Outcomes by Role",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_shard_read_total[$__rate_interval])) by (path, role, outcome)",
|
||||
"legendFormat": "{{path}} {{role}}/{{outcome}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Request Result Status Rate",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_request_results_total[$__rate_interval])) by (status)",
|
||||
"legendFormat": "{{status}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 38,
|
||||
"style": "dark",
|
||||
"tags": ["rustfs", "get-optimization", "data-integrity"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "Prometheus",
|
||||
"value": "Prometheus"
|
||||
},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"multi": false,
|
||||
"name": "DS_PROMETHEUS",
|
||||
"options": [],
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "RustFS GET Data Integrity",
|
||||
"uid": "rustfs-get-data-integrity",
|
||||
"version": 1
|
||||
}
|
||||
@@ -1,716 +0,0 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"liveNow": false,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_strategy_total[$__rate_interval])) or vector(0)",
|
||||
"legendFormat": "{{strategy}} / {{mode}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "GET Reader Setup Strategy Rate",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_strategy_by_size_total[$__rate_interval])) or vector(0)",
|
||||
"legendFormat": "{{path}} / {{strategy}} / {{size_bucket}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "GET Reader Setup Strategy by Size",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "none",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_count[$__rate_interval])), 1) or vector(0)",
|
||||
"legendFormat": "scheduled {{strategy}} / {{mode}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_count[$__rate_interval])), 1) or vector(0)",
|
||||
"legendFormat": "attempted {{strategy}} / {{mode}}",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_count[$__rate_interval])), 1) or vector(0)",
|
||||
"legendFormat": "ready {{strategy}} / {{mode}}",
|
||||
"refId": "C"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_count[$__rate_interval])), 1) or vector(0)",
|
||||
"legendFormat": "deferred {{strategy}} / {{mode}}",
|
||||
"refId": "D"
|
||||
}
|
||||
],
|
||||
"title": "GET Reader Setup Fanout Average",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "none",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_count[$__rate_interval])), 1) or vector(0)",
|
||||
"legendFormat": "scheduled {{path}} / {{strategy}} / {{size_bucket}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_count[$__rate_interval])), 1) or vector(0)",
|
||||
"legendFormat": "ready {{path}} / {{strategy}} / {{size_bucket}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "GET Reader Setup Fanout by Size",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
|
||||
"legendFormat": "p95 {{path}} / {{stage}} / {{size_bucket}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
|
||||
"legendFormat": "p99 {{path}} / {{stage}} / {{size_bucket}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "GET Hot Stage Latency by Size",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
|
||||
"legendFormat": "p95 {{path}} / {{stage}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
|
||||
"legendFormat": "p99 {{path}} / {{stage}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "GET Reader, Handler and Body Latency",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (subpath) (rate(rustfs_io_get_object_direct_memory_subpath_total[$__rate_interval])) or vector(0)",
|
||||
"legendFormat": "{{subpath}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (subpath, size_bucket) (rate(rustfs_io_get_object_direct_memory_subpath_by_size_total[$__rate_interval])) or vector(0)",
|
||||
"legendFormat": "{{subpath}} / {{size_bucket}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "GET Direct-Memory Subpath Rate",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 24
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_mmap_page_faults_total[$__rate_interval])) or vector(0)",
|
||||
"legendFormat": "mmap {{path}} / {{stage}} / {{kind}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_direct_read_page_faults_total[$__rate_interval])) or vector(0)",
|
||||
"legendFormat": "direct-read {{path}} / {{stage}} / {{kind}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "GET mmap vs Direct-Read Page Fault Rate",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 38,
|
||||
"style": "dark",
|
||||
"tags": [
|
||||
"rustfs",
|
||||
"get",
|
||||
"performance",
|
||||
"attribution"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "Prometheus",
|
||||
"value": "Prometheus"
|
||||
},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"multi": false,
|
||||
"name": "DS_PROMETHEUS",
|
||||
"options": [],
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timezone": "",
|
||||
"title": "RustFS GET Performance Attribution",
|
||||
"uid": "rustfs-get-performance-attribution",
|
||||
"version": 1,
|
||||
"weekStart": ""
|
||||
}
|
||||
@@ -1,525 +0,0 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"liveNow": false,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "none",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 20,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Concurrent GET Requests",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rustfs_io_get_object_concurrent_requests",
|
||||
"legendFormat": "concurrent GETs",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent",
|
||||
"min": 0,
|
||||
"max": 100,
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 20,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "line+area"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "transparent",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 70
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "IO Queue Utilization",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rustfs_io_queue_utilization_percent",
|
||||
"legendFormat": "queue utilization %",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rustfs_io_queue_permits_in_use",
|
||||
"legendFormat": "permits in use",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rustfs_io_queue_permits_available",
|
||||
"legendFormat": "permits available",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Disk Permit Wait Duration (p50 / p95 / p99)",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p50",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p95",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p99",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 15,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["lastNotNull"]
|
||||
}
|
||||
},
|
||||
"title": "RSS Trend (Process Memory)",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "process_resident_memory_bytes{job=~\"rustfs.*\"}",
|
||||
"legendFormat": "RSS {{instance}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rustfs_process_memory_bytes",
|
||||
"legendFormat": "RustFS memory {{instance}}",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "IO Queue Operations by Priority",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_queue_operations[$__rate_interval])) by (operation, priority)",
|
||||
"legendFormat": "{{operation}} / {{priority}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 15,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["sum"]
|
||||
}
|
||||
},
|
||||
"title": "IO Queue Congestion Events",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_queue_congestion_total[$__rate_interval]))",
|
||||
"legendFormat": "congestion events/s",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_timeout_total[$__rate_interval])) by (stage)",
|
||||
"legendFormat": "timeout {{stage}}",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 38,
|
||||
"style": "dark",
|
||||
"tags": ["rustfs", "get-optimization", "resource-impact"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "Prometheus",
|
||||
"value": "Prometheus"
|
||||
},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"multi": false,
|
||||
"name": "DS_PROMETHEUS",
|
||||
"options": [],
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "RustFS GET Resource Impact",
|
||||
"uid": "rustfs-get-resource-impact",
|
||||
"version": 1
|
||||
}
|
||||
@@ -1,530 +0,0 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"liveNow": false,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"axisLabel": "",
|
||||
"axisColorMode": "text",
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "GET Latency by Reader Path (p50 / p95 / p99)",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
|
||||
"legendFormat": "p50 {{path}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
|
||||
"legendFormat": "p95 {{path}}",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
|
||||
"legendFormat": "p99 {{path}}",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1,
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 15,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 0.3
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.7
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Early-Stop Hit Rate",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_metadata_early_stop_total{decision=\"hit\"}[$__rate_interval])) by (path) / clamp_min(sum(rate(rustfs_io_get_object_metadata_early_stop_total[$__rate_interval])) by (path), 1)",
|
||||
"legendFormat": "hit rate {{path}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1,
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 15,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "Codec Streaming Usage Rate",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])) by (decision) / clamp_min(sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])), 1)",
|
||||
"legendFormat": "codec_streaming {{decision}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])) by (strategy) / clamp_min(sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])), 1)",
|
||||
"legendFormat": "stream_strategy {{strategy}}",
|
||||
"refId": "B"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "bars",
|
||||
"fillOpacity": 80,
|
||||
"lineWidth": 0,
|
||||
"pointSize": 5,
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"mode": "normal",
|
||||
"group": "A"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["sum"]
|
||||
}
|
||||
},
|
||||
"title": "Pipeline Failure Rate by Stage",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total[$__rate_interval])) by (stage, reason)",
|
||||
"legendFormat": "{{stage}} / {{reason}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "GET Request Rate by Path",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_io_get_object_reader_path_total[$__rate_interval])) by (path)",
|
||||
"legendFormat": "{{path}}",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"values": false,
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": ""
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
},
|
||||
"legend": {
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": ["mean", "max"]
|
||||
}
|
||||
},
|
||||
"title": "GET Total Latency (p50 / p95 / p99)",
|
||||
"type": "timeseries",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p50",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p95",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
|
||||
"legendFormat": "p99",
|
||||
"refId": "C"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 38,
|
||||
"style": "dark",
|
||||
"tags": ["rustfs", "get-optimization"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "Prometheus",
|
||||
"value": "Prometheus"
|
||||
},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"multi": false,
|
||||
"name": "DS_PROMETHEUS",
|
||||
"options": [],
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "",
|
||||
"title": "RustFS GET Rollout Health",
|
||||
"uid": "rustfs-get-rollout-health",
|
||||
"version": 1
|
||||
}
|
||||
@@ -1,534 +0,0 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"id": null,
|
||||
"links": [],
|
||||
"liveNow": false,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (path, eager_status, size_bucket, buffer_bucket, large_concurrency_tuning) (rate(rustfs_s3_put_object_diagnostics_total[$__rate_interval])) or vector(0)",
|
||||
"legendFormat": "{{path}} / {{eager_status}} / {{size_bucket}} / {{buffer_bucket}} / large={{large_concurrency_tuning}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "PUT Diagnostics Decision Rate",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (path) (rate(rustfs_s3_put_object_path_total[$__rate_interval])) or vector(0)",
|
||||
"legendFormat": "{{path}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "PUT Path Rate",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ms",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
|
||||
"legendFormat": "p95 {{stage}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
|
||||
"legendFormat": "p99 {{stage}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "PUT Hot Stage Latency",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.50, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
|
||||
"legendFormat": "p50 {{path}} / {{size_bucket}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.95, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
|
||||
"legendFormat": "p95 {{path}} / {{size_bucket}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "PUT Selected Buffer Size",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1,
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 15,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_s3_put_object_zero_copy_eligible_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
|
||||
"legendFormat": "eligible / total",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_s3_put_object_zero_copy_enabled_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
|
||||
"legendFormat": "enabled / total",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "PUT Zero-Copy Eligibility Ratio",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"lineInterpolation": "smooth",
|
||||
"fillOpacity": 10,
|
||||
"pointSize": 5,
|
||||
"lineWidth": 1,
|
||||
"showPoints": "auto",
|
||||
"spanNulls": false,
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [
|
||||
"mean",
|
||||
"max"
|
||||
],
|
||||
"displayMode": "table",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(rustfs_s3_put_object_total[$__rate_interval])) or vector(0)",
|
||||
"legendFormat": "PUT total",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum by (stage) (rate(rustfs_system_storage_erasure_write_quorum_failures_total[$__rate_interval])) or vector(0)",
|
||||
"legendFormat": "write quorum failure {{stage}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "PUT Throughput and Quorum Failure Signals",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 38,
|
||||
"style": "dark",
|
||||
"tags": [
|
||||
"rustfs",
|
||||
"put",
|
||||
"performance",
|
||||
"attribution"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "Prometheus",
|
||||
"value": "Prometheus"
|
||||
},
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"multi": false,
|
||||
"name": "DS_PROMETHEUS",
|
||||
"options": [],
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"type": "datasource"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timezone": "",
|
||||
"title": "RustFS PUT Performance Attribution",
|
||||
"uid": "rustfs-put-performance-attribution",
|
||||
"version": 1,
|
||||
"weekStart": ""
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,25 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: "default"
|
||||
orgId: 1
|
||||
folder: ""
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 10
|
||||
options:
|
||||
path: /etc/grafana/dashboards
|
||||
@@ -1,98 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
uid: prometheus
|
||||
url: http://prometheus:9090
|
||||
access: proxy
|
||||
isDefault: true
|
||||
editable: false
|
||||
jsonData:
|
||||
httpMethod: GET
|
||||
exemplarTraceIdDestinations:
|
||||
- name: trace_id
|
||||
datasourceUid: tempo
|
||||
|
||||
- name: Tempo
|
||||
type: tempo
|
||||
uid: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
isDefault: false
|
||||
editable: false
|
||||
jsonData:
|
||||
httpMethod: GET
|
||||
serviceMap:
|
||||
datasourceUid: prometheus
|
||||
tracesToLogs:
|
||||
datasourceUid: loki
|
||||
tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
|
||||
mappedTags: [ { key: 'service.name', value: 'app' } ]
|
||||
spanStartTimeShift: '-1h'
|
||||
spanEndTimeShift: '1h'
|
||||
filterByTraceID: true
|
||||
filterBySpanID: false
|
||||
tracesToMetrics:
|
||||
datasourceUid: prometheus
|
||||
tags: [ { key: 'service.name' }, { key: 'job' } ]
|
||||
queries:
|
||||
- name: 'Service-Level Latency'
|
||||
query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
|
||||
- name: 'Service-Level Calls'
|
||||
query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
|
||||
- name: 'Service-Level Errors'
|
||||
query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
|
||||
- name: Loki
|
||||
type: loki
|
||||
uid: loki
|
||||
url: http://loki:3100
|
||||
basicAuth: false
|
||||
isDefault: false
|
||||
editable: false
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- datasourceUid: tempo
|
||||
matcherRegex: 'trace_id=(\w+)'
|
||||
name: 'TraceID'
|
||||
url: '$${__value.raw}'
|
||||
|
||||
- name: Jaeger
|
||||
type: jaeger
|
||||
uid: jaeger
|
||||
url: http://jaeger:16686
|
||||
access: proxy
|
||||
isDefault: false
|
||||
editable: false
|
||||
jsonData:
|
||||
tracesToLogs:
|
||||
datasourceUid: loki
|
||||
tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
|
||||
mappedTags: [ { key: 'service.name', value: 'app' } ]
|
||||
spanStartTimeShift: '1s'
|
||||
spanEndTimeShift: '-1s'
|
||||
filterByTraceID: true
|
||||
filterBySpanID: false
|
||||
|
||||
- name: Pyroscope
|
||||
type: grafana-pyroscope-datasource
|
||||
url: http://pyroscope:4040
|
||||
jsonData:
|
||||
minStep: '15s'
|
||||
@@ -0,0 +1,112 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
service:
|
||||
extensions: [ jaeger_storage, jaeger_query, remote_sampling, healthcheckv2 ]
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [ otlp, jaeger, zipkin ]
|
||||
processors: [ batch, adaptive_sampling ]
|
||||
exporters: [ jaeger_storage_exporter ]
|
||||
telemetry:
|
||||
resource:
|
||||
service.name: jaeger
|
||||
metrics:
|
||||
level: detailed
|
||||
readers:
|
||||
- pull:
|
||||
exporter:
|
||||
prometheus:
|
||||
host: 0.0.0.0
|
||||
port: 8888
|
||||
logs:
|
||||
level: debug
|
||||
# TODO Initialize telemetry tracer once OTEL released new feature.
|
||||
# https://github.com/open-telemetry/opentelemetry-collector/issues/10663
|
||||
|
||||
extensions:
|
||||
healthcheckv2:
|
||||
use_v2: true
|
||||
http:
|
||||
|
||||
# pprof:
|
||||
# endpoint: 0.0.0.0:1777
|
||||
# zpages:
|
||||
# endpoint: 0.0.0.0:55679
|
||||
|
||||
jaeger_query:
|
||||
storage:
|
||||
traces: some_store
|
||||
traces_archive: another_store
|
||||
ui:
|
||||
config_file: ./cmd/jaeger/config-ui.json
|
||||
log_access: true
|
||||
# The maximum duration that is considered for clock skew adjustments.
|
||||
# Defaults to 0 seconds, which means it's disabled.
|
||||
max_clock_skew_adjust: 0s
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:16685
|
||||
http:
|
||||
endpoint: 0.0.0.0:16686
|
||||
|
||||
jaeger_storage:
|
||||
backends:
|
||||
some_store:
|
||||
memory:
|
||||
max_traces: 1000000
|
||||
another_store:
|
||||
memory:
|
||||
max_traces: 1000000
|
||||
metric_backends:
|
||||
some_metrics_storage:
|
||||
prometheus:
|
||||
endpoint: http://prometheus:9090
|
||||
normalize_calls: true
|
||||
normalize_duration: true
|
||||
|
||||
remote_sampling:
|
||||
# You can either use file or adaptive sampling strategy in remote_sampling
|
||||
# file:
|
||||
# path: ./cmd/jaeger/sampling-strategies.json
|
||||
adaptive:
|
||||
sampling_store: some_store
|
||||
initial_sampling_probability: 0.1
|
||||
http:
|
||||
grpc:
|
||||
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
http:
|
||||
|
||||
jaeger:
|
||||
protocols:
|
||||
grpc:
|
||||
thrift_binary:
|
||||
thrift_compact:
|
||||
thrift_http:
|
||||
|
||||
zipkin:
|
||||
|
||||
processors:
|
||||
batch:
|
||||
# Adaptive Sampling Processor is required to support adaptive sampling.
|
||||
# It expects remote_sampling extension with `adaptive:` config to be enabled.
|
||||
adaptive_sampling:
|
||||
|
||||
exporters:
|
||||
jaeger_storage_exporter:
|
||||
trace_storage: some_store
|
||||
|
||||
@@ -1,74 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
service:
|
||||
extensions: [jaeger_storage, jaeger_query]
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [batch]
|
||||
exporters: [jaeger_storage_exporter, spanmetrics]
|
||||
metrics/spanmetrics:
|
||||
receivers: [spanmetrics]
|
||||
exporters: [prometheus]
|
||||
telemetry:
|
||||
resource:
|
||||
service.name: jaeger
|
||||
metrics:
|
||||
level: detailed
|
||||
readers:
|
||||
- pull:
|
||||
exporter:
|
||||
prometheus:
|
||||
host: 0.0.0.0
|
||||
port: 8888
|
||||
logs:
|
||||
level: DEBUG
|
||||
|
||||
extensions:
|
||||
jaeger_query:
|
||||
storage:
|
||||
traces: some_storage
|
||||
metrics: some_metrics_storage
|
||||
jaeger_storage:
|
||||
backends:
|
||||
some_storage:
|
||||
memory:
|
||||
max_traces: 100000
|
||||
metric_backends:
|
||||
some_metrics_storage:
|
||||
prometheus:
|
||||
endpoint: http://prometheus:9090
|
||||
normalize_calls: true
|
||||
normalize_duration: true
|
||||
|
||||
connectors:
|
||||
spanmetrics:
|
||||
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: "0.0.0.0:4317"
|
||||
http:
|
||||
endpoint: "0.0.0.0:4318"
|
||||
|
||||
processors:
|
||||
batch:
|
||||
|
||||
exporters:
|
||||
jaeger_storage_exporter:
|
||||
trace_storage: some_storage
|
||||
prometheus:
|
||||
endpoint: "0.0.0.0:8889"
|
||||
@@ -11,21 +11,22 @@
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
auth_enabled: false
|
||||
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
grpc_listen_port: 9095
|
||||
log_level: info
|
||||
grpc_listen_port: 9096
|
||||
log_level: debug
|
||||
grpc_server_max_concurrent_streams: 1000
|
||||
|
||||
common:
|
||||
instance_addr: 127.0.0.1
|
||||
path_prefix: /loki
|
||||
path_prefix: /tmp/loki
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /loki/chunks
|
||||
rules_directory: /loki/rules
|
||||
chunks_directory: /tmp/loki/chunks
|
||||
rules_directory: /tmp/loki/rules
|
||||
replication_factor: 1
|
||||
ring:
|
||||
kvstore:
|
||||
@@ -38,6 +39,9 @@ query_range:
|
||||
enabled: true
|
||||
max_size_mb: 100
|
||||
|
||||
limits_config:
|
||||
metric_aggregation_enabled: true
|
||||
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2020-10-24
|
||||
@@ -48,16 +52,26 @@ schema_config:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
|
||||
limits_config:
|
||||
reject_old_samples: true
|
||||
reject_old_samples_max_age: 168h
|
||||
allow_structured_metadata: true
|
||||
max_line_size: 256KB
|
||||
|
||||
pattern_ingester:
|
||||
enabled: true
|
||||
metric_aggregation:
|
||||
loki_address: localhost:3100
|
||||
|
||||
ruler:
|
||||
alertmanager_url: http://localhost:9093
|
||||
|
||||
frontend:
|
||||
encoding: protobuf
|
||||
|
||||
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
|
||||
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
|
||||
#
|
||||
# Statistics help us better understand how Loki is used, and they show us performance
|
||||
# levels for most users. This helps us prioritize features and documentation.
|
||||
# For more information on what's sent, look at
|
||||
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
|
||||
# Refer to the buildReport method to see what goes into a report.
|
||||
#
|
||||
# If you would like to disable reporting, uncomment the following lines:
|
||||
#analytics:
|
||||
# reporting_enabled: false
|
||||
@@ -15,102 +15,67 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
grpc: # OTLP gRPC 接收器
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
http: # OTLP HTTP 接收器
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
processors:
|
||||
batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 1024
|
||||
batch: # 批处理处理器，提升吞吐量
|
||||
timeout: 5s
|
||||
send_batch_size: 1000
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 1024
|
||||
spike_limit_mib: 256
|
||||
transform/logs:
|
||||
log_statements:
|
||||
- context: log
|
||||
statements:
|
||||
- set(attributes["message"], body.string)
|
||||
- set(attributes["log.body"], body.string)
|
||||
limit_mib: 512
|
||||
|
||||
exporters:
|
||||
otlp/tempo:
|
||||
endpoint: "tempo:4317"
|
||||
otlp/traces: # OTLP 导出器，用于跟踪数据
|
||||
endpoint: "jaeger:4317" # Jaeger 的 OTLP gRPC 端点
|
||||
tls:
|
||||
insecure: true # 开发环境禁用 TLS，生产环境需配置证书
|
||||
otlp/tempo: # OTLP 导出器，用于跟踪数据
|
||||
endpoint: "tempo:4317" # tempo 的 OTLP gRPC 端点
|
||||
tls:
|
||||
insecure: true # 开发环境禁用 TLS，生产环境需配置证书
|
||||
prometheus: # Prometheus 导出器，用于指标数据
|
||||
endpoint: "0.0.0.0:8889" # Prometheus 刮取端点
|
||||
namespace: "rustfs" # 指标前缀
|
||||
send_timestamps: true # 发送时间戳
|
||||
# enable_open_metrics: true
|
||||
otlphttp/loki: # Loki 导出器，用于日志数据
|
||||
# endpoint: "http://loki:3100/otlp/v1/logs"
|
||||
endpoint: "http://loki:3100/otlp/v1/logs"
|
||||
tls:
|
||||
insecure: true
|
||||
compression: gzip
|
||||
retry_on_failure:
|
||||
enabled: true
|
||||
initial_interval: 1s
|
||||
max_interval: 30s
|
||||
max_elapsed_time: 300s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 10
|
||||
queue_size: 5000
|
||||
|
||||
otlp/jaeger:
|
||||
endpoint: "jaeger:4317"
|
||||
tls:
|
||||
insecure: true
|
||||
compression: gzip
|
||||
retry_on_failure:
|
||||
enabled: true
|
||||
initial_interval: 1s
|
||||
max_interval: 30s
|
||||
max_elapsed_time: 300s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 10
|
||||
queue_size: 5000
|
||||
|
||||
prometheus:
|
||||
endpoint: "0.0.0.0:8889"
|
||||
send_timestamps: true
|
||||
metric_expiration: 5m
|
||||
resource_to_telemetry_conversion:
|
||||
enabled: true
|
||||
|
||||
otlphttp/loki:
|
||||
endpoint: "http://loki:3100/otlp"
|
||||
tls:
|
||||
insecure: true
|
||||
compression: gzip
|
||||
|
||||
extensions:
|
||||
health_check:
|
||||
endpoint: 0.0.0.0:13133
|
||||
pprof:
|
||||
endpoint: 0.0.0.0:1888
|
||||
zpages:
|
||||
endpoint: 0.0.0.0:55679
|
||||
|
||||
service:
|
||||
extensions: [ health_check, pprof, zpages ]
|
||||
extensions: [ health_check, pprof, zpages ] # 启用扩展
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [ otlp ]
|
||||
processors: [ memory_limiter, batch ]
|
||||
exporters: [ otlp/tempo, otlp/jaeger ]
|
||||
processors: [ memory_limiter,batch ]
|
||||
exporters: [ otlp/traces,otlp/tempo ]
|
||||
metrics:
|
||||
receivers: [ otlp ]
|
||||
processors: [ batch ]
|
||||
exporters: [ prometheus ]
|
||||
logs:
|
||||
receivers: [ otlp ]
|
||||
processors: [ batch, transform/logs ]
|
||||
processors: [ batch ]
|
||||
exporters: [ otlphttp/loki ]
|
||||
telemetry:
|
||||
logs:
|
||||
level: "info"
|
||||
encoding: "json"
|
||||
level: "info" # Collector 日志级别
|
||||
metrics:
|
||||
level: "normal"
|
||||
level: "detailed" # 可以是 basic, normal, detailed
|
||||
readers:
|
||||
- pull:
|
||||
- periodic:
|
||||
exporter:
|
||||
prometheus:
|
||||
host: '0.0.0.0'
|
||||
port: 8888
|
||||
otlp:
|
||||
protocol: http/protobuf
|
||||
endpoint: http://otel-collector:4318
|
||||
|
||||
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
groups:
|
||||
- name: rustfs-dashboard
|
||||
interval: 30s
|
||||
rules:
|
||||
- record: rustfs:http_server_requests:rate5m
|
||||
expr: sum by (job) (rate(rustfs_http_server_requests_total[5m]))
|
||||
|
||||
- record: rustfs:http_server_request_duration_seconds:p50_5m
|
||||
expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
|
||||
- record: rustfs:http_server_request_duration_seconds:p95_5m
|
||||
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
|
||||
- record: rustfs:http_server_request_duration_seconds:p99_5m
|
||||
expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
|
||||
|
||||
- record: rustfs:http_server_response_body_size_bytes:p50_5m
|
||||
expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
|
||||
- record: rustfs:http_server_response_body_size_bytes:p95_5m
|
||||
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
|
||||
- record: rustfs:http_server_response_body_size_bytes:p99_5m
|
||||
expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
|
||||
|
||||
- record: rustfs:log_cleaner_runs:rate15m
|
||||
expr: sum by (job) (rate(rustfs_log_cleaner_runs_total[15m]))
|
||||
- record: rustfs:log_cleaner_failure_ratio:rate5m
|
||||
expr: sum by (job) (rate(rustfs_log_cleaner_run_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_runs_total[5m])), 1e-9)
|
||||
- record: rustfs:log_cleaner_rotation_failure_ratio:rate5m
|
||||
expr: sum by (job) (rate(rustfs_log_cleaner_rotation_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_rotation_total[5m])), 1e-9)
|
||||
- record: rustfs:log_cleaner_rotation_duration_seconds:p95_5m
|
||||
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_rotation_duration_seconds_bucket[5m])))
|
||||
- record: rustfs:log_cleaner_compress_duration_seconds:p95_5m
|
||||
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])))
|
||||
|
||||
- record: rustfs:scanner_objects_scanned:rate5m
|
||||
expr: sum by (job) (rate(rustfs_scanner_objects_scanned_total[5m]))
|
||||
- record: rustfs:scanner_directories_scanned:rate5m
|
||||
expr: sum by (job) (rate(rustfs_scanner_directories_scanned_total[5m]))
|
||||
- record: rustfs:scanner_buckets_scanned:rate5m
|
||||
expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
|
||||
- record: rustfs:scanner_cycles_success:rate5m
|
||||
expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))
|
||||
|
||||
- record: rustfs:log_chain_op_event_mismatch:rate5m
|
||||
expr: sum by (job) (rate(rustfs_log_chain_op_event_mismatch_total[5m]))
|
||||
|
||||
- alert: RustFSLogChainOpEventMismatchDetected
|
||||
expr: rustfs:log_chain_op_event_mismatch:rate5m > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
component: s3-log-chain
|
||||
annotations:
|
||||
summary: "RustFS log-chain op/event mismatch detected"
|
||||
description: "job={{ $labels.job }} has non-zero rustfs_log_chain_op_event_mismatch_total rate for more than 10m. Check s3 op/event mapping changes."
|
||||
@@ -1,260 +0,0 @@
|
||||
# =============================================================================
|
||||
# RustFS GET Optimization — Prometheus Alerting Rules
|
||||
# =============================================================================
|
||||
#
|
||||
# Import into Prometheus:
|
||||
# 1. Copy this file to your Prometheus rules directory
|
||||
# 2. Add to prometheus.yml:
|
||||
# rule_files:
|
||||
# - "prometheus-alert-rules.yaml"
|
||||
# 3. Validate: promtool check rules prometheus-alert-rules.yaml
|
||||
# 4. Reload: curl -X POST http://localhost:9090/-/reload
|
||||
#
|
||||
# All metric names match those registered in crates/io-metrics/src/lib.rs
|
||||
# and documented in crates/ecstore/src/diagnostics/get.rs.
|
||||
#
|
||||
# Baseline comparison uses "offset 1d" — adjust to "offset 7d" for weekly
|
||||
# seasonality if your traffic pattern varies by day of week.
|
||||
# =============================================================================
|
||||
|
||||
groups:
|
||||
# ==========================================================================
|
||||
# Critical alerts — immediate action required
|
||||
# ==========================================================================
|
||||
- name: rustfs-get-optimization-critical
|
||||
interval: 30s
|
||||
rules:
|
||||
# ------------------------------------------------------------------
|
||||
# 1. GetP99Regression
|
||||
# GET p99 latency exceeds 2x the baseline (same time yesterday)
|
||||
# sustained for 10 minutes.
|
||||
# Action: Roll back the GET optimization immediately.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: GetP99Regression
|
||||
expr: |
|
||||
histogram_quantile(0.99,
|
||||
sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m])) by (le)
|
||||
)
|
||||
>
|
||||
2
|
||||
*
|
||||
histogram_quantile(0.99,
|
||||
sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m] offset 1d)) by (le)
|
||||
)
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "GET p99 latency regression detected (>2x baseline for 10m)"
|
||||
description: >-
|
||||
The 99th-percentile GET object latency is {{ $value | humanizeDuration }}
|
||||
which is more than double the baseline measured 24 hours ago.
|
||||
This indicates a severe performance regression introduced by
|
||||
a recent GET optimization change.
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/get-p99-regression"
|
||||
action: >
|
||||
1. Verify the regression is not caused by external factors
|
||||
(disk health, network, load spike).
|
||||
2. If confirmed optimization-related, roll back:
|
||||
- Set RUSTFS_GET_CODEC_STREAMING=0
|
||||
- Set RUSTFS_GET_METADATA_EARLY_STOP=0
|
||||
- Restart affected nodes.
|
||||
3. Collect flamegraphs and open a P0 incident.
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 2. PipelineFailureSpike
|
||||
# Pipeline failure rate exceeds 5x the baseline sustained for
|
||||
# 5 minutes. Covers all failure reasons: bitrot_mismatch,
|
||||
# decode_error, downstream_closed, io, read_quorum, timeout, etc.
|
||||
# Action: Investigate pipeline health and roll back if needed.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: PipelineFailureSpike
|
||||
expr: |
|
||||
sum(rate(rustfs_io_get_object_pipeline_failures_total[5m]))
|
||||
>
|
||||
5
|
||||
*
|
||||
sum(rate(rustfs_io_get_object_pipeline_failures_total[5m] offset 1d))
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "GET pipeline failure rate spike (>5x baseline for 5m)"
|
||||
description: >-
|
||||
The GET pipeline failure rate is {{ $value | printf "%.2f" }}/s,
|
||||
more than 5x the baseline from 24 hours ago.
|
||||
Failure reasons may include: bitrot_mismatch, decode_error,
|
||||
downstream_closed, io, range_or_length_invalid, read_quorum,
|
||||
short_read, timeout, unknown.
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/pipeline-failure-spike"
|
||||
action: >
|
||||
1. Check Grafana "GET Data Integrity" dashboard for failure
|
||||
breakdown by reason label.
|
||||
2. If decode_error or bitrot_mismatch dominates, stop
|
||||
optimization and investigate data integrity.
|
||||
3. If io or timeout dominates, check disk and network health.
|
||||
4. Roll back optimization if failures persist.
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 3. BitrotMismatchSpike
|
||||
# Bitrot verification mismatch rate exceeds 3x baseline for
|
||||
# 5 minutes. This is a data-integrity signal — shard checksums
|
||||
# do not match after read.
|
||||
#
|
||||
# The "bitrot_mismatch" reason is recorded on the
|
||||
# rustfs_io_get_object_pipeline_failures_total counter when a
|
||||
# StorageError::FileCorrupt or DiskError::FileCorrupt /
|
||||
# DiskError::PartMissingOrCorrupt is classified during the GET
|
||||
# pipeline (see classify_storage_error / classify_disk_error in
|
||||
# crates/ecstore/src/diagnostics/get.rs).
|
||||
#
|
||||
# Action: Stop optimization, investigate data integrity urgently.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: BitrotMismatchSpike
|
||||
expr: |
|
||||
sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m]))
|
||||
>
|
||||
3
|
||||
*
|
||||
sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m] offset 1d))
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "Bitrot mismatch rate spike (>3x baseline for 5m)"
|
||||
description: >-
|
||||
The rate of pipeline failures classified as bitrot_mismatch is
|
||||
{{ $value | printf "%.2f" }}/s, more than 3x the baseline from
|
||||
24 hours ago. This indicates shard checksum verification
|
||||
failures (FileCorrupt / PartMissingOrCorrupt) which may point
|
||||
to data corruption introduced by the GET optimization pipeline
|
||||
(e.g., incorrect decode, buffer reuse bug).
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/bitrot-mismatch-spike"
|
||||
action: >
|
||||
1. Immediately disable codec streaming:
|
||||
RUSTFS_GET_CODEC_STREAMING=0
|
||||
2. Run "mc admin scan" on affected buckets to verify on-disk
|
||||
integrity independent of the GET path.
|
||||
3. Compare xl.meta checksums across erasure shards.
|
||||
4. If corruption confirmed, initiate data recovery from parity.
|
||||
5. Do NOT re-enable optimization until root cause is identified.
|
||||
|
||||
# ==========================================================================
|
||||
# Warning alerts — investigation needed
|
||||
# ==========================================================================
|
||||
- name: rustfs-get-optimization-warning
|
||||
interval: 30s
|
||||
rules:
|
||||
# ------------------------------------------------------------------
|
||||
# 4. EarlyStopInsufficientQuorum
|
||||
# The metadata early-stop path is hitting "insufficient_quorum"
|
||||
# at a rate above 0.1/s for 5 minutes. This means too many
|
||||
# disks are failing to return valid metadata in time.
|
||||
# Action: Check disk health and metadata fanout latency.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: EarlyStopInsufficientQuorum
|
||||
expr: |
|
||||
sum(rate(rustfs_io_get_object_metadata_early_stop_total{reason="insufficient_quorum"}[5m]))
|
||||
> 0.1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "Early-stop insufficient quorum rate elevated (>0.1/s for 5m)"
|
||||
description: >-
|
||||
The metadata early-stop path is returning "insufficient_quorum"
|
||||
at {{ $value | printf "%.3f" }}/s. This means the bounded
|
||||
metadata fanout cannot gather enough valid responses before
|
||||
the quorum deadline, suggesting disk or network issues.
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/early-stop-quorum"
|
||||
action: >
|
||||
1. Check disk health: mc admin info --json | jq '.disks'
|
||||
2. Review rustfs_io_get_object_metadata_response_total by
|
||||
outcome (error, timeout, disk_not_found) in Grafana.
|
||||
3. Check rustfs_io_disk_permit_wait_duration_seconds for
|
||||
I/O scheduler saturation.
|
||||
4. If disks are healthy, consider increasing the early-stop
|
||||
timeout or temporarily disabling early-stop.
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 5. CodecStreamingFallbackSpike
|
||||
# The codec streaming fallback rate is >10x baseline for 10
|
||||
# minutes. This means the optimized codec streaming path is
|
||||
# being bypassed much more often than expected.
|
||||
# Action: Check fallback reasons and object eligibility.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: CodecStreamingFallbackSpike
|
||||
expr: |
|
||||
sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
|
||||
>
|
||||
10
|
||||
*
|
||||
sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m] offset 1d))
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "Codec streaming fallback rate spike (>10x baseline for 10m)"
|
||||
description: >-
|
||||
The codec streaming fallback rate is {{ $value | printf "%.2f" }}/s,
|
||||
more than 10x the baseline from 24 hours ago. Fallback reasons
|
||||
are labeled by "reason" — check Grafana for breakdown.
|
||||
Common reasons: object too small, multipart not supported,
|
||||
unsupported erasure layout, feature flag disabled.
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/codec-fallback-spike"
|
||||
action: >
|
||||
1. Query by reason label:
|
||||
sum by (reason) (rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
|
||||
2. If dominated by a single reason, investigate why that
|
||||
condition became more frequent (e.g., workload change,
|
||||
configuration drift).
|
||||
3. Cross-reference with rustfs_io_get_object_reader_path_total
|
||||
to verify the fallback path (legacy_duplex) is healthy.
|
||||
4. If fallback is expected (e.g., workload shifted to small
|
||||
objects), update the alert baseline.
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# 6. IoQueueSaturation
|
||||
# I/O queue utilization exceeds 90% for 5 minutes. High
|
||||
# utilization causes disk permit wait latency to increase and
|
||||
# can cascade into pipeline timeouts.
|
||||
# Action: Check disk load and consider reducing concurrency.
|
||||
# ------------------------------------------------------------------
|
||||
- alert: IoQueueSaturation
|
||||
expr: |
|
||||
rustfs_io_queue_utilization_percent > 90
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
team: rustfs-storage
|
||||
area: get-optimization
|
||||
annotations:
|
||||
summary: "I/O queue utilization >90% for 5m"
|
||||
description: >-
|
||||
The I/O queue utilization is {{ $value | printf "%.1f" }}%,
|
||||
sustained above 90% for 5 minutes. This indicates the disk
|
||||
I/O scheduler is near saturation, which will increase
|
||||
rustfs_io_disk_permit_wait_duration_seconds and may trigger
|
||||
pipeline timeouts.
|
||||
runbook_url: "https://internal.wiki/runbooks/rustfs/io-queue-saturation"
|
||||
action: >
|
||||
1. Check disk I/O metrics (iostat, node_exporter) for
|
||||
individual disk saturation.
|
||||
2. Review rustfs_io_queue_permits_in_use vs
|
||||
rustfs_io_queue_permits_available for permit exhaustion.
|
||||
3. Check rustfs_io_starvation_events for priority starvation.
|
||||
4. If GET optimization increased concurrency, consider:
|
||||
- Reducing RUSTFS_GET_PIPELINE_PARALLELISM
|
||||
- Lowering RUSTFS_IO_QUEUE_PERMITS
|
||||
5. If caused by background operations (ILM, healing), throttle
|
||||
those before adjusting GET concurrency.
|
||||
@@ -13,72 +13,16 @@
|
||||
# limitations under the License.
|
||||
|
||||
global:
|
||||
scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
|
||||
evaluation_interval: 15s
|
||||
external_labels:
|
||||
cluster: 'rustfs-dev' # Label to identify the cluster
|
||||
replica: '1' # Replica identifier
|
||||
|
||||
rule_files:
|
||||
- /etc/prometheus/rules/*.yml
|
||||
scrape_interval: 5s # 刮取间隔
|
||||
|
||||
scrape_configs:
|
||||
- job_name: 'otel-collector'
|
||||
static_configs:
|
||||
- targets: [ 'otel-collector:8888' ] # Scrape metrics from Collector
|
||||
scrape_interval: 10s
|
||||
|
||||
- job_name: 'rustfs-app-metrics'
|
||||
- targets: [ 'otel-collector:8888' ] # 从 Collector 刮取指标
|
||||
- job_name: 'otel-metrics'
|
||||
static_configs:
|
||||
- targets: [ 'otel-collector:8889' ] # Application indicators
|
||||
scrape_interval: 15s
|
||||
metric_relabel_configs:
|
||||
- source_labels: [ __name__ ]
|
||||
regex: 'go_.*'
|
||||
action: drop # Drop Go runtime metrics if not needed
|
||||
|
||||
- targets: [ 'otel-collector:8889' ] # 应用指标
|
||||
- job_name: 'tempo'
|
||||
static_configs:
|
||||
- targets: [ 'tempo:3200' ] # Scrape metrics from Tempo
|
||||
|
||||
- job_name: 'jaeger'
|
||||
static_configs:
|
||||
- targets: [ 'jaeger:14269' ] # Jaeger admin port (14269 is standard for admin/metrics)
|
||||
|
||||
- job_name: 'loki'
|
||||
static_configs:
|
||||
- targets: [ 'loki:3100' ]
|
||||
|
||||
- job_name: 'prometheus'
|
||||
static_configs:
|
||||
- targets: [ 'localhost:9090' ]
|
||||
|
||||
- job_name: 'vulture'
|
||||
static_configs:
|
||||
- targets:
|
||||
- 'vulture:8080'
|
||||
|
||||
otlp:
|
||||
promote_resource_attributes:
|
||||
- service.instance.id
|
||||
- service.name
|
||||
- service.namespace
|
||||
- cloud.availability_zone
|
||||
- cloud.region
|
||||
- container.name
|
||||
- deployment.environment.name
|
||||
- k8s.cluster.name
|
||||
- k8s.container.name
|
||||
- k8s.cronjob.name
|
||||
- k8s.daemonset.name
|
||||
- k8s.deployment.name
|
||||
- k8s.job.name
|
||||
- k8s.namespace.name
|
||||
- k8s.pod.name
|
||||
- k8s.replicaset.name
|
||||
- k8s.statefulset.name
|
||||
translation_strategy: NoUTF8EscapingWithSuffixes
|
||||
|
||||
storage:
|
||||
tsdb:
|
||||
out_of_order_time_window: 30m
|
||||
- targets: [ 'tempo:3200' ]
|
||||
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
kafka:
|
||||
brokers:
|
||||
- redpanda:9092
|
||||
@@ -0,0 +1 @@
|
||||
*
|
||||
@@ -1,286 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# High Availability Tempo Configuration for docker-compose-example-for-rustfs.yml
|
||||
# Features:
|
||||
# - Distributed architecture with multiple components
|
||||
# - Kafka-based ingestion for fault tolerance
|
||||
# - Replication factor of 3 for data resilience
|
||||
# - Query frontend for load balancing
|
||||
# - Metrics generation from traces
|
||||
# - WAL for durability
|
||||
|
||||
partition_ring_live_store: true
|
||||
stream_over_http_enabled: true
|
||||
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
http_server_read_timeout: 30s
|
||||
http_server_write_timeout: 30s
|
||||
grpc_server_max_recv_msg_size: 4194304 # 4MB
|
||||
grpc_server_max_send_msg_size: 4194304
|
||||
log_level: info
|
||||
log_format: json
|
||||
|
||||
# Memberlist configuration for distributed mode
|
||||
memberlist:
|
||||
node_name: tempo
|
||||
bind_port: 7946
|
||||
join_members:
|
||||
- tempo:7946
|
||||
retransmit_factor: 4
|
||||
node_timeout: 15s
|
||||
retransmit_interval: 300ms
|
||||
dead_node_reclaim_time: 30s
|
||||
|
||||
# Distributor configuration - receives traces and routes to ingesters
|
||||
distributor:
|
||||
ingester_write_path_enabled: true
|
||||
kafka_write_path_enabled: true
|
||||
rate_limit_bytes: 10MB
|
||||
rate_limit_enabled: true
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: "0.0.0.0:4317"
|
||||
max_concurrent_streams: 0
|
||||
max_receive_message_size: 4194304
|
||||
http:
|
||||
endpoint: "0.0.0.0:4318"
|
||||
cors:
|
||||
allowed_origins:
|
||||
- "*"
|
||||
max_age: 86400
|
||||
jaeger:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: "0.0.0.0:14250"
|
||||
thrift_http:
|
||||
endpoint: "0.0.0.0:14268"
|
||||
zipkin:
|
||||
endpoint: "0.0.0.0:9411"
|
||||
ring:
|
||||
kvstore:
|
||||
store: memberlist
|
||||
heartbeat_timeout: 5s
|
||||
replication_factor: 3
|
||||
heartbeat_interval: 5s
|
||||
|
||||
# Ingester configuration - stores traces and querying
|
||||
ingester:
|
||||
lifecycler:
|
||||
address: tempo
|
||||
ring:
|
||||
kvstore:
|
||||
store: memberlist
|
||||
replication_factor: 3
|
||||
max_cache_freshness_per_sec: 10s
|
||||
heartbeat_interval: 5s
|
||||
heartbeat_timeout: 5s
|
||||
num_tokens: 128
|
||||
tokens_file_path: /var/tempo/tokens.json
|
||||
claim_on_rollout: true
|
||||
trace_idle_period: 20s
|
||||
max_block_bytes: 10_000_000
|
||||
max_block_duration: 10m
|
||||
chunk_size_bytes: 1_000_000
|
||||
chunk_encoding: snappy
|
||||
wal:
|
||||
checkpoint_duration: 5s
|
||||
max_wal_blocks: 4
|
||||
metrics:
|
||||
enabled: true
|
||||
level: block
|
||||
target_info_duration: 15m
|
||||
|
||||
# WAL configuration for data durability
|
||||
wal:
|
||||
checkpoint_duration: 5s
|
||||
flush_on_shutdown: true
|
||||
path: /var/tempo/wal
|
||||
|
||||
# Kafka ingestion configuration - for high throughput scenarios
|
||||
ingest:
|
||||
enabled: true
|
||||
kafka:
|
||||
brokers: [ redpanda:9092 ]
|
||||
topic: tempo-ingest
|
||||
encoding: protobuf
|
||||
consumer_group: tempo-ingest-consumer
|
||||
session_timeout: 10s
|
||||
rebalance_timeout: 1m
|
||||
partition: auto
|
||||
verbosity: 2
|
||||
|
||||
# Query frontend configuration - distributed querying
|
||||
query_frontend:
|
||||
compression: gzip
|
||||
downstream_url: http://localhost:3200
|
||||
log_queries_longer_than: 5s
|
||||
cache_uncompressed_bytes: 100MB
|
||||
max_outstanding_requests_per_tenant: 100
|
||||
max_query_length: 48h
|
||||
max_query_lookback: 30d
|
||||
default_result_cache_ttl: 1m
|
||||
result_cache:
|
||||
cache:
|
||||
enable_fifocache: true
|
||||
default_validity: 1m
|
||||
rf1_after: "1999-01-01T00:00:00Z"
|
||||
mcp_server:
|
||||
enabled: true
|
||||
|
||||
# Querier configuration - queries traces
|
||||
querier:
|
||||
frontend_worker:
|
||||
frontend_address: localhost:3200
|
||||
grpc_client_config:
|
||||
max_recv_msg_size: 104857600
|
||||
max_concurrent_queries: 20
|
||||
max_metric_bytes_per_trace: 1MB
|
||||
|
||||
# Query scheduler configuration - for distributed querying
|
||||
query_scheduler:
|
||||
use_scheduler_ring: false
|
||||
|
||||
# Metrics generator configuration - generates metrics from traces
|
||||
metrics_generator:
|
||||
enabled: true
|
||||
registry:
|
||||
enabled: true
|
||||
external_labels:
|
||||
source: tempo
|
||||
cluster: rustfs-docker-ha
|
||||
environment: production
|
||||
storage:
|
||||
path: /var/tempo/generator/wal
|
||||
remote_write:
|
||||
- url: http://prometheus:9090/api/v1/write
|
||||
send_exemplars: true
|
||||
resource_to_telemetry_conversion:
|
||||
enabled: true
|
||||
processor:
|
||||
batch:
|
||||
timeout: 10s
|
||||
send_batch_size: 1024
|
||||
memory_limiter:
|
||||
check_interval: 5s
|
||||
limit_mib: 512
|
||||
spike_limit_mib: 128
|
||||
processors:
|
||||
- span-metrics
|
||||
- local-blocks
|
||||
- service-graphs
|
||||
generate_native_histograms: both
|
||||
|
||||
# Backend worker configuration
|
||||
backend_worker:
|
||||
backend_scheduler_addr: localhost:3200
|
||||
compaction:
|
||||
block_retention: 24h
|
||||
compacted_block_retention: 1h
|
||||
ring:
|
||||
kvstore:
|
||||
store: memberlist
|
||||
|
||||
# Backend scheduler configuration
|
||||
backend_scheduler:
|
||||
enabled: true
|
||||
provider:
|
||||
compaction:
|
||||
compaction:
|
||||
block_retention: 24h
|
||||
compacted_block_retention: 1h
|
||||
concurrency: 25
|
||||
v2_out_path: /var/tempo/blocks/compaction
|
||||
|
||||
# Storage configuration - local backend with proper retention
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
checkpoint_duration: 5s
|
||||
flush_on_shutdown: true
|
||||
local:
|
||||
path: /var/tempo/blocks
|
||||
bloom_filter_false_positive: 0.05
|
||||
bloom_shift: 4
|
||||
index:
|
||||
downsample_bytes: 1000000
|
||||
page_size_bytes: 0
|
||||
cache_size_bytes: 0
|
||||
pool:
|
||||
max_workers: 400
|
||||
queue_depth: 10000
|
||||
|
||||
# Compactor configuration - manages block compaction
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 168h # 7 days
|
||||
compacted_block_retention: 1h
|
||||
concurrency: 25
|
||||
v2_out_path: /var/tempo/blocks/compaction
|
||||
shard_count: 32
|
||||
max_block_bytes: 107374182400 # 100GB
|
||||
max_compaction_objects: 6000000
|
||||
max_time_per_tenant: 5m
|
||||
block_size_bytes: 107374182400
|
||||
ring:
|
||||
kvstore:
|
||||
store: memberlist
|
||||
heartbeat_interval: 5s
|
||||
heartbeat_timeout: 5s
|
||||
|
||||
# Limits configuration - rate limiting and quotas
|
||||
limits:
|
||||
max_traces_per_user: 10000
|
||||
max_bytes_per_trace: 10485760 # 10MB
|
||||
max_search_bytes_per_trace: 0
|
||||
forgiving_oversize_traces: true
|
||||
rate_limit_bytes: 10MB
|
||||
rate_limit_enabled: true
|
||||
ingestion_burst_size_bytes: 20MB
|
||||
ingestion_rate_limit_bytes: 10MB
|
||||
max_bytes_per_second: 10485760
|
||||
metrics_generator_max_active_series: 10000
|
||||
metrics_generator_max_churned_series: 10000
|
||||
metrics_generator_forta_out_of_order_ttl: 5m
|
||||
|
||||
# Override configuration
|
||||
overrides:
|
||||
defaults:
|
||||
metrics_generator:
|
||||
processors:
|
||||
- span-metrics
|
||||
- local-blocks
|
||||
- service-graphs
|
||||
generate_native_histograms: both
|
||||
max_active_series: 10000
|
||||
max_churned_series: 10000
|
||||
|
||||
# Usage reporting configuration
|
||||
usage_report:
|
||||
reporting_enabled: false
|
||||
|
||||
# Tracing configuration for debugging
|
||||
tracing:
|
||||
enabled: true
|
||||
jaeger:
|
||||
sampler:
|
||||
name: probabilistic
|
||||
param: 0.1
|
||||
reporter_log_spans: false
|
||||
|
||||
@@ -1,72 +1,55 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
stream_over_http_enabled: true
|
||||
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
log_level: info
|
||||
|
||||
memberlist:
|
||||
node_name: tempo
|
||||
bind_port: 7946
|
||||
join_members:
|
||||
- tempo:7946
|
||||
query_frontend:
|
||||
search:
|
||||
duration_slo: 5s
|
||||
throughput_bytes_slo: 1.073741824e+09
|
||||
metadata_slo:
|
||||
duration_slo: 5s
|
||||
throughput_bytes_slo: 1.073741824e+09
|
||||
trace_by_id:
|
||||
duration_slo: 5s
|
||||
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: "0.0.0.0:4317"
|
||||
http:
|
||||
endpoint: "0.0.0.0:4318"
|
||||
endpoint: "tempo:4317"
|
||||
|
||||
ingester:
|
||||
max_block_duration: 5m
|
||||
max_block_duration: 5m # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally
|
||||
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 1h # overall Tempo trace retention. set for demo purposes
|
||||
|
||||
metrics_generator:
|
||||
registry:
|
||||
external_labels:
|
||||
source: tempo
|
||||
cluster: docker-compose
|
||||
traces_storage:
|
||||
path: /var/tempo/generator/traces
|
||||
storage:
|
||||
path: /var/tempo/generator/wal
|
||||
remote_write:
|
||||
- url: http://prometheus:9090/api/v1/write
|
||||
send_exemplars: true
|
||||
|
||||
query_frontend:
|
||||
rf1_after: "1999-01-01T00:00:00Z"
|
||||
mcp_server:
|
||||
enabled: true
|
||||
traces_storage:
|
||||
path: /var/tempo/generator/traces
|
||||
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
backend: local # backend configuration to use
|
||||
wal:
|
||||
path: /var/tempo/wal # where to store the wal locally
|
||||
local:
|
||||
path: /var/tempo/blocks # where to store the traces locally
|
||||
path: /var/tempo/blocks
|
||||
|
||||
overrides:
|
||||
defaults:
|
||||
metrics_generator:
|
||||
processors: [ "span-metrics", "service-graphs", "local-blocks" ]
|
||||
generate_native_histograms: both
|
||||
|
||||
usage_report:
|
||||
reporting_enabled: false
|
||||
processors: [ service-graphs, span-metrics, local-blocks ] # enables metrics generator
|
||||
generate_native_histograms: both
|
||||
@@ -5,57 +5,71 @@
|
||||
|
||||
English | [中文](README_ZH.md)
|
||||
|
||||
This directory contains the configuration for an **alternative** observability stack using OpenObserve.
|
||||
This directory contains the configuration files for setting up an observability stack with OpenObserve and OpenTelemetry
|
||||
Collector.
|
||||
|
||||
## ⚠️ Note
|
||||
### Overview
|
||||
|
||||
For the **recommended** observability stack (Prometheus, Grafana, Tempo, Loki), please see `../observability/`.
|
||||
This setup provides a complete observability solution for your applications:
|
||||
|
||||
## 🌟 Overview
|
||||
- **OpenObserve**: A modern, open-source observability platform for logs, metrics, and traces.
|
||||
- **OpenTelemetry Collector**: Collects and processes telemetry data before sending it to OpenObserve.
|
||||
|
||||
OpenObserve is a lightweight, all-in-one observability platform that handles logs, metrics, and traces in a single binary. This setup is ideal for:
|
||||
- Resource-constrained environments.
|
||||
- Quick setup and testing.
|
||||
- Users who prefer a unified UI.
|
||||
### Setup Instructions
|
||||
|
||||
## 🚀 Quick Start
|
||||
1. **Prerequisites**:
|
||||
- Docker and Docker Compose installed
|
||||
- Sufficient memory resources (minimum 2GB recommended)
|
||||
|
||||
### 1. Start Services
|
||||
2. **Starting the Services**:
|
||||
```bash
|
||||
cd .docker/openobserve-otel
|
||||
docker compose -f docker-compose.yml up -d
|
||||
```
|
||||
|
||||
3. **Accessing the Dashboard**:
|
||||
- OpenObserve UI: http://localhost:5080
|
||||
- Default credentials:
|
||||
- Username: root@rustfs.com
|
||||
- Password: rustfs123
|
||||
|
||||
### Configuration
|
||||
|
||||
#### OpenObserve Configuration
|
||||
|
||||
The OpenObserve service is configured with:
|
||||
|
||||
- Root user credentials
|
||||
- Data persistence through a volume mount
|
||||
- Memory cache enabled
|
||||
- Health checks
|
||||
- Exposed ports:
|
||||
- 5080: HTTP API and UI
|
||||
- 5081: OTLP gRPC
|
||||
|
||||
#### OpenTelemetry Collector Configuration
|
||||
|
||||
The collector is configured to:
|
||||
|
||||
- Receive telemetry data via OTLP (HTTP and gRPC)
|
||||
- Collect logs from files
|
||||
- Process data in batches
|
||||
- Export data to OpenObserve
|
||||
- Manage memory usage
|
||||
|
||||
### Integration with Your Application
|
||||
|
||||
To send telemetry data from your application, configure your OpenTelemetry SDK to send data to:
|
||||
|
||||
- OTLP gRPC: `localhost:4317`
|
||||
- OTLP HTTP: `localhost:4318`
|
||||
|
||||
For example, in a Rust application using the `rustfs-obs` library:
|
||||
|
||||
```bash
|
||||
cd .docker/openobserve-otel
|
||||
docker compose up -d
|
||||
export RUSTFS_OBS_ENDPOINT=http://localhost:4317
|
||||
export RUSTFS_OBS_SERVICE_NAME=yourservice
|
||||
export RUSTFS_OBS_SERVICE_VERSION=1.0.0
|
||||
export RUSTFS_OBS_ENVIRONMENT=development
|
||||
```
|
||||
|
||||
### 2. Access Dashboard
|
||||
|
||||
- **URL**: [http://localhost:5080](http://localhost:5080)
|
||||
- **Username**: `root@rustfs.com`
|
||||
- **Password**: `rustfs123`
|
||||
|
||||
## 🛠️ Configuration
|
||||
|
||||
### OpenObserve
|
||||
|
||||
- **Persistence**: Data is persisted to a Docker volume.
|
||||
- **Ports**:
|
||||
- `5080`: HTTP API and UI
|
||||
- `5081`: OTLP gRPC
|
||||
|
||||
### OpenTelemetry Collector
|
||||
|
||||
- **Receivers**: OTLP (gRPC `4317`, HTTP `4318`)
|
||||
- **Exporters**: Sends data to OpenObserve.
|
||||
|
||||
## 🔗 Integration
|
||||
|
||||
Configure your application to send OTLP data to the collector:
|
||||
|
||||
- **Endpoint**: `http://localhost:4318` (HTTP) or `localhost:4317` (gRPC)
|
||||
|
||||
Example for RustFS:
|
||||
|
||||
```bash
|
||||
export RUSTFS_OBS_ENDPOINT=http://localhost:4318
|
||||
export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
|
||||
```
|
||||
|
||||
@@ -5,57 +5,71 @@
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
本目录包含使用 OpenObserve 的**替代**可观测性技术栈配置。
|
||||
## 中文
|
||||
|
||||
## ⚠️ 注意
|
||||
本目录包含搭建 OpenObserve 和 OpenTelemetry Collector 可观测性栈的配置文件。
|
||||
|
||||
对于**推荐**的可观测性技术栈（Prometheus, Grafana, Tempo, Loki），请参阅 `../observability/`。
|
||||
### 概述
|
||||
|
||||
## 🌟 概览
|
||||
此设置为应用程序提供了完整的可观测性解决方案：
|
||||
|
||||
OpenObserve 是一个轻量级、一体化的可观测性平台，在一个二进制文件中处理日志、指标和追踪。此设置非常适合：
|
||||
- 资源受限的环境。
|
||||
- 快速设置和测试。
|
||||
- 喜欢统一 UI 的用户。
|
||||
- **OpenObserve**：现代化、开源的可观测性平台，用于日志、指标和追踪。
|
||||
- **OpenTelemetry Collector**：收集和处理遥测数据，然后将其发送到 OpenObserve。
|
||||
|
||||
## 🚀 快速开始
|
||||
### 设置说明
|
||||
|
||||
### 1. 启动服务
|
||||
1. **前提条件**：
|
||||
- 已安装 Docker 和 Docker Compose
|
||||
- 足够的内存资源（建议至少 2GB）
|
||||
|
||||
2. **启动服务**：
|
||||
```bash
|
||||
cd .docker/openobserve-otel
|
||||
docker compose -f docker-compose.yml up -d
|
||||
```
|
||||
|
||||
3. **访问仪表板**：
|
||||
- OpenObserve UI：http://localhost:5080
|
||||
- 默认凭据：
|
||||
- 用户名：root@rustfs.com
|
||||
- 密码：rustfs123
|
||||
|
||||
### 配置
|
||||
|
||||
#### OpenObserve 配置
|
||||
|
||||
OpenObserve 服务配置：
|
||||
|
||||
- 根用户凭据
|
||||
- 通过卷挂载实现数据持久化
|
||||
- 启用内存缓存
|
||||
- 健康检查
|
||||
- 暴露端口：
|
||||
- 5080：HTTP API 和 UI
|
||||
- 5081：OTLP gRPC
|
||||
|
||||
#### OpenTelemetry Collector 配置
|
||||
|
||||
收集器配置为：
|
||||
|
||||
- 通过 OTLP（HTTP 和 gRPC）接收遥测数据
|
||||
- 从文件中收集日志
|
||||
- 批处理数据
|
||||
- 将数据导出到 OpenObserve
|
||||
- 管理内存使用
|
||||
|
||||
### 与应用程序集成
|
||||
|
||||
要从应用程序发送遥测数据，将 OpenTelemetry SDK 配置为发送数据到：
|
||||
|
||||
- OTLP gRPC:`localhost:4317`
|
||||
- OTLP HTTP:`localhost:4318`
|
||||
|
||||
例如，在使用 `rustfs-obs` 库的 Rust 应用程序中：
|
||||
|
||||
```bash
|
||||
cd .docker/openobserve-otel
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### 2. 访问仪表盘
|
||||
|
||||
- **URL**: [http://localhost:5080](http://localhost:5080)
|
||||
- **用户名**: `root@rustfs.com`
|
||||
- **密码**: `rustfs123`
|
||||
|
||||
## 🛠️ 配置
|
||||
|
||||
### OpenObserve
|
||||
|
||||
- **持久化**: 数据持久化到 Docker 卷。
|
||||
- **端口**:
|
||||
- `5080`: HTTP API 和 UI
|
||||
- `5081`: OTLP gRPC
|
||||
|
||||
### OpenTelemetry Collector
|
||||
|
||||
- **接收器**: OTLP (gRPC `4317`, HTTP `4318`)
|
||||
- **导出器**: 将数据发送到 OpenObserve。
|
||||
|
||||
## 🔗 集成
|
||||
|
||||
配置您的应用程序将 OTLP 数据发送到收集器：
|
||||
|
||||
- **端点**: `http://localhost:4318` (HTTP) 或 `localhost:4317` (gRPC)
|
||||
|
||||
RustFS 示例：
|
||||
|
||||
```bash
|
||||
export RUSTFS_OBS_ENDPOINT=http://localhost:4318
|
||||
export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
|
||||
```
|
||||
export RUSTFS_OBS_ENDPOINT=http://localhost:4317
|
||||
export RUSTFS_OBS_SERVICE_NAME=yourservice
|
||||
export RUSTFS_OBS_SERVICE_VERSION=1.0.0
|
||||
export RUSTFS_OBS_ENVIRONMENT=development
|
||||
```
|
||||
@@ -1,91 +0,0 @@
|
||||
# Rio compatibility compose files
|
||||
|
||||
These compose files prepare 4-node, 4-disk clusters for rio/rio-v2 storage format compatibility checks. All disks are bind-mounted under `.docker/compat/data` so the on-disk files remain available on the host.
|
||||
|
||||
## Clusters
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d --build
|
||||
docker compose -f .docker/compat/docker-compose.minio.yml up -d
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
||||
```
|
||||
|
||||
Default API endpoints:
|
||||
|
||||
- RustFS `1.0.0-beta.5`: `http://127.0.0.1:9100`
|
||||
- MinIO: `http://127.0.0.1:9200`
|
||||
- current main with `rio-v2`: `http://127.0.0.1:9300`
|
||||
|
||||
## Reading old datasets with rio-v2
|
||||
|
||||
Stop the writer cluster before mounting its disks into the rio-v2 cluster.
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml down
|
||||
RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
||||
|
||||
docker compose -f .docker/compat/docker-compose.minio.yml down
|
||||
RUSTFS_RIO_V2_DATASET=./data/minio \
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
||||
```
|
||||
|
||||
## 200G object mix
|
||||
|
||||
Use the same bucket/object matrix against the beta5 and MinIO endpoints, then read it back through the rio-v2 endpoint. A practical 200G mix is:
|
||||
|
||||
- 1 KiB x 1024
|
||||
- 1 MiB x 1024
|
||||
- 64 MiB x 512
|
||||
- 1 GiB x 64
|
||||
- 8 GiB x 12
|
||||
- 6 GiB x 1
|
||||
|
||||
Compression is enabled by default for RustFS and MinIO. Server-side KMS/SSE settings are intentionally left to environment variables or mounted key directories so real key material is not committed. For SSE-C cases, run the clusters with TLS because MinIO requires HTTPS for SSE-C.
|
||||
|
||||
## High-concurrency write/read stress
|
||||
|
||||
Use `run_rw_compat_stress.sh` to generate a manifest on an old endpoint, then verify the same objects through the rio-v2 endpoint after mounting the old disks.
|
||||
|
||||
```bash
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode write \
|
||||
--endpoint http://127.0.0.1:9100 \
|
||||
--access-key rustfsadmin \
|
||||
--secret-key rustfsadmin \
|
||||
--bucket compat-beta5 \
|
||||
--concurrency 96
|
||||
|
||||
RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
||||
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode verify \
|
||||
--endpoint http://127.0.0.1:9300 \
|
||||
--access-key rustfsadmin \
|
||||
--secret-key rustfsadmin \
|
||||
--bucket compat-beta5 \
|
||||
--concurrency 96 \
|
||||
--manifest target/compat/rw-stress-YYYYmmdd-HHMMSS/manifest.csv
|
||||
```
|
||||
|
||||
For encrypted datasets, add `--encryption sse-s3`, `--encryption sse-kms --sse-kms-key-id <key-id>`, or `--encryption sse-c --sse-c-key-file <raw-32-byte-key-file>` to both the write and verify commands.
|
||||
|
||||
## 5 GiB encrypted compatibility run
|
||||
|
||||
The `5g` profile covers 1 KiB, 1 MiB, 16 MiB, 64 MiB, and 1 GiB objects and totals exactly 5 GiB. Generate `compat-key.key` under `.docker/compat/kms/rustfs-compat`, enable local KMS on both RustFS clusters, then use the same encryption arguments while writing with beta5 and verifying with rio-v2. Set non-default local test credentials first because distributed listeners reject the built-in default credentials.
|
||||
|
||||
```bash
|
||||
export COMPAT_ACCESS_KEY='<non-default-access-key>'
|
||||
export COMPAT_SECRET_KEY='<non-default-secret-key>'
|
||||
|
||||
RUSTFS_ACCESS_KEY="$COMPAT_ACCESS_KEY" RUSTFS_SECRET_KEY="$COMPAT_SECRET_KEY" \
|
||||
RUSTFS_KMS_ENABLE=true RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true \
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d
|
||||
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode write --endpoint http://127.0.0.1:9100 \
|
||||
--access-key "$COMPAT_ACCESS_KEY" --secret-key "$COMPAT_SECRET_KEY" \
|
||||
--bucket compat-beta5-sse-s3 --profile 5g --concurrency 16 \
|
||||
--encryption sse-s3
|
||||
```
|
||||
@@ -1,88 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0.
|
||||
|
||||
x-minio-env: &minio-env
|
||||
MINIO_ROOT_USER: "${MINIO_ROOT_USER:-minioadmin}"
|
||||
MINIO_ROOT_PASSWORD: "${MINIO_ROOT_PASSWORD:-minioadmin}"
|
||||
MINIO_COMPRESSION_ENABLE: "${MINIO_COMPRESSION_ENABLE:-on}"
|
||||
MINIO_COMPRESSION_ALLOW_ENCRYPTION: "${MINIO_COMPRESSION_ALLOW_ENCRYPTION:-on}"
|
||||
MINIO_COMPRESSION_EXTENSIONS: "${MINIO_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
|
||||
MINIO_COMPRESSION_MIME_TYPES: "${MINIO_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
|
||||
|
||||
x-minio-node: &minio-node
|
||||
image: "${MINIO_IMAGE:-quay.io/minio/minio:latest}"
|
||||
command: server --console-address ":9001" "http://minio{1...4}:9000/data/disk{1...4}"
|
||||
environment: *minio-env
|
||||
networks:
|
||||
- minio-compat-net
|
||||
restart: unless-stopped
|
||||
|
||||
services:
|
||||
minio-permission-helper:
|
||||
image: alpine:3.23
|
||||
command: sh -c "mkdir -p /compat-data && chown -R 1000:1000 /compat-data"
|
||||
volumes:
|
||||
- ./data/minio:/compat-data
|
||||
restart: "no"
|
||||
|
||||
minio1:
|
||||
<<: *minio-node
|
||||
hostname: minio1
|
||||
depends_on:
|
||||
minio-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
volumes:
|
||||
- ./data/minio/node1/disk1:/data/disk1
|
||||
- ./data/minio/node1/disk2:/data/disk2
|
||||
- ./data/minio/node1/disk3:/data/disk3
|
||||
- ./data/minio/node1/disk4:/data/disk4
|
||||
ports:
|
||||
- "${MINIO_API_PORT:-9200}:9000"
|
||||
- "${MINIO_CONSOLE_PORT:-9201}:9001"
|
||||
|
||||
minio2:
|
||||
<<: *minio-node
|
||||
hostname: minio2
|
||||
depends_on:
|
||||
minio-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
volumes:
|
||||
- ./data/minio/node2/disk1:/data/disk1
|
||||
- ./data/minio/node2/disk2:/data/disk2
|
||||
- ./data/minio/node2/disk3:/data/disk3
|
||||
- ./data/minio/node2/disk4:/data/disk4
|
||||
ports:
|
||||
- "${MINIO_NODE2_PORT:-9202}:9000"
|
||||
|
||||
minio3:
|
||||
<<: *minio-node
|
||||
hostname: minio3
|
||||
depends_on:
|
||||
minio-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
volumes:
|
||||
- ./data/minio/node3/disk1:/data/disk1
|
||||
- ./data/minio/node3/disk2:/data/disk2
|
||||
- ./data/minio/node3/disk3:/data/disk3
|
||||
- ./data/minio/node3/disk4:/data/disk4
|
||||
ports:
|
||||
- "${MINIO_NODE3_PORT:-9203}:9000"
|
||||
|
||||
minio4:
|
||||
<<: *minio-node
|
||||
hostname: minio4
|
||||
depends_on:
|
||||
minio-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
volumes:
|
||||
- ./data/minio/node4/disk1:/data/disk1
|
||||
- ./data/minio/node4/disk2:/data/disk2
|
||||
- ./data/minio/node4/disk3:/data/disk3
|
||||
- ./data/minio/node4/disk4:/data/disk4
|
||||
ports:
|
||||
- "${MINIO_NODE4_PORT:-9204}:9000"
|
||||
|
||||
networks:
|
||||
minio-compat-net:
|
||||
driver: bridge
|
||||
@@ -1,104 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0.
|
||||
|
||||
x-rustfs-env: &rustfs-env
|
||||
RUSTFS_VOLUMES: "http://rustfs-beta5-node{1...4}:9000/data/disk{1...4}"
|
||||
RUSTFS_ADDRESS: ":9000"
|
||||
RUSTFS_CONSOLE_ADDRESS: ":9001"
|
||||
RUSTFS_CONSOLE_ENABLE: "true"
|
||||
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
|
||||
RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
|
||||
RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
|
||||
RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
|
||||
RUSTFS_OBS_LOG_DIRECTORY: "/logs"
|
||||
RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
|
||||
RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
|
||||
RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
|
||||
RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
|
||||
RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
|
||||
RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
|
||||
RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
|
||||
RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
|
||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
|
||||
|
||||
x-rustfs-node: &rustfs-node
|
||||
image: "${RUSTFS_BETA5_IMAGE:-rustfs/rustfs:1.0.0-beta.5}"
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile
|
||||
args:
|
||||
RELEASE: "1.0.0-beta.5"
|
||||
environment: *rustfs-env
|
||||
depends_on:
|
||||
rustfs-beta5-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
networks:
|
||||
- rustfs-beta5-net
|
||||
restart: unless-stopped
|
||||
|
||||
services:
|
||||
rustfs-beta5-permission-helper:
|
||||
image: alpine:3.23
|
||||
command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
|
||||
volumes:
|
||||
- ./data/rustfs-beta5:/compat-data
|
||||
- ./kms/rustfs-compat:/kms
|
||||
restart: "no"
|
||||
|
||||
rustfs-beta5-node1:
|
||||
<<: *rustfs-node
|
||||
hostname: rustfs-beta5-node1
|
||||
volumes:
|
||||
- ./data/rustfs-beta5/node1/disk1:/data/disk1
|
||||
- ./data/rustfs-beta5/node1/disk2:/data/disk2
|
||||
- ./data/rustfs-beta5/node1/disk3:/data/disk3
|
||||
- ./data/rustfs-beta5/node1/disk4:/data/disk4
|
||||
- ./data/rustfs-beta5/logs/node1:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_BETA5_API_PORT:-9100}:9000"
|
||||
- "${RUSTFS_BETA5_CONSOLE_PORT:-9101}:9001"
|
||||
|
||||
rustfs-beta5-node2:
|
||||
<<: *rustfs-node
|
||||
hostname: rustfs-beta5-node2
|
||||
volumes:
|
||||
- ./data/rustfs-beta5/node2/disk1:/data/disk1
|
||||
- ./data/rustfs-beta5/node2/disk2:/data/disk2
|
||||
- ./data/rustfs-beta5/node2/disk3:/data/disk3
|
||||
- ./data/rustfs-beta5/node2/disk4:/data/disk4
|
||||
- ./data/rustfs-beta5/logs/node2:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_BETA5_NODE2_PORT:-9102}:9000"
|
||||
|
||||
rustfs-beta5-node3:
|
||||
<<: *rustfs-node
|
||||
hostname: rustfs-beta5-node3
|
||||
volumes:
|
||||
- ./data/rustfs-beta5/node3/disk1:/data/disk1
|
||||
- ./data/rustfs-beta5/node3/disk2:/data/disk2
|
||||
- ./data/rustfs-beta5/node3/disk3:/data/disk3
|
||||
- ./data/rustfs-beta5/node3/disk4:/data/disk4
|
||||
- ./data/rustfs-beta5/logs/node3:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_BETA5_NODE3_PORT:-9103}:9000"
|
||||
|
||||
rustfs-beta5-node4:
|
||||
<<: *rustfs-node
|
||||
hostname: rustfs-beta5-node4
|
||||
volumes:
|
||||
- ./data/rustfs-beta5/node4/disk1:/data/disk1
|
||||
- ./data/rustfs-beta5/node4/disk2:/data/disk2
|
||||
- ./data/rustfs-beta5/node4/disk3:/data/disk3
|
||||
- ./data/rustfs-beta5/node4/disk4:/data/disk4
|
||||
- ./data/rustfs-beta5/logs/node4:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_BETA5_NODE4_PORT:-9104}:9000"
|
||||
|
||||
networks:
|
||||
rustfs-beta5-net:
|
||||
driver: bridge
|
||||
@@ -1,115 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0.
|
||||
|
||||
x-rustfs-rio-v2-env: &rustfs-rio-v2-env
|
||||
RUSTFS_VOLUMES: "http://rustfs-rio-v2-node{1...4}:9000/data/disk{1...4}"
|
||||
RUSTFS_ADDRESS: ":9000"
|
||||
RUSTFS_CONSOLE_ADDRESS: ":9001"
|
||||
RUSTFS_CONSOLE_ENABLE: "true"
|
||||
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
|
||||
RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
|
||||
RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
|
||||
RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
|
||||
RUSTFS_OBS_LOG_DIRECTORY: "/logs"
|
||||
RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
|
||||
RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
|
||||
RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
|
||||
RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
|
||||
RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
|
||||
RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
|
||||
RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
|
||||
RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
|
||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
|
||||
|
||||
x-rustfs-rio-v2-node: &rustfs-rio-v2-node
|
||||
image: "${RUSTFS_RIO_V2_IMAGE:-rustfs/rustfs:compat-rio-v2}"
|
||||
entrypoint:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
until getent hosts rustfs-rio-v2-node1 >/dev/null &&
|
||||
getent hosts rustfs-rio-v2-node2 >/dev/null &&
|
||||
getent hosts rustfs-rio-v2-node3 >/dev/null &&
|
||||
getent hosts rustfs-rio-v2-node4 >/dev/null; do
|
||||
sleep 1
|
||||
done
|
||||
exec /entrypoint.sh /usr/bin/rustfs
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile.source
|
||||
args:
|
||||
RUSTFS_BUILD_FEATURES: "rio-v2"
|
||||
environment: *rustfs-rio-v2-env
|
||||
depends_on:
|
||||
rustfs-rio-v2-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
networks:
|
||||
- rustfs-rio-v2-net
|
||||
restart: unless-stopped
|
||||
|
||||
services:
|
||||
rustfs-rio-v2-permission-helper:
|
||||
image: alpine:3.23
|
||||
command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
|
||||
volumes:
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}:/compat-data
|
||||
- ./kms/rustfs-compat:/kms
|
||||
restart: "no"
|
||||
|
||||
rustfs-rio-v2-node1:
|
||||
<<: *rustfs-rio-v2-node
|
||||
hostname: rustfs-rio-v2-node1
|
||||
volumes:
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk1:/data/disk1
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk2:/data/disk2
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk3:/data/disk3
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk4:/data/disk4
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node1:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_RIO_V2_API_PORT:-9300}:9000"
|
||||
- "${RUSTFS_RIO_V2_CONSOLE_PORT:-9301}:9001"
|
||||
|
||||
rustfs-rio-v2-node2:
|
||||
<<: *rustfs-rio-v2-node
|
||||
hostname: rustfs-rio-v2-node2
|
||||
volumes:
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk1:/data/disk1
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk2:/data/disk2
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk3:/data/disk3
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk4:/data/disk4
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node2:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_RIO_V2_NODE2_PORT:-9302}:9000"
|
||||
|
||||
rustfs-rio-v2-node3:
|
||||
<<: *rustfs-rio-v2-node
|
||||
hostname: rustfs-rio-v2-node3
|
||||
volumes:
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk1:/data/disk1
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk2:/data/disk2
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk3:/data/disk3
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk4:/data/disk4
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node3:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_RIO_V2_NODE3_PORT:-9303}:9000"
|
||||
|
||||
rustfs-rio-v2-node4:
|
||||
<<: *rustfs-rio-v2-node
|
||||
hostname: rustfs-rio-v2-node4
|
||||
volumes:
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk1:/data/disk1
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk2:/data/disk2
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk3:/data/disk3
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk4:/data/disk4
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node4:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_RIO_V2_NODE4_PORT:-9304}:9000"
|
||||
|
||||
networks:
|
||||
rustfs-rio-v2-net:
|
||||
driver: bridge
|
||||
@@ -1,639 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# High-concurrency S3 read/write stress runner for rio/rio-v2 format compatibility.
|
||||
# Write a manifest on an old endpoint, then verify the same manifest through rio-v2.
|
||||
|
||||
MODE="write"
|
||||
ENDPOINT=""
|
||||
ACCESS_KEY="${AWS_ACCESS_KEY_ID:-}"
|
||||
SECRET_KEY="${AWS_SECRET_ACCESS_KEY:-}"
|
||||
BUCKET="compat-rw-stress"
|
||||
REGION="us-east-1"
|
||||
CONCURRENCY=64
|
||||
OUT_DIR=""
|
||||
WORK_DIR=""
|
||||
MANIFEST=""
|
||||
PROFILE="200g"
|
||||
OBJECT_SPEC=""
|
||||
DATA_PATTERN="compressible"
|
||||
ENCRYPTION="none"
|
||||
SSE_KMS_KEY_ID=""
|
||||
SSE_C_KEY_FILE=""
|
||||
CLIENT="mc"
|
||||
AWS_BIN="${AWS_BIN:-aws}"
|
||||
MC_BIN="${MC_BIN:-}"
|
||||
KEEP_PAYLOADS=false
|
||||
DRY_RUN=false
|
||||
RESUME=false
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
.docker/compat/run_rw_compat_stress.sh --mode <write|verify|mixed> \
|
||||
--endpoint <url> --access-key <ak> --secret-key <sk> [options]
|
||||
|
||||
Modes:
|
||||
write Create bucket, upload objects concurrently, and write manifest.csv.
|
||||
verify Read objects concurrently from --endpoint and verify against manifest.csv.
|
||||
mixed Write and verify against the same endpoint.
|
||||
|
||||
Required:
|
||||
--endpoint S3 endpoint URL, for example http://127.0.0.1:9100
|
||||
--access-key S3 access key
|
||||
--secret-key S3 secret key
|
||||
|
||||
Core options:
|
||||
--bucket Bucket name (default: compat-rw-stress)
|
||||
--region Region (default: us-east-1)
|
||||
--concurrency Parallel object operations (default: 64)
|
||||
--out-dir Output directory (default: target/compat/rw-stress-<timestamp>)
|
||||
--work-dir Payload scratch directory (default: <out-dir>/payloads)
|
||||
--manifest Manifest path (default: <out-dir>/manifest.csv for write/mixed)
|
||||
--profile compact | 5g | 200g (default: 200g)
|
||||
--object-spec Override profile. Format: size:count,size:count
|
||||
Example: 1KiB:1024,1MiB:1024,64MiB:512,1GiB:64
|
||||
--data-pattern compressible | random | mixed (default: compressible)
|
||||
--keep-payloads Do not delete local payload files after upload
|
||||
--resume Skip write tasks that already have rows in <out-dir>/tasks/write-rows
|
||||
--dry-run Print planned tasks and commands without executing S3 operations
|
||||
--client mc | aws (default: mc)
|
||||
--mc-bin Path to mc binary (default: first tmp/mc.* or mc in PATH)
|
||||
--aws-bin Path to aws binary (used with --client aws)
|
||||
|
||||
Encryption options:
|
||||
--encryption none | sse-s3 | sse-kms | sse-c (default: none)
|
||||
--sse-kms-key-id KMS key id for --encryption sse-kms
|
||||
--sse-c-key-file Raw 32-byte SSE-C key file for --encryption sse-c
|
||||
|
||||
Examples:
|
||||
# Generate the old RustFS beta5 dataset.
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode write --endpoint http://127.0.0.1:9100 \
|
||||
--access-key rustfsadmin --secret-key rustfsadmin \
|
||||
--bucket compat-beta5 --concurrency 96
|
||||
|
||||
# Verify that dataset after mounting beta5 disks into the rio-v2 compose.
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode verify --endpoint http://127.0.0.1:9300 \
|
||||
--access-key rustfsadmin --secret-key rustfsadmin \
|
||||
--bucket compat-beta5 --concurrency 96 \
|
||||
--manifest target/compat/rw-stress-YYYYmmdd-HHMMSS/manifest.csv
|
||||
|
||||
# Faster smoke run.
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode mixed --endpoint http://127.0.0.1:9300 \
|
||||
--access-key rustfsadmin --secret-key rustfsadmin \
|
||||
--profile compact --concurrency 16
|
||||
USAGE
|
||||
}
|
||||
|
||||
die() {
|
||||
echo "ERROR: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
command -v "$1" >/dev/null 2>&1 || die "command not found: $1"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--mode) MODE="$2"; shift 2 ;;
|
||||
--endpoint) ENDPOINT="$2"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$2"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$2"; shift 2 ;;
|
||||
--bucket) BUCKET="$2"; shift 2 ;;
|
||||
--region) REGION="$2"; shift 2 ;;
|
||||
--concurrency) CONCURRENCY="$2"; shift 2 ;;
|
||||
--out-dir) OUT_DIR="$2"; shift 2 ;;
|
||||
--work-dir) WORK_DIR="$2"; shift 2 ;;
|
||||
--manifest) MANIFEST="$2"; shift 2 ;;
|
||||
--profile) PROFILE="$2"; shift 2 ;;
|
||||
--object-spec) OBJECT_SPEC="$2"; shift 2 ;;
|
||||
--data-pattern) DATA_PATTERN="$2"; shift 2 ;;
|
||||
--encryption) ENCRYPTION="$2"; shift 2 ;;
|
||||
--sse-kms-key-id) SSE_KMS_KEY_ID="$2"; shift 2 ;;
|
||||
--sse-c-key-file) SSE_C_KEY_FILE="$2"; shift 2 ;;
|
||||
--client) CLIENT="$2"; shift 2 ;;
|
||||
--mc-bin) MC_BIN="$2"; shift 2 ;;
|
||||
--aws-bin) AWS_BIN="$2"; shift 2 ;;
|
||||
--keep-payloads) KEEP_PAYLOADS=true; shift ;;
|
||||
--resume) RESUME=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) die "unknown arg: $1" ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
[[ "$MODE" =~ ^(write|verify|mixed)$ ]] || die "--mode must be write, verify, or mixed"
|
||||
[[ "$CLIENT" =~ ^(mc|aws)$ ]] || die "--client must be mc or aws"
|
||||
[[ "$PROFILE" =~ ^(compact|5g|200g)$ ]] || die "--profile must be compact, 5g, or 200g"
|
||||
[[ "$DATA_PATTERN" =~ ^(compressible|random|mixed)$ ]] || die "--data-pattern must be compressible, random, or mixed"
|
||||
[[ "$ENCRYPTION" =~ ^(none|sse-s3|sse-kms|sse-c)$ ]] || die "--encryption must be none, sse-s3, sse-kms, or sse-c"
|
||||
[[ -n "$ENDPOINT" && -n "$ACCESS_KEY" && -n "$SECRET_KEY" ]] || die "--endpoint/--access-key/--secret-key are required"
|
||||
[[ "$CONCURRENCY" =~ ^[0-9]+$ && "$CONCURRENCY" -gt 0 ]] || die "--concurrency must be a positive integer"
|
||||
if [[ "$ENCRYPTION" == "sse-kms" && -z "$SSE_KMS_KEY_ID" ]]; then
|
||||
die "--sse-kms-key-id is required for --encryption sse-kms"
|
||||
fi
|
||||
if [[ "$ENCRYPTION" == "sse-c" ]]; then
|
||||
[[ -n "$SSE_C_KEY_FILE" && -f "$SSE_C_KEY_FILE" ]] || die "--sse-c-key-file must point to an existing key file"
|
||||
fi
|
||||
if [[ "$MODE" == "verify" && -z "$MANIFEST" ]]; then
|
||||
die "--manifest is required for --mode verify"
|
||||
fi
|
||||
}
|
||||
|
||||
setup_paths() {
|
||||
if [[ -z "$OUT_DIR" ]]; then
|
||||
OUT_DIR="target/compat/rw-stress-$(date +%Y%m%d-%H%M%S)"
|
||||
fi
|
||||
if [[ -z "$WORK_DIR" ]]; then
|
||||
WORK_DIR="$OUT_DIR/payloads"
|
||||
fi
|
||||
if [[ -z "$MANIFEST" ]]; then
|
||||
MANIFEST="$OUT_DIR/manifest.csv"
|
||||
fi
|
||||
|
||||
mkdir -p "$OUT_DIR" "$WORK_DIR" "$OUT_DIR/tasks" "$OUT_DIR/logs"
|
||||
TASKS_FILE="$OUT_DIR/tasks/tasks.tsv"
|
||||
WRITE_ROWS_DIR="$OUT_DIR/tasks/write-rows"
|
||||
VERIFY_ROWS_DIR="$OUT_DIR/tasks/verify-rows"
|
||||
MC_CONFIG_DIR_LOCAL="$OUT_DIR/mc-config"
|
||||
MC_ALIAS="compat"
|
||||
mkdir -p "$WRITE_ROWS_DIR" "$VERIFY_ROWS_DIR"
|
||||
}
|
||||
|
||||
resolve_mc_bin() {
|
||||
if [[ -n "$MC_BIN" ]]; then
|
||||
echo "$MC_BIN"
|
||||
return
|
||||
fi
|
||||
|
||||
local candidate
|
||||
candidate="$(find tmp -maxdepth 1 -type f -name 'mc.*' -perm -111 2>/dev/null | sort | tail -n 1 || true)"
|
||||
if [[ -n "$candidate" ]]; then
|
||||
echo "$candidate"
|
||||
return
|
||||
fi
|
||||
|
||||
command -v mc 2>/dev/null || true
|
||||
}
|
||||
|
||||
size_to_bytes() {
|
||||
local raw="$1"
|
||||
local num unit
|
||||
if [[ "$raw" =~ ^([0-9]+)(B|KiB|MiB|GiB|KB|MB|GB)?$ ]]; then
|
||||
num="${BASH_REMATCH[1]}"
|
||||
unit="${BASH_REMATCH[2]:-B}"
|
||||
else
|
||||
die "invalid size: $raw"
|
||||
fi
|
||||
|
||||
case "$unit" in
|
||||
B) echo "$num" ;;
|
||||
KiB) echo $((num * 1024)) ;;
|
||||
MiB) echo $((num * 1024 * 1024)) ;;
|
||||
GiB) echo $((num * 1024 * 1024 * 1024)) ;;
|
||||
KB) echo $((num * 1000)) ;;
|
||||
MB) echo $((num * 1000 * 1000)) ;;
|
||||
GB) echo $((num * 1000 * 1000 * 1000)) ;;
|
||||
*) die "invalid size unit: $unit" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
profile_spec() {
|
||||
if [[ -n "$OBJECT_SPEC" ]]; then
|
||||
echo "$OBJECT_SPEC"
|
||||
return
|
||||
fi
|
||||
|
||||
case "$PROFILE" in
|
||||
compact)
|
||||
echo "1KiB:64,1MiB:64,16MiB:16,128MiB:4"
|
||||
;;
|
||||
5g)
|
||||
echo "1KiB:1024,1MiB:255,16MiB:64,64MiB:28,1GiB:2"
|
||||
;;
|
||||
200g)
|
||||
echo "1KiB:1024,1MiB:1024,64MiB:512,1GiB:64,8GiB:12,6GiB:1"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
content_for_index() {
|
||||
local index="$1"
|
||||
case $((index % 3)) in
|
||||
0) echo "txt|text/plain" ;;
|
||||
1) echo "json|application/json" ;;
|
||||
2) echo "bin|binary/octet-stream" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
generate_tasks() {
|
||||
local spec item size count bytes i content ext mime key seed index=0
|
||||
spec="$(profile_spec)"
|
||||
: > "$TASKS_FILE"
|
||||
|
||||
IFS=',' read -r -a items <<< "$spec"
|
||||
for item in "${items[@]}"; do
|
||||
item="${item//[[:space:]]/}"
|
||||
[[ -n "$item" ]] || continue
|
||||
[[ "$item" =~ ^([^:]+):([0-9]+)$ ]] || die "invalid object spec item: $item"
|
||||
size="${BASH_REMATCH[1]}"
|
||||
count="${BASH_REMATCH[2]}"
|
||||
bytes="$(size_to_bytes "$size")"
|
||||
|
||||
for ((i = 1; i <= count; i++)); do
|
||||
content="$(content_for_index "$index")"
|
||||
ext="${content%%|*}"
|
||||
mime="${content#*|}"
|
||||
key="rw-stress/${size}/obj-$(printf '%06d' "$i").${ext}"
|
||||
seed="${BUCKET}:${key}:${bytes}"
|
||||
printf '%s\t%s\t%s\t%s\t%s\n' "$key" "$size" "$bytes" "$mime" "$seed" >> "$TASKS_FILE"
|
||||
index=$((index + 1))
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
write_row_file_for_key() {
|
||||
local key="$1"
|
||||
echo "$WRITE_ROWS_DIR/${key//\//_}.csv"
|
||||
}
|
||||
|
||||
prepare_write_input() {
|
||||
WRITE_INPUT="$TASKS_FILE"
|
||||
if [[ "$RESUME" != "true" ]]; then
|
||||
return
|
||||
fi
|
||||
|
||||
WRITE_INPUT="$OUT_DIR/tasks/write-input.tsv"
|
||||
: > "$WRITE_INPUT"
|
||||
|
||||
local line key _size _bytes _mime _seed row_file
|
||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
||||
IFS=$'\t' read -r key _size _bytes _mime _seed <<< "$line"
|
||||
row_file="$(write_row_file_for_key "$key")"
|
||||
[[ -s "$row_file" ]] && continue
|
||||
printf '%s\n' "$line" >> "$WRITE_INPUT"
|
||||
done < "$TASKS_FILE"
|
||||
}
|
||||
|
||||
print_plan() {
|
||||
local total_objects total_bytes profile_label
|
||||
if [[ -f "$TASKS_FILE" ]]; then
|
||||
total_objects="$(wc -l < "$TASKS_FILE" | tr -d ' ')"
|
||||
total_bytes="$(awk -F '\t' '{sum += $3} END {print sum + 0}' "$TASKS_FILE")"
|
||||
profile_label="$(profile_spec)"
|
||||
else
|
||||
total_objects="$(awk 'END {count = NR - 1; if (count < 0) count = 0; print count}' "$MANIFEST")"
|
||||
total_bytes="$(awk -F ',' 'NR > 1 {sum += $3} END {print sum + 0}' "$MANIFEST")"
|
||||
profile_label="from manifest"
|
||||
fi
|
||||
|
||||
cat <<PLAN
|
||||
Mode: $MODE
|
||||
Endpoint: $ENDPOINT
|
||||
Bucket: $BUCKET
|
||||
Profile: $profile_label
|
||||
Objects: $total_objects
|
||||
Bytes: $total_bytes
|
||||
Concurrency: $CONCURRENCY
|
||||
Encryption: $ENCRYPTION
|
||||
Client: $CLIENT
|
||||
Manifest: $MANIFEST
|
||||
Out dir: $OUT_DIR
|
||||
Work dir: $WORK_DIR
|
||||
PLAN
|
||||
}
|
||||
|
||||
aws_base() {
|
||||
AWS_ACCESS_KEY_ID="$ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SECRET_KEY" AWS_DEFAULT_REGION="$REGION" \
|
||||
"$AWS_BIN" --endpoint-url "$ENDPOINT" "$@"
|
||||
}
|
||||
|
||||
mc_base() {
|
||||
"$MC_BIN" --config-dir "$MC_CONFIG_DIR_LOCAL" "$@"
|
||||
}
|
||||
|
||||
aws_cp_args() {
|
||||
case "$ENCRYPTION" in
|
||||
none) ;;
|
||||
sse-s3) printf '%s\n' "--sse" "AES256" ;;
|
||||
sse-kms) printf '%s\n' "--sse" "aws:kms" "--sse-kms-key-id" "$SSE_KMS_KEY_ID" ;;
|
||||
sse-c) printf '%s\n' "--sse-c" "AES256" "--sse-c-key" "fileb://$SSE_C_KEY_FILE" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
mc_enc_target() {
|
||||
printf '%s/%s/rw-stress/' "$MC_ALIAS" "$BUCKET"
|
||||
}
|
||||
|
||||
mc_cp_args() {
|
||||
local target
|
||||
target="$(mc_enc_target)"
|
||||
case "$ENCRYPTION" in
|
||||
none) ;;
|
||||
sse-s3) printf '%s\n' "--enc-s3" "$target" ;;
|
||||
sse-kms) printf '%s\n' "--enc-kms" "${target}=${SSE_KMS_KEY_ID}" ;;
|
||||
sse-c)
|
||||
local key_b64
|
||||
key_b64="$(base64 < "$SSE_C_KEY_FILE" | tr -d '\n')"
|
||||
printf '%s\n' "--enc-c" "${target}=${key_b64}"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
aws_get_args() {
|
||||
if [[ "$ENCRYPTION" == "sse-c" ]]; then
|
||||
printf '%s\n' "--sse-c" "AES256" "--sse-c-key" "fileb://$SSE_C_KEY_FILE"
|
||||
fi
|
||||
}
|
||||
|
||||
mc_get_args() {
|
||||
if [[ "$ENCRYPTION" == "sse-c" ]]; then
|
||||
local target key_b64
|
||||
target="$(mc_enc_target)"
|
||||
key_b64="$(base64 < "$SSE_C_KEY_FILE" | tr -d '\n')"
|
||||
printf '%s\n' "--enc-c" "${target}=${key_b64}"
|
||||
fi
|
||||
}
|
||||
|
||||
setup_mc_alias() {
|
||||
if [[ "$CLIENT" != "mc" || "$DRY_RUN" == "true" ]]; then
|
||||
return
|
||||
fi
|
||||
mkdir -p "$MC_CONFIG_DIR_LOCAL"
|
||||
mc_base alias set "$MC_ALIAS" "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY" --api S3v4 --path auto >/dev/null
|
||||
}
|
||||
|
||||
create_bucket_if_needed() {
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] create bucket if missing: $BUCKET"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$CLIENT" == "mc" ]]; then
|
||||
mc_base mb --ignore-existing --region "$REGION" "$MC_ALIAS/$BUCKET" >/dev/null
|
||||
return
|
||||
fi
|
||||
|
||||
if aws_base s3api head-bucket --bucket "$BUCKET" >/dev/null 2>&1; then
|
||||
return
|
||||
fi
|
||||
|
||||
aws_base s3api create-bucket --bucket "$BUCKET" >/dev/null
|
||||
}
|
||||
|
||||
payload_path_for_key() {
|
||||
local key="$1"
|
||||
echo "$WORK_DIR/${key//\//_}"
|
||||
}
|
||||
|
||||
generate_payload() {
|
||||
local file="$1"
|
||||
local bytes="$2"
|
||||
local seed="$3"
|
||||
local pattern="$DATA_PATTERN"
|
||||
mkdir -p "$(dirname "$file")"
|
||||
|
||||
if [[ "$pattern" == "mixed" ]]; then
|
||||
if [[ $((bytes % 2)) -eq 0 ]]; then
|
||||
pattern="compressible"
|
||||
else
|
||||
pattern="random"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$pattern" == "random" ]]; then
|
||||
head -c "$bytes" /dev/zero | openssl enc -aes-256-ctr -nosalt -pass "pass:$seed" -out "$file"
|
||||
else
|
||||
yes "$seed payload-for-rio-compatibility" | tr '\n' ' ' | head -c "$bytes" > "$file"
|
||||
fi
|
||||
}
|
||||
|
||||
sha256_file() {
|
||||
shasum -a 256 "$1" | awk '{print $1}'
|
||||
}
|
||||
|
||||
sha256_object() {
|
||||
local key="$1"
|
||||
shift
|
||||
if [[ "$CLIENT" == "mc" ]]; then
|
||||
mc_base cat "$@" "$MC_ALIAS/$BUCKET/$key" | shasum -a 256 | awk '{print $1}'
|
||||
else
|
||||
aws_base s3 cp "s3://$BUCKET/$key" - --no-progress "$@" | shasum -a 256 | awk '{print $1}'
|
||||
fi
|
||||
}
|
||||
|
||||
collect_args() {
|
||||
local generator="$1"
|
||||
extra_args=()
|
||||
while IFS= read -r arg; do
|
||||
extra_args+=("$arg")
|
||||
done < <("$generator")
|
||||
}
|
||||
|
||||
write_one() {
|
||||
local line="$1"
|
||||
local key size bytes mime seed file sha row_file
|
||||
local -a extra_args
|
||||
IFS=$'\t' read -r key size bytes mime seed <<< "$line"
|
||||
file="$(payload_path_for_key "$key")"
|
||||
row_file="$(write_row_file_for_key "$key")"
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] upload $bytes bytes to s3://$BUCKET/$key content-type=$mime"
|
||||
printf '%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "DRY_RUN" "$ENCRYPTION" > "$row_file"
|
||||
return
|
||||
fi
|
||||
|
||||
generate_payload "$file" "$bytes" "$seed"
|
||||
sha="$(sha256_file "$file")"
|
||||
|
||||
if [[ "$CLIENT" == "mc" ]]; then
|
||||
collect_args mc_cp_args
|
||||
mc_base cp --quiet --attr "Content-Type=$mime" ${extra_args[@]+"${extra_args[@]}"} "$file" "$MC_ALIAS/$BUCKET/$key" \
|
||||
> "$OUT_DIR/logs/${key//\//_}.put.log" 2>&1
|
||||
else
|
||||
collect_args aws_cp_args
|
||||
aws_base s3 cp "$file" "s3://$BUCKET/$key" --no-progress --content-type "$mime" ${extra_args[@]+"${extra_args[@]}"} \
|
||||
> "$OUT_DIR/logs/${key//\//_}.put.log" 2>&1
|
||||
fi
|
||||
|
||||
printf '%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$sha" "$ENCRYPTION" > "$row_file"
|
||||
if [[ "$KEEP_PAYLOADS" != "true" ]]; then
|
||||
rm -f "$file"
|
||||
fi
|
||||
}
|
||||
|
||||
verify_one() {
|
||||
local line="$1"
|
||||
local key size bytes mime expected encryption actual row_file
|
||||
local -a extra_args
|
||||
IFS=',' read -r key size bytes mime expected encryption <<< "$line"
|
||||
row_file="$VERIFY_ROWS_DIR/${key//\//_}.csv"
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] verify s3://$BUCKET/$key expected=$expected"
|
||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "DRY_RUN" "dry-run" > "$row_file"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$CLIENT" == "mc" ]]; then
|
||||
collect_args mc_get_args
|
||||
else
|
||||
collect_args aws_get_args
|
||||
fi
|
||||
if ! actual="$(sha256_object "$key" ${extra_args[@]+"${extra_args[@]}"})"; then
|
||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "ERROR" "download failed" > "$row_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ "$actual" == "$expected" ]]; then
|
||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "$actual" "ok" > "$row_file"
|
||||
else
|
||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "$actual" "sha256-mismatch" > "$row_file"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_parallel_tasks() {
|
||||
local action="$1"
|
||||
local input_file="$2"
|
||||
local failure_file="$OUT_DIR/tasks/${action}.failed"
|
||||
local line
|
||||
rm -f "$failure_file"
|
||||
|
||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
||||
while [[ "$(jobs -rp | wc -l | tr -d ' ')" -ge "$CONCURRENCY" ]]; do
|
||||
sleep 0.2
|
||||
done
|
||||
|
||||
({
|
||||
if [[ "$action" == "write" ]]; then
|
||||
write_one "$line"
|
||||
else
|
||||
verify_one "$line"
|
||||
fi
|
||||
} || touch "$failure_file") &
|
||||
done < "$input_file"
|
||||
|
||||
wait
|
||||
[[ ! -f "$failure_file" ]]
|
||||
}
|
||||
|
||||
combine_write_manifest() {
|
||||
echo "key,size,bytes,content_type,sha256,encryption" > "$MANIFEST"
|
||||
find "$WRITE_ROWS_DIR" -type f -name '*.csv' -print0 | sort -z | xargs -0 cat >> "$MANIFEST"
|
||||
|
||||
local expected actual
|
||||
expected="$(wc -l < "$TASKS_FILE" | tr -d ' ')"
|
||||
actual="$(( $(wc -l < "$MANIFEST" | tr -d ' ') - 1 ))"
|
||||
if [[ "$actual" -ne "$expected" ]]; then
|
||||
die "manifest row count mismatch: expected $expected, got $actual"
|
||||
fi
|
||||
}
|
||||
|
||||
prepare_verify_input() {
|
||||
VERIFY_INPUT="$OUT_DIR/tasks/verify-input.csv"
|
||||
tail -n +2 "$MANIFEST" > "$VERIFY_INPUT"
|
||||
}
|
||||
|
||||
combine_verify_summary() {
|
||||
VERIFY_SUMMARY="$OUT_DIR/verify-summary.csv"
|
||||
echo "key,size,bytes,content_type,expected_sha256,actual_sha256,status" > "$VERIFY_SUMMARY"
|
||||
find "$VERIFY_ROWS_DIR" -type f -name '*.csv' -print0 | sort -z | xargs -0 cat >> "$VERIFY_SUMMARY"
|
||||
|
||||
local failed
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "Verification dry run complete. Summary: $VERIFY_SUMMARY"
|
||||
return
|
||||
fi
|
||||
|
||||
failed="$(awk -F ',' 'NR > 1 && $7 != "ok" {count++} END {print count + 0}' "$VERIFY_SUMMARY")"
|
||||
local expected actual
|
||||
expected="$(wc -l < "$VERIFY_INPUT" | tr -d ' ')"
|
||||
actual="$(( $(wc -l < "$VERIFY_SUMMARY" | tr -d ' ') - 1 ))"
|
||||
if [[ "$actual" -ne "$expected" ]]; then
|
||||
echo "Verification row count mismatch: expected $expected, got $actual" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$failed" -ne 0 ]]; then
|
||||
echo "Verification failed: $failed object(s). See $VERIFY_SUMMARY" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "Verification passed. Summary: $VERIFY_SUMMARY"
|
||||
}
|
||||
|
||||
export_functions() {
|
||||
export ENDPOINT ACCESS_KEY SECRET_KEY BUCKET REGION ENCRYPTION SSE_KMS_KEY_ID SSE_C_KEY_FILE AWS_BIN
|
||||
export CLIENT MC_BIN MC_CONFIG_DIR_LOCAL MC_ALIAS
|
||||
export WORK_DIR OUT_DIR WRITE_ROWS_DIR VERIFY_ROWS_DIR DATA_PATTERN KEEP_PAYLOADS DRY_RUN
|
||||
export -f aws_base mc_base aws_cp_args aws_get_args mc_enc_target mc_cp_args mc_get_args payload_path_for_key generate_payload sha256_file sha256_object write_one verify_one
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
setup_paths
|
||||
|
||||
if [[ "$DRY_RUN" != "true" ]]; then
|
||||
if [[ "$CLIENT" == "mc" ]]; then
|
||||
MC_BIN="$(resolve_mc_bin)"
|
||||
[[ -n "$MC_BIN" ]] || die "mc binary not found; pass --mc-bin or put mc in PATH"
|
||||
require_cmd "$MC_BIN"
|
||||
else
|
||||
require_cmd "$AWS_BIN"
|
||||
fi
|
||||
require_cmd shasum
|
||||
require_cmd head
|
||||
require_cmd yes
|
||||
require_cmd openssl
|
||||
elif [[ "$CLIENT" == "mc" ]]; then
|
||||
MC_BIN="$(resolve_mc_bin)"
|
||||
fi
|
||||
|
||||
setup_mc_alias
|
||||
|
||||
if [[ "$MODE" == "write" || "$MODE" == "mixed" ]]; then
|
||||
local write_failed=false
|
||||
generate_tasks
|
||||
prepare_write_input
|
||||
print_plan
|
||||
create_bucket_if_needed
|
||||
export_functions
|
||||
if ! run_parallel_tasks write "$WRITE_INPUT"; then
|
||||
write_failed=true
|
||||
fi
|
||||
combine_write_manifest
|
||||
echo "Write manifest: $MANIFEST"
|
||||
if [[ "$write_failed" == "true" ]]; then
|
||||
die "one or more write tasks failed; see $OUT_DIR/logs"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$MODE" == "verify" || "$MODE" == "mixed" ]]; then
|
||||
local verify_failed=false
|
||||
[[ -f "$MANIFEST" ]] || die "manifest not found: $MANIFEST"
|
||||
if [[ "$MODE" == "verify" ]]; then
|
||||
cp "$MANIFEST" "$OUT_DIR/manifest.csv"
|
||||
MANIFEST="$OUT_DIR/manifest.csv"
|
||||
fi
|
||||
prepare_verify_input
|
||||
print_plan
|
||||
export_functions
|
||||
if ! run_parallel_tasks verify "$VERIFY_INPUT"; then
|
||||
verify_failed=true
|
||||
fi
|
||||
combine_verify_summary
|
||||
if [[ "$verify_failed" == "true" ]]; then
|
||||
die "one or more verify tasks failed; see $VERIFY_SUMMARY"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -1,52 +0,0 @@
|
||||
services:
|
||||
rustfs:
|
||||
image: rustfs/rustfs:1.0.0-alpha.99-glibc
|
||||
container_name: rustfs-issue-2715-test
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
ports:
|
||||
- "19000:9000"
|
||||
- "19001:9001"
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs{0...8}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=admin
|
||||
- RUSTFS_SECRET_KEY=admin
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=info
|
||||
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
|
||||
- RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
|
||||
- RUSTFS_STORAGE_CLASS_STANDARD=EC:2
|
||||
- RUSTFS_STORAGE_CLASS_RRS=EC:1
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
|
||||
- RUSTFS_OBS_LOG_DIRECTORY=/opt/rustfs/logs
|
||||
extra_hosts:
|
||||
- "otel-collector:host-gateway"
|
||||
- "pyroscope:host-gateway"
|
||||
volumes:
|
||||
- ./deploy/data/issue-2715/rustfs0:/data/rustfs0
|
||||
- ./deploy/data/issue-2715/rustfs1:/data/rustfs1
|
||||
- ./deploy/data/issue-2715/rustfs2:/data/rustfs2
|
||||
- ./deploy/data/issue-2715/rustfs3:/data/rustfs3
|
||||
- ./deploy/data/issue-2715/rustfs4:/data/rustfs4
|
||||
- ./deploy/data/issue-2715/rustfs5:/data/rustfs5
|
||||
- ./deploy/data/issue-2715/rustfs6:/data/rustfs6
|
||||
- ./deploy/data/issue-2715/rustfs7:/data/rustfs7
|
||||
- ./deploy/data/issue-2715/rustfs8:/data/rustfs8
|
||||
- ./deploy/logs/issue-2715:/opt/rustfs/logs
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"sh",
|
||||
"-c",
|
||||
"curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
|
||||
]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
@@ -1 +0,0 @@
|
||||
data/
|
||||
@@ -1,106 +0,0 @@
|
||||
# Issue 2815 Local Docker Verification
|
||||
|
||||
## Purpose
|
||||
|
||||
This directory contains the local distributed Docker verification assets used to validate issue `#2815` against the current source build.
|
||||
|
||||
The target behavior is:
|
||||
|
||||
- 4-node distributed cluster starts successfully
|
||||
- `/health/ready` becomes reachable on each node
|
||||
- logs no longer contain `storage_info failed: Io error: wrong msgpack marker FixArray(1)`
|
||||
- internode RPC authentication succeeds with an explicit non-default RPC secret
|
||||
|
||||
## Files
|
||||
|
||||
- `docker-compose.yml`: 4-node distributed cluster using a locally built image
|
||||
|
||||
## Data Directories
|
||||
|
||||
Create the bind-mount directories before `docker compose up`:
|
||||
|
||||
```bash
|
||||
mkdir -p .docker/test/issues-2815/data/rustfs{1..4}-disk{0..3}
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
Apple Silicon / arm64 host:
|
||||
|
||||
```bash
|
||||
docker build --platform linux/arm64 -f Dockerfile.source -t rustfs-issue-2815-local .
|
||||
```
|
||||
|
||||
If you intentionally want amd64 emulation:
|
||||
|
||||
```bash
|
||||
docker build --platform linux/amd64 -f Dockerfile.source -t rustfs-issue-2815-local .
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
If the image platform is not `linux/arm64`, align compose explicitly:
|
||||
|
||||
```bash
|
||||
RUSTFS_DOCKER_PLATFORM=linux/amd64 docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
## Health Checks
|
||||
|
||||
Container-level healthcheck is now included and probes:
|
||||
|
||||
```bash
|
||||
curl -fsS http://127.0.0.1:9000/health
|
||||
```
|
||||
|
||||
Manual checks:
|
||||
|
||||
```bash
|
||||
curl -i http://127.0.0.1:9101/health/ready
|
||||
curl -i http://127.0.0.1:9102/health/ready
|
||||
curl -i http://127.0.0.1:9103/health/ready
|
||||
curl -i http://127.0.0.1:9104/health/ready
|
||||
```
|
||||
|
||||
## RPC Secret Requirement
|
||||
|
||||
The current source build no longer reproduces the original `FixArray(1)` decode error from issue `#2815`.
|
||||
|
||||
Earlier local Docker attempts failed during erasure bootstrap with:
|
||||
|
||||
```text
|
||||
No valid auth token
|
||||
store init failed to load formats after 10 retries: erasure read quorum
|
||||
```
|
||||
|
||||
Root cause:
|
||||
|
||||
- RPC authentication rejects the default secret `rustfsadmin`
|
||||
- distributed local Docker validation therefore needs an explicit non-default secret
|
||||
|
||||
This compose now sets both:
|
||||
|
||||
- `RUSTFS_SECRET_KEY=issue-2815-secret`
|
||||
- `RUSTFS_RPC_SECRET=issue-2815-rpc-secret`
|
||||
|
||||
With those values in place, the current 4-node local Docker cluster reaches healthy state and `/health/ready` returns `200`.
|
||||
|
||||
In other words:
|
||||
|
||||
- `RUSTFS_ACCESS_KEY` may still be `rustfsadmin` for local service credentials if desired
|
||||
- `RUSTFS_SECRET_KEY` can still be used for service credentials
|
||||
- but RPC authentication must not resolve to the default secret value `rustfsadmin`
|
||||
- if `RUSTFS_RPC_SECRET` is unset, the code falls back to `RUSTFS_SECRET_KEY`
|
||||
- so at least one of them must provide a non-default shared secret for internode RPC signing
|
||||
|
||||
## Suggested Debug Commands
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/issues-2815/docker-compose.yml ps
|
||||
docker compose -f .docker/test/issues-2815/docker-compose.yml logs --no-color --tail=200
|
||||
docker compose -f .docker/test/issues-2815/docker-compose.yml down -v
|
||||
```
|
||||
@@ -1,120 +0,0 @@
|
||||
services:
|
||||
rustfs1:
|
||||
image: rustfs-issue-2815-local
|
||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
|
||||
hostname: rustfs1
|
||||
container_name: rustfs-issue-2815-rustfs1
|
||||
environment:
|
||||
RUSTFS_ADDRESS: "0.0.0.0:9000"
|
||||
RUSTFS_ACCESS_KEY: "rustfsadmin"
|
||||
RUSTFS_SECRET_KEY: "issue-2815-secret"
|
||||
RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
|
||||
RUSTFS_CONSOLE_ENABLE: "false"
|
||||
RUST_LOG: "info"
|
||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
|
||||
RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
|
||||
volumes:
|
||||
- ./data/rustfs1-disk0:/data/rustfs0
|
||||
- ./data/rustfs1-disk1:/data/rustfs1
|
||||
- ./data/rustfs1-disk2:/data/rustfs2
|
||||
- ./data/rustfs1-disk3:/data/rustfs3
|
||||
networks: [rustfs-issue-2815-net]
|
||||
ports:
|
||||
- "9101:9000"
|
||||
healthcheck:
|
||||
test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 8
|
||||
start_period: 30s
|
||||
|
||||
rustfs2:
|
||||
image: rustfs-issue-2815-local
|
||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
|
||||
hostname: rustfs2
|
||||
container_name: rustfs-issue-2815-rustfs2
|
||||
environment:
|
||||
RUSTFS_ADDRESS: "0.0.0.0:9000"
|
||||
RUSTFS_ACCESS_KEY: "rustfsadmin"
|
||||
RUSTFS_SECRET_KEY: "issue-2815-secret"
|
||||
RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
|
||||
RUSTFS_CONSOLE_ENABLE: "false"
|
||||
RUST_LOG: "info"
|
||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
|
||||
RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
|
||||
volumes:
|
||||
- ./data/rustfs2-disk0:/data/rustfs0
|
||||
- ./data/rustfs2-disk1:/data/rustfs1
|
||||
- ./data/rustfs2-disk2:/data/rustfs2
|
||||
- ./data/rustfs2-disk3:/data/rustfs3
|
||||
networks: [rustfs-issue-2815-net]
|
||||
ports:
|
||||
- "9102:9000"
|
||||
healthcheck:
|
||||
test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 8
|
||||
start_period: 30s
|
||||
|
||||
rustfs3:
|
||||
image: rustfs-issue-2815-local
|
||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
|
||||
hostname: rustfs3
|
||||
container_name: rustfs-issue-2815-rustfs3
|
||||
environment:
|
||||
RUSTFS_ADDRESS: "0.0.0.0:9000"
|
||||
RUSTFS_ACCESS_KEY: "rustfsadmin"
|
||||
RUSTFS_SECRET_KEY: "issue-2815-secret"
|
||||
RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
|
||||
RUSTFS_CONSOLE_ENABLE: "false"
|
||||
RUST_LOG: "info"
|
||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
|
||||
RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
|
||||
volumes:
|
||||
- ./data/rustfs3-disk0:/data/rustfs0
|
||||
- ./data/rustfs3-disk1:/data/rustfs1
|
||||
- ./data/rustfs3-disk2:/data/rustfs2
|
||||
- ./data/rustfs3-disk3:/data/rustfs3
|
||||
networks: [rustfs-issue-2815-net]
|
||||
ports:
|
||||
- "9103:9000"
|
||||
healthcheck:
|
||||
test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 8
|
||||
start_period: 30s
|
||||
|
||||
rustfs4:
|
||||
image: rustfs-issue-2815-local
|
||||
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
|
||||
hostname: rustfs4
|
||||
container_name: rustfs-issue-2815-rustfs4
|
||||
environment:
|
||||
RUSTFS_ADDRESS: "0.0.0.0:9000"
|
||||
RUSTFS_ACCESS_KEY: "rustfsadmin"
|
||||
RUSTFS_SECRET_KEY: "issue-2815-secret"
|
||||
RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
|
||||
RUSTFS_CONSOLE_ENABLE: "false"
|
||||
RUST_LOG: "info"
|
||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
|
||||
RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
|
||||
volumes:
|
||||
- ./data/rustfs4-disk0:/data/rustfs0
|
||||
- ./data/rustfs4-disk1:/data/rustfs1
|
||||
- ./data/rustfs4-disk2:/data/rustfs2
|
||||
- ./data/rustfs4-disk3:/data/rustfs3
|
||||
networks: [rustfs-issue-2815-net]
|
||||
ports:
|
||||
- "9104:9000"
|
||||
healthcheck:
|
||||
test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 8
|
||||
start_period: 30s
|
||||
|
||||
networks:
|
||||
rustfs-issue-2815-net:
|
||||
name: rustfs-issue-2815-net
|
||||
@@ -1,183 +0,0 @@
|
||||
# Site Replication Docker Compose Test
|
||||
|
||||
## Purpose
|
||||
|
||||
This directory contains a local three-site RustFS site replication check. It is intended to verify the admin site-replication flow against real containers:
|
||||
|
||||
- three independent RustFS sites start successfully
|
||||
- the MinIO-compatible `mc admin replicate add` command configures all three sites
|
||||
- a bucket and object written to site 1 are replicated to site 2 and site 3
|
||||
- `mc admin replicate status` can read the resulting site-replication state
|
||||
|
||||
The compose file uses named volumes so the test does not require preparing host bind-mount directories.
|
||||
Because Docker named volumes usually share the same physical device in local desktop environments, this test compose defaults `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true`. Keep that setting limited to local test and CI environments.
|
||||
|
||||
## Files
|
||||
|
||||
- `docker-compose.yml`: three RustFS sites, a volume permission helper, and a one-shot setup/check container
|
||||
- `run-object-flow-check.sh`: host-side upload/download verification for replicated 10 MiB to 100 MiB objects
|
||||
|
||||
## Ports
|
||||
|
||||
Default host endpoints:
|
||||
|
||||
- Site 1 API: `http://127.0.0.1:9000`
|
||||
- Site 1 Console: `http://127.0.0.1:9001`
|
||||
- Site 2 API: `http://127.0.0.1:9010`
|
||||
- Site 2 Console: `http://127.0.0.1:9011`
|
||||
- Site 3 API: `http://127.0.0.1:9020`
|
||||
- Site 3 Console: `http://127.0.0.1:9021`
|
||||
|
||||
Default credentials are `rustfsadmin` / `rustfsadmin`. These are for local testing only.
|
||||
|
||||
## Run
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
To test a locally built image instead of Docker Hub `rustfs/rustfs:latest`, set `RUSTFS_SITE_REPL_IMAGE`:
|
||||
|
||||
```bash
|
||||
docker build -f Dockerfile.source -t rustfs-site-repl-local:latest .
|
||||
RUSTFS_SITE_REPL_IMAGE=rustfs-site-repl-local:latest \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
For a detached run:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up -d
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml logs -f site-replication-setup
|
||||
```
|
||||
|
||||
## Test Flow
|
||||
|
||||
The compose stack performs these steps:
|
||||
|
||||
1. `site-replication-volume-permission-helper` fixes ownership on all named volumes for the RustFS runtime user.
|
||||
2. `rustfs-site1`, `rustfs-site2`, and `rustfs-site3` start as separate RustFS sites.
|
||||
3. Each site exposes its S3 API and Console on a unique host port.
|
||||
4. Health checks wait for `/health/ready` on each RustFS container.
|
||||
5. `site-replication-setup` configures `mc` aliases for all three sites.
|
||||
6. The setup container waits until `mc admin info` succeeds for all sites.
|
||||
7. It runs:
|
||||
|
||||
```bash
|
||||
mc admin replicate add site1 site2 site3
|
||||
```
|
||||
|
||||
8. It creates the test bucket on site 1 and uploads `from-site1.txt`.
|
||||
9. It polls site 2 and site 3 until the replicated object is visible.
|
||||
10. It prints `mc admin replicate status site1`.
|
||||
|
||||
The setup container exits with status `0` only after the object replication check passes.
|
||||
|
||||
## Object Flow Check
|
||||
|
||||
After the compose setup succeeds, run the larger object flow check from the repository root:
|
||||
|
||||
```bash
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
The script creates five local files and uploads them from different sites:
|
||||
|
||||
- 10 MiB from site 1
|
||||
- 25 MiB from site 2
|
||||
- 50 MiB from site 3
|
||||
- 75 MiB from site 1
|
||||
- 100 MiB from site 2
|
||||
|
||||
For each uploaded object, the script waits for replication to the other two sites, downloads the object from those sites, and verifies both byte size and SHA-256 checksum. It uses a temporary `mc` config directory, so it does not overwrite existing host aliases.
|
||||
|
||||
The default bucket is `site-repl-flow-check`. Override it when needed:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_FLOW_BUCKET='site-repl-large-flow' \
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
The script keeps uploaded objects under a timestamped prefix. Override the prefix for repeatable runs:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_FLOW_PREFIX='manual-check-001' \
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
If replication is slow on the local machine, increase polling:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_WAIT_ATTEMPTS=180 \
|
||||
RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS=2 \
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
## Optional Settings
|
||||
|
||||
Override local test credentials:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_ACCESS_KEY='localadmin' \
|
||||
RUSTFS_SITE_REPL_SECRET_KEY='localadmin-secret' \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
Use a different test bucket:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_BUCKET='site-repl-check' \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
Enable ILM expiry rule replication during site setup:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=true \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
Use this only when the test needs lifecycle expiry metadata included in site replication.
|
||||
|
||||
## Manual Checks
|
||||
|
||||
After the setup container succeeds, you can inspect the sites with `mc` from the host:
|
||||
|
||||
```bash
|
||||
mc alias set site1 http://127.0.0.1:9000 rustfsadmin rustfsadmin
|
||||
mc alias set site2 http://127.0.0.1:9010 rustfsadmin rustfsadmin
|
||||
mc alias set site3 http://127.0.0.1:9020 rustfsadmin rustfsadmin
|
||||
|
||||
mc admin replicate info site1
|
||||
mc admin replicate status site1
|
||||
mc stat site2/site-repl-demo/from-site1.txt
|
||||
mc stat site3/site-repl-demo/from-site1.txt
|
||||
```
|
||||
|
||||
After larger object flow checks, replication should converge without a growing queue:
|
||||
|
||||
```bash
|
||||
mc admin replicate status site1
|
||||
mc admin replicate status site2
|
||||
mc admin replicate status site3
|
||||
```
|
||||
|
||||
Useful Docker commands:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml ps
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color --tail=200
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color site-replication-setup
|
||||
```
|
||||
|
||||
## Cleanup
|
||||
|
||||
Remove containers and named volumes:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml down -v
|
||||
```
|
||||
|
||||
Use `down -v` before rerunning the full setup from scratch. Site replication state is persisted in the named volumes, so rerunning without deleting volumes may attempt to add an already-configured replication topology.
|
||||
@@ -1,243 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
services:
|
||||
rustfs-site1:
|
||||
image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
|
||||
container_name: rustfs-site-repl-1
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
ports:
|
||||
- "9000:9000"
|
||||
- "9001:9001"
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
volumes:
|
||||
- site1_data_0:/data/rustfs0
|
||||
- site1_data_1:/data/rustfs1
|
||||
- site1_data_2:/data/rustfs2
|
||||
- site1_data_3:/data/rustfs3
|
||||
- site1_logs:/app/logs
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
depends_on:
|
||||
site-replication-volume-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
|
||||
rustfs-site2:
|
||||
image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
|
||||
container_name: rustfs-site-repl-2
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
ports:
|
||||
- "9010:9000"
|
||||
- "9011:9001"
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
volumes:
|
||||
- site2_data_0:/data/rustfs0
|
||||
- site2_data_1:/data/rustfs1
|
||||
- site2_data_2:/data/rustfs2
|
||||
- site2_data_3:/data/rustfs3
|
||||
- site2_logs:/app/logs
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
depends_on:
|
||||
site-replication-volume-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
|
||||
rustfs-site3:
|
||||
image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
|
||||
container_name: rustfs-site-repl-3
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
ports:
|
||||
- "9020:9000"
|
||||
- "9021:9001"
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
volumes:
|
||||
- site3_data_0:/data/rustfs0
|
||||
- site3_data_1:/data/rustfs1
|
||||
- site3_data_2:/data/rustfs2
|
||||
- site3_data_3:/data/rustfs3
|
||||
- site3_logs:/app/logs
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
depends_on:
|
||||
site-replication-volume-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
|
||||
site-replication-setup:
|
||||
image: minio/mc:latest
|
||||
container_name: rustfs-site-repl-setup
|
||||
depends_on:
|
||||
rustfs-site1:
|
||||
condition: service_healthy
|
||||
rustfs-site2:
|
||||
condition: service_healthy
|
||||
rustfs-site3:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- RUSTFS_SITE_REPL_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SITE_REPL_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_SITE_REPL_BUCKET=${RUSTFS_SITE_REPL_BUCKET:-site-repl-demo}
|
||||
- RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY:-false}
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
command:
|
||||
- |
|
||||
set -eu
|
||||
|
||||
access_key="$${RUSTFS_SITE_REPL_ACCESS_KEY}"
|
||||
secret_key="$${RUSTFS_SITE_REPL_SECRET_KEY}"
|
||||
bucket="$${RUSTFS_SITE_REPL_BUCKET}"
|
||||
|
||||
mc alias set site1 http://rustfs-site1:9000 "$${access_key}" "$${secret_key}"
|
||||
mc alias set site2 http://rustfs-site2:9000 "$${access_key}" "$${secret_key}"
|
||||
mc alias set site3 http://rustfs-site3:9000 "$${access_key}" "$${secret_key}"
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
until mc ls "$${site}" >/dev/null 2>&1; do
|
||||
echo "waiting for $${site} S3 API"
|
||||
sleep 2
|
||||
done
|
||||
done
|
||||
|
||||
ilm_flag=""
|
||||
if [ "$${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY}" = "true" ]; then
|
||||
ilm_flag="--replicate-ilm-expiry"
|
||||
fi
|
||||
|
||||
echo "configuring 3-site replication"
|
||||
if ! mc admin replicate add site1 site2 site3 $${ilm_flag}; then
|
||||
echo "replicate add failed; showing current replication info before exiting"
|
||||
mc admin replicate info site1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mc mb --ignore-existing "site1/$${bucket}"
|
||||
printf 'rustfs 3-site replication check\n' > /tmp/site-replication-check.txt
|
||||
mc cp /tmp/site-replication-check.txt "site1/$${bucket}/from-site1.txt"
|
||||
|
||||
for site in site2 site3; do
|
||||
for attempt in $$(seq 1 60); do
|
||||
if mc stat "$${site}/$${bucket}/from-site1.txt" >/dev/null 2>&1; then
|
||||
echo "$${site} received replicated object"
|
||||
break
|
||||
fi
|
||||
if [ "$${attempt}" = "60" ]; then
|
||||
echo "$${site} did not receive replicated object in time"
|
||||
exit 1
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
done
|
||||
|
||||
mc admin replicate status site1
|
||||
echo "3-site replication example is ready"
|
||||
restart: "no"
|
||||
|
||||
site-replication-volume-permission-helper:
|
||||
image: alpine:3.23.4
|
||||
volumes:
|
||||
- site1_data_0:/site1/data0
|
||||
- site1_data_1:/site1/data1
|
||||
- site1_data_2:/site1/data2
|
||||
- site1_data_3:/site1/data3
|
||||
- site1_logs:/site1/logs
|
||||
- site2_data_0:/site2/data0
|
||||
- site2_data_1:/site2/data1
|
||||
- site2_data_2:/site2/data2
|
||||
- site2_data_3:/site2/data3
|
||||
- site2_logs:/site2/logs
|
||||
- site3_data_0:/site3/data0
|
||||
- site3_data_1:/site3/data1
|
||||
- site3_data_2:/site3/data2
|
||||
- site3_data_3:/site3/data3
|
||||
- site3_logs:/site3/logs
|
||||
command: >
|
||||
sh -c "
|
||||
chown -R 10001:10001 /site1 /site2 /site3 &&
|
||||
echo 'site replication volume permissions fixed'
|
||||
"
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: "no"
|
||||
|
||||
networks:
|
||||
rustfs-site-replication:
|
||||
|
||||
volumes:
|
||||
site1_data_0:
|
||||
site1_data_1:
|
||||
site1_data_2:
|
||||
site1_data_3:
|
||||
site1_logs:
|
||||
site2_data_0:
|
||||
site2_data_1:
|
||||
site2_data_2:
|
||||
site2_data_3:
|
||||
site2_logs:
|
||||
site3_data_0:
|
||||
site3_data_1:
|
||||
site3_data_2:
|
||||
site3_data_3:
|
||||
site3_logs:
|
||||
@@ -1,191 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
set -eu
|
||||
|
||||
ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}"
|
||||
SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}"
|
||||
BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}"
|
||||
PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}"
|
||||
WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}"
|
||||
WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}"
|
||||
|
||||
SITE1_ENDPOINT="${RUSTFS_SITE1_ENDPOINT:-http://127.0.0.1:9000}"
|
||||
SITE2_ENDPOINT="${RUSTFS_SITE2_ENDPOINT:-http://127.0.0.1:9010}"
|
||||
SITE3_ENDPOINT="${RUSTFS_SITE3_ENDPOINT:-http://127.0.0.1:9020}"
|
||||
|
||||
command_exists() {
|
||||
command -v "$1" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
require_command() {
|
||||
if ! command_exists "$1"; then
|
||||
echo "missing required command: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
checksum_file() {
|
||||
if command_exists sha256sum; then
|
||||
sha256sum "$1" | awk '{print $1}'
|
||||
elif command_exists shasum; then
|
||||
shasum -a 256 "$1" | awk '{print $1}'
|
||||
else
|
||||
echo "missing required command: sha256sum or shasum" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
site_endpoint() {
|
||||
case "$1" in
|
||||
site1) printf '%s\n' "$SITE1_ENDPOINT" ;;
|
||||
site2) printf '%s\n' "$SITE2_ENDPOINT" ;;
|
||||
site3) printf '%s\n' "$SITE3_ENDPOINT" ;;
|
||||
*) echo "unknown site alias: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
other_sites() {
|
||||
case "$1" in
|
||||
site1) printf '%s\n' "site2 site3" ;;
|
||||
site2) printf '%s\n' "site1 site3" ;;
|
||||
site3) printf '%s\n' "site1 site2" ;;
|
||||
*) echo "unknown source site: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
wait_for_object() {
|
||||
site="$1"
|
||||
object="$2"
|
||||
attempt=1
|
||||
|
||||
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
|
||||
if mc stat "$site/$BUCKET/$object" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep "$WAIT_SLEEP_SECONDS"
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
echo "object was not replicated in time: $site/$BUCKET/$object" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_for_bucket() {
|
||||
site="$1"
|
||||
attempt=1
|
||||
|
||||
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
|
||||
if mc stat "$site/$BUCKET" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep "$WAIT_SLEEP_SECONDS"
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
echo "bucket was not replicated in time: $site/$BUCKET" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
require_command mc
|
||||
require_command dd
|
||||
require_command awk
|
||||
require_command date
|
||||
require_command mktemp
|
||||
require_command tr
|
||||
require_command wc
|
||||
|
||||
WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/rustfs-site-repl-flow.XXXXXX")"
|
||||
MC_CONFIG_DIR="$WORK_DIR/mc"
|
||||
export MC_CONFIG_DIR
|
||||
|
||||
cleanup() {
|
||||
rm -rf "$WORK_DIR"
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
mkdir -p "$MC_CONFIG_DIR" "$WORK_DIR/src" "$WORK_DIR/downloads"
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
mc alias set "$site" "$(site_endpoint "$site")" "$ACCESS_KEY" "$SECRET_KEY" >/dev/null
|
||||
done
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
echo "checking S3 API for $site"
|
||||
mc ls "$site" >/dev/null
|
||||
done
|
||||
|
||||
echo "ensuring bucket exists on site1: $BUCKET"
|
||||
mc mb --ignore-existing "site1/$BUCKET" >/dev/null
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
wait_for_bucket "$site"
|
||||
done
|
||||
|
||||
cat <<'EOF' | while read -r size_mb source_site object_name; do
|
||||
10 site1 object-010m.bin
|
||||
25 site2 object-025m.bin
|
||||
50 site3 object-050m.bin
|
||||
75 site1 object-075m.bin
|
||||
100 site2 object-100m.bin
|
||||
EOF
|
||||
src_file="$WORK_DIR/src/$object_name"
|
||||
object="$PREFIX/$object_name"
|
||||
|
||||
echo "creating ${size_mb}MiB file: $object_name"
|
||||
dd if=/dev/urandom of="$src_file" bs=1048576 count="$size_mb" >/dev/null 2>&1
|
||||
src_checksum="$(checksum_file "$src_file")"
|
||||
src_bytes="$(wc -c < "$src_file" | tr -d ' ')"
|
||||
|
||||
echo "uploading $object_name to $source_site ($src_bytes bytes)"
|
||||
mc cp "$src_file" "$source_site/$BUCKET/$object" >/dev/null
|
||||
|
||||
for site in $(other_sites "$source_site"); do
|
||||
wait_for_object "$site" "$object"
|
||||
|
||||
dst_file="$WORK_DIR/downloads/$site-$object_name"
|
||||
verified=false
|
||||
attempt=1
|
||||
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
|
||||
rm -f "$dst_file"
|
||||
echo "downloading $object_name from $site"
|
||||
if mc cp "$site/$BUCKET/$object" "$dst_file" >/dev/null 2>&1; then
|
||||
dst_checksum="$(checksum_file "$dst_file")"
|
||||
dst_bytes="$(wc -c < "$dst_file" | tr -d ' ')"
|
||||
|
||||
if [ "$dst_checksum" = "$src_checksum" ] && [ "$dst_bytes" = "$src_bytes" ]; then
|
||||
verified=true
|
||||
break
|
||||
fi
|
||||
fi
|
||||
|
||||
sleep "$WAIT_SLEEP_SECONDS"
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
if [ "$verified" != "true" ]; then
|
||||
echo "download verification failed for $site/$BUCKET/$object" >&2
|
||||
echo "expected checksum: $src_checksum" >&2
|
||||
echo "expected bytes: $src_bytes" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo "verified replicated downloads for $object_name"
|
||||
done
|
||||
|
||||
echo "site replication object flow check passed"
|
||||
echo "bucket: $BUCKET"
|
||||
echo "prefix: $PREFIX"
|
||||
@@ -1,4 +1 @@
|
||||
target
|
||||
.docker/compat/data
|
||||
.docker/compat/kms
|
||||
target/compat
|
||||
|
||||
@@ -1,25 +0,0 @@
|
||||
# GitHub Workflow Instructions
|
||||
|
||||
Applies to `.github/` and repository pull-request operations.
|
||||
|
||||
## Pull Requests
|
||||
|
||||
- PR titles and descriptions must be in English.
|
||||
- Use `.github/pull_request_template.md` for every PR body.
|
||||
- Keep all template section headings.
|
||||
- Use `N/A` for non-applicable sections.
|
||||
- Include verification commands in the PR details.
|
||||
- For `gh pr create` and `gh pr edit`, always write markdown body to a file and pass `--body-file`.
|
||||
- Do not use multiline inline `--body`; backticks and shell expansion can corrupt content or trigger unintended commands.
|
||||
- Recommended pattern:
|
||||
- `cat > /tmp/pr_body.md <<'EOF'`
|
||||
- `...markdown...`
|
||||
- `EOF`
|
||||
- `gh pr create ... --body-file /tmp/pr_body.md`
|
||||
|
||||
## CI Alignment
|
||||
|
||||
When changing CI-sensitive behavior, keep local validation aligned with
|
||||
`.github/workflows/ci.yml`. Read the workflow file directly for the current
|
||||
gate steps — do not rely on (or add) a copied command list here; copies go
|
||||
stale. `make pre-pr` is the local equivalent of the main gate.
|
||||
@@ -52,25 +52,27 @@ runs:
|
||||
sudo apt-get install -y \
|
||||
musl-tools \
|
||||
build-essential \
|
||||
lld \
|
||||
libdbus-1-dev \
|
||||
libwayland-dev \
|
||||
libwebkit2gtk-4.1-dev \
|
||||
libxdo-dev \
|
||||
pkg-config \
|
||||
libssl-dev \
|
||||
ripgrep \
|
||||
unzip \
|
||||
protobuf-compiler
|
||||
libssl-dev
|
||||
|
||||
- name: Install protoc
|
||||
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
|
||||
uses: arduino/setup-protoc@v3
|
||||
with:
|
||||
version: "34.1"
|
||||
repo-token: ${{ github.token }}
|
||||
version: "31.1"
|
||||
repo-token: ${{ inputs.github-token }}
|
||||
|
||||
- name: Install flatc
|
||||
uses: Nugine/setup-flatc@e7855e994773ce90094a3f1626d4afc9080c23ae # v1
|
||||
uses: Nugine/setup-flatc@v1
|
||||
with:
|
||||
version: "25.12.19"
|
||||
version: "25.2.10"
|
||||
|
||||
- name: Install Rust toolchain
|
||||
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
|
||||
uses: dtolnay/rust-toolchain@stable
|
||||
with:
|
||||
toolchain: ${{ inputs.rust-version }}
|
||||
targets: ${{ inputs.target }}
|
||||
@@ -78,17 +80,17 @@ runs:
|
||||
|
||||
- name: Install Zig
|
||||
if: inputs.install-cross-tools == 'true'
|
||||
uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2
|
||||
uses: mlugg/setup-zig@v2
|
||||
|
||||
- name: Install cargo-zigbuild
|
||||
if: inputs.install-cross-tools == 'true'
|
||||
uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
|
||||
uses: taiki-e/install-action@cargo-zigbuild
|
||||
|
||||
- name: Install cargo-nextest
|
||||
uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
|
||||
uses: taiki-e/install-action@cargo-nextest
|
||||
|
||||
- name: Setup Rust cache
|
||||
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
cache-all-crates: true
|
||||
cache-on-failure: true
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
enabled: true
|
||||
|
||||
document:
|
||||
version: v1
|
||||
url: https://github.com/rustfs/cla/blob/main/cla/v1.md
|
||||
|
||||
signing:
|
||||
mode: comment
|
||||
comment_pattern: I have read and agree to the CLA.
|
||||
|
||||
registry:
|
||||
type: json-repo
|
||||
repository: rustfs/cla
|
||||
path_prefix: signatures
|
||||
|
||||
status:
|
||||
check_name: CLA Check
|
||||
@@ -1 +0,0 @@
|
||||
../AGENTS.md
|
||||
+2
-28
@@ -22,34 +22,8 @@ updates:
|
||||
- package-ecosystem: "cargo" # See documentation for possible values
|
||||
directory: "/" # Location of package manifests
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
day: "monday"
|
||||
timezone: "Asia/Shanghai"
|
||||
time: "08:00"
|
||||
assignees:
|
||||
- "houseme"
|
||||
reviewers:
|
||||
- "overtrue"
|
||||
- "majinghe"
|
||||
ignore:
|
||||
- dependency-name: "object_store"
|
||||
versions: [ "0.13.x" ]
|
||||
- dependency-name: "libunftp"
|
||||
versions: [ "0.23.x" ]
|
||||
- dependency-name: "ratelimit"
|
||||
versions: [ "1.x" ]
|
||||
- dependency-name: "ratelimit"
|
||||
versions: [ "2.x" ]
|
||||
- dependency-name: "pyroscope"
|
||||
versions: [ "2.x" ]
|
||||
interval: "monthly"
|
||||
groups:
|
||||
s3s:
|
||||
update-types:
|
||||
- "minor"
|
||||
- "patch"
|
||||
patterns:
|
||||
- "s3s"
|
||||
- "s3s-*"
|
||||
dependencies:
|
||||
patterns:
|
||||
- "*"
|
||||
- "*"
|
||||
|
||||
@@ -2,35 +2,36 @@
|
||||
Pull Request Template for RustFS
|
||||
-->
|
||||
|
||||
## Type of Change
|
||||
- [ ] New Feature
|
||||
- [ ] Bug Fix
|
||||
- [ ] Documentation
|
||||
- [ ] Performance Improvement
|
||||
- [ ] Test/CI
|
||||
- [ ] Refactor
|
||||
- [ ] Other:
|
||||
|
||||
## Related Issues
|
||||
<!--
|
||||
List related issues, e.g. Fixes #123.
|
||||
Use N/A when there is no related issue.
|
||||
-->
|
||||
<!-- List related Issue numbers, e.g. #123 -->
|
||||
|
||||
## Summary of Changes
|
||||
<!--
|
||||
Briefly explain what changed and why reviewers should accept it.
|
||||
Focus on behavior, compatibility, and review-relevant context.
|
||||
-->
|
||||
<!-- Briefly describe the main changes and motivation for this PR -->
|
||||
|
||||
## Verification
|
||||
<!--
|
||||
List the commands or checks you ran, for example:
|
||||
- `make pre-commit`
|
||||
|
||||
Use N/A only when verification is not applicable.
|
||||
-->
|
||||
## Checklist
|
||||
- [ ] I have read and followed the [CONTRIBUTING.md](CONTRIBUTING.md) guidelines
|
||||
- [ ] Passed `make pre-commit`
|
||||
- [ ] Added/updated necessary tests
|
||||
- [ ] Documentation updated (if needed)
|
||||
- [ ] CI/CD passed (if applicable)
|
||||
|
||||
## Impact
|
||||
<!--
|
||||
Describe user-facing, compatibility, API, deployment, configuration, or
|
||||
documentation impact. Use N/A when there is no expected impact.
|
||||
-->
|
||||
- [ ] Breaking change (compatibility)
|
||||
- [ ] Requires doc/config/deployment update
|
||||
- [ ] Other impact:
|
||||
|
||||
## Additional Notes
|
||||
<!-- Any extra information for reviewers, or N/A. -->
|
||||
<!-- Any extra information for reviewers -->
|
||||
|
||||
---
|
||||
|
||||
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
|
||||
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)) and sign the CLA if this is your first contribution.
|
||||
|
||||
@@ -1,81 +0,0 @@
|
||||
# S3 Compatibility Tests Configuration
|
||||
|
||||
This directory contains the configuration template for running
|
||||
[Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
|
||||
|
||||
The test runner itself lives in [`scripts/s3-tests/`](../../scripts/s3-tests/)
|
||||
— see its README for how tests are selected, executed, and reported. This
|
||||
directory only holds the shared config template.
|
||||
|
||||
## Configuration File
|
||||
|
||||
`s3tests.conf` is based on the official `s3tests.conf.SAMPLE` from the
|
||||
ceph/s3-tests repository. It is a template: `scripts/s3-tests/run.sh` renders
|
||||
it with `envsubst` before every run.
|
||||
|
||||
Required environment variables (all have defaults in `run.sh`):
|
||||
|
||||
| Variable | Meaning |
|
||||
|----------|---------|
|
||||
| `S3_HOST` | RustFS endpoint host |
|
||||
| `S3_PORT` | RustFS endpoint port |
|
||||
| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | main user credentials |
|
||||
| `S3_ALT_ACCESS_KEY` / `S3_ALT_SECRET_KEY` | alt user credentials (must differ from main) |
|
||||
|
||||
TLS is disabled (`is_secure = False`).
|
||||
|
||||
## Test Selection Strategy
|
||||
|
||||
Tests are selected by exact pytest node ids from classification list files in
|
||||
`scripts/s3-tests/` — **not** by pytest markers:
|
||||
|
||||
- `implemented_tests.txt` — expected to pass; run as the per-PR gate
|
||||
(`ci.yml`, job `s3-implemented-tests`)
|
||||
- `unimplemented_tests.txt` — standard S3 features not yet implemented
|
||||
- `excluded_tests.txt` — intentionally excluded (vendor-specific behavior or
|
||||
product decisions)
|
||||
|
||||
The upstream s3-tests repository is pinned to a fixed commit (`S3TESTS_REV` in
|
||||
`run.sh`) so results are reproducible; bump it deliberately and reclassify the
|
||||
lists when upstream changes.
|
||||
|
||||
## Where Tests Run
|
||||
|
||||
- **Per PR**: `.github/workflows/ci.yml` job `s3-implemented-tests` runs the
|
||||
implemented whitelist against a single-node debug binary. Any failure blocks
|
||||
the PR.
|
||||
- **Weekly + manual**: `.github/workflows/e2e-s3tests.yml` runs the full
|
||||
upstream suite (`TEST_SCOPE=all`) against a Docker deployment (single node
|
||||
or a 4-node distributed cluster behind HAProxy). It fails only on
|
||||
regressions in the implemented whitelist and publishes a classification
|
||||
report (`compat-report.md`, also shown in the job summary) listing promotion
|
||||
candidates and unclassified tests.
|
||||
|
||||
## Running Tests Locally
|
||||
|
||||
```bash
|
||||
# Whitelist run against a locally built binary (the same thing the PR gate does)
|
||||
./scripts/s3-tests/run.sh
|
||||
|
||||
# Full sweep, never stop on failures, 4 pytest workers
|
||||
TEST_SCOPE=all MAXFAIL=0 XDIST=4 ./scripts/s3-tests/run.sh
|
||||
|
||||
# A specific test against an already-running server
|
||||
DEPLOY_MODE=existing TESTEXPR="test_bucket_list_empty" ./scripts/s3-tests/run.sh
|
||||
```
|
||||
|
||||
Results land in `artifacts/s3tests-single/` (`junit.xml`, `pytest.log`,
|
||||
`compat-report.md`).
|
||||
|
||||
## Adding New Feature Support
|
||||
|
||||
Follow [`scripts/s3-tests/S3_COMPAT_WORKFLOW.md`](../../scripts/s3-tests/S3_COMPAT_WORKFLOW.md).
|
||||
In short: fix the behavior, verify with `TESTEXPR=...`, then move the test
|
||||
from `unimplemented_tests.txt` to `implemented_tests.txt` so the PR gate locks
|
||||
it in.
|
||||
|
||||
## References
|
||||
|
||||
- [Ceph S3 Tests Repository](https://github.com/ceph/s3-tests)
|
||||
- [S3 API Compatibility](https://docs.aws.amazon.com/AmazonS3/latest/API/)
|
||||
- [RustFS test runner](../../scripts/s3-tests/README.md)
|
||||
@@ -1,193 +0,0 @@
|
||||
# RustFS s3-tests configuration
|
||||
# Based on: https://github.com/ceph/s3-tests/blob/master/s3tests.conf.SAMPLE
|
||||
#
|
||||
# Usage:
|
||||
# Single-node: S3_HOST=rustfs-single envsubst < s3tests.conf > /tmp/s3tests.conf
|
||||
# Multi-node: S3_HOST=lb envsubst < s3tests.conf > /tmp/s3tests.conf
|
||||
|
||||
[DEFAULT]
|
||||
## this section is just used for host, port and bucket_prefix
|
||||
|
||||
# host set for RustFS - will be substituted via envsubst
|
||||
host = ${S3_HOST}
|
||||
|
||||
# port for RustFS - will be substituted via envsubst
|
||||
port = ${S3_PORT}
|
||||
|
||||
## say "False" to disable TLS
|
||||
is_secure = False
|
||||
|
||||
## say "False" to disable SSL Verify
|
||||
ssl_verify = False
|
||||
|
||||
[fixtures]
|
||||
## all the buckets created will start with this prefix;
|
||||
## {random} will be filled with random characters to pad
|
||||
## the prefix to 30 characters long, and avoid collisions
|
||||
bucket prefix = rustfs-{random}-
|
||||
|
||||
# all the iam account resources (users, roles, etc) created
|
||||
# will start with this name prefix
|
||||
iam name prefix = s3-tests-
|
||||
|
||||
# all the iam account resources (users, roles, etc) created
|
||||
# will start with this path prefix
|
||||
iam path prefix = /s3-tests/
|
||||
|
||||
[s3 main]
|
||||
# main display_name
|
||||
display_name = RustFS Tester
|
||||
|
||||
# main user_id
|
||||
user_id = rustfsadmin
|
||||
|
||||
# main email
|
||||
email = tester@rustfs.local
|
||||
|
||||
# zonegroup api_name for bucket location
|
||||
api_name = default
|
||||
|
||||
## main AWS access key
|
||||
access_key = ${S3_ACCESS_KEY}
|
||||
|
||||
## main AWS secret key
|
||||
secret_key = ${S3_SECRET_KEY}
|
||||
|
||||
## replace with key id obtained when secret is created, or delete if KMS not tested
|
||||
#kms_keyid = 01234567-89ab-cdef-0123-456789abcdef
|
||||
|
||||
## Storage classes
|
||||
#storage_classes = "LUKEWARM, FROZEN"
|
||||
|
||||
## Lifecycle debug interval (default: 10)
|
||||
#lc_debug_interval = 20
|
||||
## Restore debug interval (default: 100)
|
||||
#rgw_restore_debug_interval = 60
|
||||
#rgw_restore_processor_period = 60
|
||||
|
||||
[s3 alt]
|
||||
# alt display_name
|
||||
display_name = RustFS Alt Tester
|
||||
|
||||
## alt email
|
||||
email = alt@rustfs.local
|
||||
|
||||
# alt user_id
|
||||
user_id = rustfsalt
|
||||
|
||||
# alt AWS access key (must be different from s3 main for many tests)
|
||||
access_key = ${S3_ALT_ACCESS_KEY}
|
||||
|
||||
# alt AWS secret key
|
||||
secret_key = ${S3_ALT_SECRET_KEY}
|
||||
|
||||
#[s3 cloud]
|
||||
## to run the testcases with "cloud_transition" for transition
|
||||
## and "cloud_restore" for restore attribute.
|
||||
## Note: the waiting time may have to tweaked depending on
|
||||
## the I/O latency to the cloud endpoint.
|
||||
|
||||
## host set for cloud endpoint
|
||||
# host = localhost
|
||||
|
||||
## port set for cloud endpoint
|
||||
# port = 8001
|
||||
|
||||
## say "False" to disable TLS
|
||||
# is_secure = False
|
||||
|
||||
## cloud endpoint credentials
|
||||
# access_key = 0555b35654ad1656d804
|
||||
# secret_key = h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
|
||||
|
||||
## storage class configured as cloud tier on local rgw server
|
||||
# cloud_storage_class = CLOUDTIER
|
||||
|
||||
## Below are optional -
|
||||
|
||||
## Above configured cloud storage class config options
|
||||
# retain_head_object = false
|
||||
# allow_read_through = false # change it to enable read_through
|
||||
# read_through_restore_days = 2
|
||||
# target_storage_class = Target_SC
|
||||
# target_path = cloud-bucket
|
||||
|
||||
## another regular storage class to test multiple transition rules,
|
||||
# storage_class = S1
|
||||
|
||||
[s3 tenant]
|
||||
# tenant display_name
|
||||
display_name = RustFS Tenant Tester
|
||||
|
||||
# tenant user_id
|
||||
# Note: Using same user_id as main to avoid teardown failures.
|
||||
# RustFS does not currently support multi-tenancy, so the tenant client
|
||||
# effectively operates as the main user. This ensures nuke_prefixed_buckets()
|
||||
# in s3-tests teardown can successfully clean up resources.
|
||||
user_id = rustfsadmin
|
||||
|
||||
# tenant AWS access key
|
||||
access_key = ${S3_ACCESS_KEY}
|
||||
|
||||
# tenant AWS secret key
|
||||
secret_key = ${S3_SECRET_KEY}
|
||||
|
||||
# tenant email
|
||||
email = tenant@rustfs.local
|
||||
|
||||
# tenant name
|
||||
# Note: Empty tenant name to avoid multi-tenant path issues during teardown.
|
||||
# When s3-tests calls get_tenant_client(), it uses this tenant value in requests.
|
||||
# An empty value makes the tenant client behave like the main client, preventing
|
||||
# "bucket not found" errors when teardown tries to clean up test buckets.
|
||||
tenant =
|
||||
|
||||
#following section needs to be added for all sts-tests
|
||||
[iam]
|
||||
#used for iam operations in sts-tests
|
||||
#email
|
||||
email = s3@rustfs.local
|
||||
|
||||
#user_id
|
||||
user_id = rustfsiam
|
||||
|
||||
#access_key
|
||||
access_key = ${S3_ACCESS_KEY}
|
||||
|
||||
#secret_key
|
||||
secret_key = ${S3_SECRET_KEY}
|
||||
|
||||
#display_name
|
||||
display_name = RustFS IAM User
|
||||
|
||||
# iam account root user for iam_account tests
|
||||
[iam root]
|
||||
access_key = ${S3_ACCESS_KEY}
|
||||
secret_key = ${S3_SECRET_KEY}
|
||||
user_id = RGW11111111111111111
|
||||
email = account1@rustfs.local
|
||||
|
||||
# iam account root user in a different account than [iam root]
|
||||
[iam alt root]
|
||||
access_key = ${S3_ACCESS_KEY}
|
||||
secret_key = ${S3_SECRET_KEY}
|
||||
user_id = RGW22222222222222222
|
||||
email = account2@rustfs.local
|
||||
|
||||
#following section needs to be added when you want to run Assume Role With Webidentity test
|
||||
[webidentity]
|
||||
#used for assume role with web identity test in sts-tests
|
||||
#all parameters will be obtained from ceph/qa/tasks/keycloak.py
|
||||
#token=<access_token>
|
||||
|
||||
#aud=<obtained after introspecting token>
|
||||
|
||||
#sub=<obtained after introspecting token>
|
||||
|
||||
#azp=<obtained after introspecting token>
|
||||
|
||||
#user_token=<access token for a user, with attribute Department=[Engineering, Marketing>]
|
||||
|
||||
#thumbprint=<obtained from x509 certificate>
|
||||
|
||||
#KC_REALM=<name of the realm>
|
||||
@@ -1,57 +0,0 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
name: Architecture Migration Rules
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [ opened, synchronize, reopened, closed ]
|
||||
branches: [ main ]
|
||||
paths:
|
||||
- "ARCHITECTURE.md"
|
||||
- "docs/architecture/**"
|
||||
- "scripts/check_architecture_migration_rules.sh"
|
||||
- ".github/workflows/architecture-migration-rules.yml"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
cancel-closed-pr-runs:
|
||||
name: Cancel Closed PR Runs
|
||||
if: github.event_name == 'pull_request' && github.event.action == 'closed'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Explain cancellation run
|
||||
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
|
||||
|
||||
architecture-migration-rules:
|
||||
name: Architecture Migration Rules
|
||||
if: github.event_name != 'pull_request' || github.event.action != 'closed'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
|
||||
- name: Install ripgrep
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ripgrep
|
||||
|
||||
- name: Check architecture migration rules
|
||||
run: ./scripts/check_architecture_migration_rules.sh
|
||||
@@ -20,20 +20,13 @@ on:
|
||||
paths:
|
||||
- '**/Cargo.toml'
|
||||
- '**/Cargo.lock'
|
||||
- 'deny.toml'
|
||||
- '.github/actions/**'
|
||||
- '.github/workflows/**'
|
||||
- 'scripts/security/check_workflow_pins.sh'
|
||||
- '.github/workflows/audit.yml'
|
||||
pull_request:
|
||||
types: [ opened, synchronize, reopened, closed ]
|
||||
branches: [ main ]
|
||||
paths:
|
||||
- '**/Cargo.toml'
|
||||
- '**/Cargo.lock'
|
||||
- 'deny.toml'
|
||||
- '.github/actions/**'
|
||||
- '.github/workflows/**'
|
||||
- 'scripts/security/check_workflow_pins.sh'
|
||||
- '.github/workflows/audit.yml'
|
||||
schedule:
|
||||
- cron: '0 0 * * 0' # Weekly on Sunday at midnight UTC
|
||||
workflow_dispatch:
|
||||
@@ -41,33 +34,20 @@ on:
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
CARGO_TERM_COLOR: always
|
||||
|
||||
jobs:
|
||||
cancel-closed-pr-runs:
|
||||
name: Cancel Closed PR Runs
|
||||
if: github.event_name == 'pull_request' && github.event.action == 'closed'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Explain cancellation run
|
||||
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
|
||||
|
||||
security-audit:
|
||||
name: Security Audit
|
||||
if: github.event_name != 'pull_request' || github.event.action != 'closed'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Install cargo-audit
|
||||
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
|
||||
uses: taiki-e/install-action@v2
|
||||
with:
|
||||
tool: cargo-audit
|
||||
|
||||
@@ -77,60 +57,25 @@ jobs:
|
||||
|
||||
- name: Upload audit results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: security-audit-results-${{ github.run_number }}
|
||||
path: audit-results.json
|
||||
retention-days: 30
|
||||
|
||||
cargo-deny:
|
||||
name: Cargo Deny
|
||||
if: github.event_name != 'pull_request' || github.event.action != 'closed'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
|
||||
- name: Setup Rust environment
|
||||
uses: ./.github/actions/setup
|
||||
with:
|
||||
cache-shared-key: rustfs-cargo-deny
|
||||
|
||||
- name: Install cargo-deny
|
||||
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
|
||||
with:
|
||||
tool: cargo-deny
|
||||
|
||||
- name: Run cargo-deny
|
||||
run: cargo deny check --hide-inclusion-graph advisories sources bans licenses
|
||||
|
||||
workflow-pin-report:
|
||||
name: Workflow Pin Report
|
||||
if: github.event_name != 'pull_request' || github.event.action != 'closed'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
|
||||
- name: Report unpinned GitHub Actions
|
||||
run: ./scripts/security/check_workflow_pins.sh --enforce
|
||||
|
||||
dependency-review:
|
||||
name: Dependency Review
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'pull_request' && github.event.action != 'closed'
|
||||
if: github.event_name == 'pull_request'
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Dependency Review
|
||||
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5
|
||||
uses: actions/dependency-review-action@v4
|
||||
with:
|
||||
fail-on-severity: moderate
|
||||
allow-ghsas: GHSA-2f9f-gq7v-9h6m
|
||||
comment-summary-in-pr: always
|
||||
comment-summary-in-pr: true
|
||||
|
||||
+155
-238
@@ -23,7 +23,6 @@
|
||||
#
|
||||
# Manual Parameters:
|
||||
# - build_docker: Build and push Docker images (default: true)
|
||||
# - platforms: Comma-separated platform IDs or 'all' (default: all)
|
||||
|
||||
name: Build and Release
|
||||
|
||||
@@ -45,6 +44,22 @@ on:
|
||||
- "**/*.svg"
|
||||
- ".gitignore"
|
||||
- ".dockerignore"
|
||||
pull_request:
|
||||
branches: [ main ]
|
||||
paths-ignore:
|
||||
- "**.md"
|
||||
- "**.txt"
|
||||
- ".github/**"
|
||||
- "docs/**"
|
||||
- "deploy/**"
|
||||
- "scripts/dev_*.sh"
|
||||
- "LICENSE*"
|
||||
- "README*"
|
||||
- "**/*.png"
|
||||
- "**/*.jpg"
|
||||
- "**/*.svg"
|
||||
- ".gitignore"
|
||||
- ".dockerignore"
|
||||
schedule:
|
||||
- cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
|
||||
workflow_dispatch:
|
||||
@@ -54,19 +69,10 @@ on:
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
platforms:
|
||||
description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
|
||||
required: false
|
||||
default: "all"
|
||||
type: string
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event_name == 'push' && github.ref || github.run_id }}
|
||||
cancel-in-progress: ${{ github.event_name == 'push' }}
|
||||
|
||||
env:
|
||||
CARGO_TERM_COLOR: always
|
||||
RUST_BACKTRACE: 1
|
||||
@@ -86,7 +92,7 @@ jobs:
|
||||
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
uses: actions/checkout@v5
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
@@ -148,78 +154,56 @@ jobs:
|
||||
echo " - Is prerelease: $is_prerelease"
|
||||
|
||||
# Build RustFS binaries
|
||||
prepare-platform-matrix:
|
||||
name: Prepare Platform Matrix
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
matrix: ${{ steps.select.outputs.matrix }}
|
||||
selected: ${{ steps.select.outputs.selected }}
|
||||
steps:
|
||||
- name: Select target platforms
|
||||
id: select
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
|
||||
selected="$(echo "${selected}" | tr -d '[:space:]')"
|
||||
if [[ -z "${selected}" ]]; then
|
||||
selected="all"
|
||||
fi
|
||||
|
||||
all='{"include":[
|
||||
{"target_id":"linux-x86_64-musl","os":"sm-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
|
||||
{"target_id":"linux-aarch64-musl","os":"sm-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
|
||||
{"target_id":"linux-x86_64-gnu","os":"sm-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
|
||||
{"target_id":"linux-aarch64-gnu","os":"sm-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
|
||||
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
|
||||
{"target_id":"macos-x86_64","os":"macos-26-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
|
||||
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
|
||||
]}'
|
||||
|
||||
if [[ "${selected}" == "all" ]]; then
|
||||
matrix="$(jq -c . <<<"${all}")"
|
||||
else
|
||||
unknown="$(jq -rn --arg selected "${selected}" --argjson all "${all}" '
|
||||
($selected | split(",") | map(select(length > 0))) as $req
|
||||
| ($all.include | map(.target_id)) as $known
|
||||
| [$req[] | select(( $known | index(.) ) == null)]
|
||||
')"
|
||||
if [[ "$(jq 'length' <<<"${unknown}")" -gt 0 ]]; then
|
||||
echo "Unknown platforms: $(jq -r 'join(\",\")' <<<"${unknown}")" >&2
|
||||
echo "Allowed: $(jq -r '.include[].target_id' <<<"${all}" | paste -sd ',' -)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
matrix="$(jq -c --arg selected "${selected}" '
|
||||
($selected | split(",") | map(select(length > 0))) as $req
|
||||
| .include |= map(select(.target_id as $id | ($req | index($id))))
|
||||
' <<<"${all}")"
|
||||
fi
|
||||
|
||||
echo "selected=${selected}" >> "$GITHUB_OUTPUT"
|
||||
echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
|
||||
echo "Selected platforms: ${selected}"
|
||||
|
||||
build-rustfs:
|
||||
name: Build RustFS
|
||||
needs: [ build-check, prepare-platform-matrix ]
|
||||
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
|
||||
runs-on: ${{ matrix.platform == 'linux' && fromJSON('["self-hosted","linux","sm-standard-2"]') || matrix.os }}
|
||||
timeout-minutes: 90
|
||||
needs: [ build-check ]
|
||||
if: needs.build-check.outputs.should_build == 'true'
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 60
|
||||
env:
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
||||
# Always enable Tokio unstable features (required by dial9-tokio-telemetry).
|
||||
# The RUSTFLAGS env var takes precedence over .cargo/config.toml [build] rustflags,
|
||||
# so we must include --cfg tokio_unstable here explicitly; otherwise an empty
|
||||
# RUSTFLAGS value would shadow the config-file flag and silently break tracing.
|
||||
RUSTFLAGS: "--cfg tokio_unstable ${{ matrix.rustflags }}"
|
||||
RUSTFLAGS: ${{ matrix.cross == 'false' && '-C target-cpu=native' || '' }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
|
||||
matrix:
|
||||
include:
|
||||
# Linux builds
|
||||
- os: ubuntu-latest
|
||||
target: x86_64-unknown-linux-musl
|
||||
cross: false
|
||||
platform: linux
|
||||
- os: ubuntu-latest
|
||||
target: aarch64-unknown-linux-musl
|
||||
cross: true
|
||||
platform: linux
|
||||
- os: ubuntu-latest
|
||||
target: x86_64-unknown-linux-gnu
|
||||
cross: false
|
||||
platform: linux
|
||||
- os: ubuntu-latest
|
||||
target: aarch64-unknown-linux-gnu
|
||||
cross: true
|
||||
platform: linux
|
||||
# macOS builds
|
||||
- os: macos-latest
|
||||
target: aarch64-apple-darwin
|
||||
cross: false
|
||||
platform: macos
|
||||
- os: macos-latest
|
||||
target: x86_64-apple-darwin
|
||||
cross: false
|
||||
platform: macos
|
||||
# Windows builds (temporarily disabled)
|
||||
- os: windows-latest
|
||||
target: x86_64-pc-windows-msvc
|
||||
cross: false
|
||||
platform: windows
|
||||
#- os: windows-latest
|
||||
# target: aarch64-pc-windows-msvc
|
||||
# cross: true
|
||||
# platform: windows
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
uses: actions/checkout@v5
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
@@ -228,117 +212,35 @@ jobs:
|
||||
with:
|
||||
rust-version: stable
|
||||
target: ${{ matrix.target }}
|
||||
cache-shared-key: build-${{ matrix.target }}
|
||||
cache-shared-key: build-${{ matrix.target }}-${{ hashFiles('**/Cargo.lock') }}
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
|
||||
install-cross-tools: ${{ matrix.cross }}
|
||||
|
||||
- name: Download static console assets
|
||||
shell: bash
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
mkdir -p ./rustfs/static
|
||||
|
||||
verify_sha256() {
|
||||
local expected="$1"
|
||||
local file="$2"
|
||||
local actual=""
|
||||
local actual_line=""
|
||||
|
||||
if command -v sha256sum >/dev/null 2>&1; then
|
||||
actual_line=$(sha256sum -- "$file") || return 1
|
||||
elif command -v shasum >/dev/null 2>&1; then
|
||||
actual_line=$(shasum -a 256 -- "$file") || return 1
|
||||
if [[ "${{ matrix.platform }}" == "windows" ]]; then
|
||||
curl.exe -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300
|
||||
if [[ $? -eq 0 ]]; then
|
||||
unzip -o console.zip -d ./rustfs/static
|
||||
rm console.zip
|
||||
else
|
||||
echo "No SHA-256 verification tool found" >&2
|
||||
return 1
|
||||
echo "Warning: Failed to download console assets, continuing without them"
|
||||
echo "// Static assets not available" > ./rustfs/static/empty.txt
|
||||
fi
|
||||
actual="${actual_line%%[[:space:]]*}"
|
||||
|
||||
if [ "$actual" = "$expected" ]; then
|
||||
echo "SHA256 verified OK: $actual"
|
||||
return 0
|
||||
else
|
||||
echo "SHA256 mismatch: expected=$expected actual=$actual" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
download_console_assets() {
|
||||
local curl_bin="curl"
|
||||
local python_bin="python3"
|
||||
local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
|
||||
local console_json="console-release.json"
|
||||
local console_url
|
||||
local console_sha256
|
||||
local curl_auth_args=()
|
||||
|
||||
if [[ "${{ matrix.platform }}" == "windows" ]]; then
|
||||
curl_bin="curl.exe"
|
||||
else
|
||||
chmod +w ./rustfs/static/LICENSE || true
|
||||
fi
|
||||
|
||||
if ! command -v "$python_bin" >/dev/null 2>&1; then
|
||||
python_bin="python"
|
||||
fi
|
||||
if ! command -v "$python_bin" >/dev/null 2>&1; then
|
||||
echo "No Python interpreter found for release metadata parsing" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
|
||||
curl_auth_args=(-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "X-GitHub-Api-Version: 2022-11-28")
|
||||
fi
|
||||
|
||||
"$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
|
||||
-o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
|
||||
|
||||
read -r console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
|
||||
with open(sys.argv[1], encoding="utf-8") as handle:
|
||||
release = json.load(handle)
|
||||
|
||||
for asset in release.get("assets", []):
|
||||
name = asset.get("name", "")
|
||||
digest = asset.get("digest", "")
|
||||
url = asset.get("browser_download_url", "")
|
||||
if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
|
||||
sha256 = digest.split(":", 1)[1]
|
||||
if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
|
||||
raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
|
||||
sys.stdout.buffer.write(f"{url} {sha256}\n".encode("utf-8"))
|
||||
break
|
||||
else:
|
||||
raise SystemExit("no console zip asset with sha256 digest found")
|
||||
PY
|
||||
) || return 1
|
||||
|
||||
if [[ -z "$console_url" || -z "$console_sha256" ]]; then
|
||||
echo "Console release metadata is missing URL or SHA-256 digest" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
"$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
|
||||
verify_sha256 "$console_sha256" console.zip || return 2
|
||||
unzip -o console.zip -d ./rustfs/static || return 2
|
||||
}
|
||||
|
||||
if download_console_assets; then
|
||||
rm -f console.zip console-release.json
|
||||
else
|
||||
status=$?
|
||||
rm -f console.zip console-release.json
|
||||
if [[ "$status" -eq 2 ]]; then
|
||||
echo "Console asset integrity verification failed" >&2
|
||||
exit 1
|
||||
chmod +w ./rustfs/static/LICENSE || true
|
||||
curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
|
||||
-o console.zip --retry 3 --retry-delay 5 --max-time 300
|
||||
if [[ $? -eq 0 ]]; then
|
||||
unzip -o console.zip -d ./rustfs/static
|
||||
rm console.zip
|
||||
else
|
||||
echo "Warning: Failed to download console assets, continuing without them"
|
||||
echo "// Static assets not available" > ./rustfs/static/empty.txt
|
||||
fi
|
||||
echo "Warning: Failed to download verified console assets, continuing without them"
|
||||
echo "// Static assets not available" > ./rustfs/static/empty.txt
|
||||
fi
|
||||
|
||||
- name: Build RustFS
|
||||
@@ -348,8 +250,14 @@ jobs:
|
||||
touch rustfs/build.rs
|
||||
|
||||
if [[ "${{ matrix.cross }}" == "true" ]]; then
|
||||
# All cross targets in the matrix are Linux; zigbuild handles them.
|
||||
cargo zigbuild --release --target ${{ matrix.target }} -p rustfs --bins
|
||||
if [[ "${{ matrix.platform }}" == "windows" ]]; then
|
||||
# Use cross for Windows ARM64
|
||||
cargo install cross --git https://github.com/cross-rs/cross
|
||||
cross build --release --target ${{ matrix.target }} -p rustfs --bins
|
||||
else
|
||||
# Use zigbuild for other cross-compilation
|
||||
cargo zigbuild --release --target ${{ matrix.target }} -p rustfs --bins
|
||||
fi
|
||||
else
|
||||
cargo build --release --target ${{ matrix.target }} -p rustfs --bins
|
||||
fi
|
||||
@@ -401,26 +309,19 @@ jobs:
|
||||
;;
|
||||
esac
|
||||
|
||||
# Normalize version used for package filenames
|
||||
PACKAGE_VERSION="${VERSION}"
|
||||
if [[ "$PACKAGE_VERSION" == v* ]]; then
|
||||
PACKAGE_VERSION="${PACKAGE_VERSION#v}"
|
||||
fi
|
||||
|
||||
# Generate package name based on build type
|
||||
if [[ -n "$VARIANT" ]]; then
|
||||
ARCH_WITH_VARIANT="${ARCH}-${VARIANT}"
|
||||
else
|
||||
ARCH_WITH_VARIANT="${ARCH}"
|
||||
fi
|
||||
PACKAGE_BASENAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}"
|
||||
|
||||
if [[ "$BUILD_TYPE" == "development" ]]; then
|
||||
# Development build: rustfs-${platform}-${arch}-${variant}-dev-${short_sha}.zip
|
||||
PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}-dev-${SHORT_SHA}"
|
||||
else
|
||||
# Release/Prerelease build: rustfs-${platform}-${arch}-${variant}-v${version}.zip
|
||||
PACKAGE_NAME="${PACKAGE_BASENAME}-v${PACKAGE_VERSION}"
|
||||
PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}-v${VERSION}"
|
||||
fi
|
||||
|
||||
# Create zip packages for all platforms
|
||||
@@ -490,7 +391,7 @@ jobs:
|
||||
if [[ "$BUILD_TYPE" == "release" ]] || [[ "$BUILD_TYPE" == "prerelease" ]]; then
|
||||
# Create latest version filename
|
||||
# Convert from rustfs-linux-x86_64-musl-v1.0.0 to rustfs-linux-x86_64-musl-latest
|
||||
LATEST_FILE="${PACKAGE_BASENAME}-latest.zip"
|
||||
LATEST_FILE="${PACKAGE_NAME%-v*}-latest.zip"
|
||||
|
||||
echo "🔄 Creating latest version: ${PACKAGE_NAME}.zip -> $LATEST_FILE"
|
||||
cp "${PACKAGE_NAME}.zip" "$LATEST_FILE"
|
||||
@@ -503,7 +404,7 @@ jobs:
|
||||
# Development builds (only main branch triggers development builds)
|
||||
# Create main-latest version filename
|
||||
# Convert from rustfs-linux-x86_64-dev-abc123 to rustfs-linux-x86_64-main-latest
|
||||
MAIN_LATEST_FILE="${PACKAGE_BASENAME}-main-latest.zip"
|
||||
MAIN_LATEST_FILE="${PACKAGE_NAME%-dev-*}-main-latest.zip"
|
||||
|
||||
echo "🔄 Creating main-latest version: ${PACKAGE_NAME}.zip -> $MAIN_LATEST_FILE"
|
||||
cp "${PACKAGE_NAME}.zip" "$MAIN_LATEST_FILE"
|
||||
@@ -541,53 +442,85 @@ jobs:
|
||||
echo "📊 Version: ${VERSION}"
|
||||
|
||||
- name: Upload to GitHub artifacts
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ${{ steps.package.outputs.package_name }}
|
||||
path: "rustfs-*.zip"
|
||||
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
|
||||
|
||||
- name: Upload to Cloudflare R2
|
||||
if: env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
|
||||
- name: Upload to Aliyun OSS
|
||||
if: env.OSS_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
|
||||
env:
|
||||
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
AWS_EC2_METADATA_DISABLED: true
|
||||
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
|
||||
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
|
||||
OSS_REGION: cn-beijing
|
||||
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
|
||||
shell: bash
|
||||
run: |
|
||||
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
|
||||
|
||||
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
|
||||
echo "⚠️ R2 credentials or endpoint missing, skipping artifact upload"
|
||||
exit 0
|
||||
fi
|
||||
# Install ossutil (platform-specific)
|
||||
OSSUTIL_VERSION="2.1.1"
|
||||
case "${{ matrix.platform }}" in
|
||||
linux)
|
||||
if [[ "$(uname -m)" == "arm64" ]]; then
|
||||
ARCH="arm64"
|
||||
else
|
||||
ARCH="amd64"
|
||||
fi
|
||||
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-${ARCH}.zip"
|
||||
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-${ARCH}"
|
||||
|
||||
if ! command -v aws >/dev/null 2>&1; then
|
||||
echo "❌ aws CLI not found on runner; cannot upload to R2"
|
||||
exit 1
|
||||
fi
|
||||
curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
|
||||
unzip "$OSSUTIL_ZIP"
|
||||
mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
|
||||
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
|
||||
chmod +x /usr/local/bin/ossutil
|
||||
OSSUTIL_BIN=ossutil
|
||||
;;
|
||||
macos)
|
||||
if [[ "$(uname -m)" == "arm64" ]]; then
|
||||
ARCH="arm64"
|
||||
else
|
||||
ARCH="amd64"
|
||||
fi
|
||||
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-mac-${ARCH}.zip"
|
||||
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-mac-${ARCH}"
|
||||
|
||||
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
|
||||
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
|
||||
export AWS_DEFAULT_REGION="auto"
|
||||
curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
|
||||
unzip "$OSSUTIL_ZIP"
|
||||
mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
|
||||
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
|
||||
chmod +x /usr/local/bin/ossutil
|
||||
OSSUTIL_BIN=ossutil
|
||||
;;
|
||||
windows)
|
||||
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-windows-amd64.zip"
|
||||
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-windows-amd64"
|
||||
|
||||
curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
|
||||
unzip "$OSSUTIL_ZIP"
|
||||
mv "${OSSUTIL_DIR}/ossutil.exe" ./ossutil.exe
|
||||
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
|
||||
OSSUTIL_BIN=./ossutil.exe
|
||||
;;
|
||||
esac
|
||||
|
||||
# Determine upload path based on build type
|
||||
if [[ "$BUILD_TYPE" == "development" ]]; then
|
||||
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/dev/"
|
||||
echo "📤 Uploading development build to R2 dev directory"
|
||||
OSS_PATH="oss://rustfs-artifacts/artifacts/rustfs/dev/"
|
||||
echo "📤 Uploading development build to OSS dev directory"
|
||||
else
|
||||
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
|
||||
echo "📤 Uploading release build to R2 release directory"
|
||||
OSS_PATH="oss://rustfs-artifacts/artifacts/rustfs/release/"
|
||||
echo "📤 Uploading release build to OSS release directory"
|
||||
fi
|
||||
|
||||
# Upload all rustfs zip files to R2 using S3-compatible API
|
||||
echo "📤 Uploading all rustfs-*.zip files to $R2_PATH..."
|
||||
# Upload all rustfs zip files to OSS using glob pattern
|
||||
echo "📤 Uploading all rustfs-*.zip files to $OSS_PATH..."
|
||||
for zip_file in rustfs-*.zip; do
|
||||
if [[ -f "$zip_file" ]]; then
|
||||
echo "Uploading: $zip_file to $R2_PATH..."
|
||||
aws s3 cp "$zip_file" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
|
||||
echo "Uploading: $zip_file to $OSS_PATH..."
|
||||
$OSSUTIL_BIN cp "$zip_file" "$OSS_PATH" --force
|
||||
echo "✅ Uploaded: $zip_file"
|
||||
fi
|
||||
done
|
||||
@@ -659,7 +592,7 @@ jobs:
|
||||
release_url: ${{ steps.create.outputs.release_url }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
uses: actions/checkout@v5
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
@@ -743,10 +676,10 @@ jobs:
|
||||
actions: read
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Download all build artifacts
|
||||
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
|
||||
uses: actions/download-artifact@v5
|
||||
with:
|
||||
path: ./artifacts
|
||||
pattern: rustfs-*
|
||||
@@ -789,19 +722,6 @@ jobs:
|
||||
echo "# GPG signature will be added in future versions" >> "${file}.asc"
|
||||
done
|
||||
|
||||
cd ..
|
||||
python3 scripts/security/generate_release_supply_chain_assets.py \
|
||||
--asset-dir ./release-assets \
|
||||
--version "$VERSION" \
|
||||
--tag "$TAG" \
|
||||
--build-type "${{ needs.build-check.outputs.build_type }}" \
|
||||
--repository "${{ github.repository }}" \
|
||||
--ref "${{ github.ref }}" \
|
||||
--sha "${{ github.sha }}" \
|
||||
--run-id "${{ github.run_id }}" \
|
||||
--run-attempt "${{ github.run_attempt }}"
|
||||
cd ./release-assets
|
||||
|
||||
echo "📦 Prepared assets:"
|
||||
ls -la
|
||||
|
||||
@@ -853,15 +773,12 @@ jobs:
|
||||
OSSUTIL_VERSION="2.1.1"
|
||||
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
|
||||
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-amd64"
|
||||
OSSUTIL_SHA256="60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2"
|
||||
|
||||
curl --fail -L -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
|
||||
printf '%s %s\n' "$OSSUTIL_SHA256" "$OSSUTIL_ZIP" | sha256sum -c -
|
||||
curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
|
||||
unzip "$OSSUTIL_ZIP"
|
||||
OSSUTIL_BIN="${RUNNER_TEMP}/ossutil"
|
||||
mv "${OSSUTIL_DIR}/ossutil" "$OSSUTIL_BIN"
|
||||
mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
|
||||
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
|
||||
chmod +x "$OSSUTIL_BIN"
|
||||
chmod +x /usr/local/bin/ossutil
|
||||
|
||||
# Create latest.json
|
||||
cat > latest.json << EOF
|
||||
@@ -875,7 +792,7 @@ jobs:
|
||||
EOF
|
||||
|
||||
# Upload to OSS
|
||||
"$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
|
||||
ossutil cp latest.json oss://rustfs-version/latest.json --force
|
||||
|
||||
echo "✅ Updated latest.json for stable release $VERSION"
|
||||
|
||||
@@ -889,7 +806,7 @@ jobs:
|
||||
contents: write
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Update release notes and publish
|
||||
env:
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user