Compare commits

..

17 Commits

Author SHA1 Message Date
github-actions[bot] c43ed581d2 chore: update star history 2026-07-29 04:21:42 +00:00
github-actions[bot] 686d0e8df7 chore: update star history 2026-07-28 04:20:51 +00:00
github-actions[bot] d0193a9407 chore: update star history 2026-07-27 04:28:05 +00:00
github-actions[bot] 5a1b4bbd45 chore: update star history 2026-07-26 04:24:45 +00:00
github-actions[bot] 70c4282a5f chore: update star history 2026-07-25 04:21:07 +00:00
github-actions[bot] 03fc8e81cc chore: update star history 2026-07-24 04:22:29 +00:00
github-actions[bot] aa537f8a7b chore: update star history 2026-07-23 04:22:49 +00:00
github-actions[bot] cf3d4d663e chore: update star history 2026-07-22 04:23:13 +00:00
github-actions[bot] c0d454e6b7 chore: update star history 2026-07-21 04:22:23 +00:00
github-actions[bot] 31fbde559c chore: update star history 2026-07-20 04:28:07 +00:00
github-actions[bot] 8cf5d6906d chore: update star history 2026-07-19 04:24:01 +00:00
github-actions[bot] 125ac23907 chore: update star history 2026-07-19 02:23:45 +00:00
github-actions[bot] e72cdfe581 chore: update star history 2026-07-18 09:50:15 +00:00
Zhengchao An af5a5f847e chore: update repository visuals 2026-07-18 17:21:35 +08:00
github-actions[bot] 0a9ecc3cbf chore: update star history 2026-07-18 08:50:41 +00:00
Zhengchao An 75aab297ca chore: update star history 2026-07-18 13:17:05 +08:00
Zhengchao An 4571fd76ad chore: initialize star history 2026-07-18 12:48:47 +08:00
1410 changed files with 3878 additions and 508956 deletions
@@ -1,78 +0,0 @@
---
name: code-change-verification
description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
---
# Code Change Verification
Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
## Quick Start
1. Read the scope: commit, PR, patch, or file list.
2. Map each changed area by risk and user impact.
3. Inspect each risky change in context.
4. Report findings first, ordered by severity.
5. Close with residual risks and verification recommendations.
## Core Workflow
### 1) Scope and assumptions
- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
- If context is missing, state assumptions before deeper analysis.
- Focus only on requested scope; avoid reviewing unrelated files.
### 2) Risk map
- Prioritize in this order:
- Data correctness and user-visible behavior
- API/contract compatibility
- Security and authz/authn boundaries
- Concurrency and lifecycle correctness
- Performance and resource usage
- Give higher priority to stateful paths, migration logic, defaults, and error handling.
### 3) Evidence-based inspection
- Read each modified hunk with neighboring context.
- Trace call paths and call-site expectations.
- Check for:
- invariant breaks and missing guards
- unchecked assumptions and null/empty/error-path handling
- stale tests, fixtures, and configs
- hidden coupling to shared helpers/constants/features
- If a point is uncertain, mark it as an open question instead of guessing.
### 4) Findings-first output
- Order findings by severity:
- P0: critical failure, security breach, or data loss risk
- P1: high-impact regression
- P2: medium risk correctness gap
- P3: low risk/quality debt
- For each finding include:
- Severity
- `path:line` reference
- concise issue statement
- impact and likely failure mode
- specific fix or mitigation
- validation step to confirm
- If no issues exist, explicitly state `No findings` and why.
### 5) Close
- Report assumptions and unknowns.
- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
## Output Template
1. Findings
2. No findings (if applicable)
3. Assumptions / Unknowns
4. Recommended verification steps
## Finding Template
- `[P1] Missing timeout for downstream call`
- Location: `path/to/file.rs:123`
- Issue: ...
- Impact: ...
- Fix suggestion: ...
- Validation: ...
@@ -1,4 +0,0 @@
interface:
display_name: "Code Change Verification"
short_description: "Prioritize risks and verify code changes before merge."
default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."
@@ -1,91 +0,0 @@
---
name: pr-creation-checker
description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
---
# PR Creation Checker
Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
## Read sources of truth first
- Read `AGENTS.md`.
- Read `.github/pull_request_template.md`.
- Use `Makefile` and `.config/make/` for local quality commands.
- Use `.github/workflows/ci.yml` for CI expectations.
- Do not restate long command matrices or template sections from memory when the files exist.
## Workflow
1. Collect PR context
- Confirm base branch, current branch, change goal, and scope.
- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
- Confirm whether the branch includes only intended changes.
2. Inspect change scope
- Review the diff and summarize what changed.
- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
- Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
- Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
3. Verify readiness requirements
- Require `make pre-commit` before marking the PR ready.
- If `make` is unavailable, use the equivalent commands from `.config/make/`.
- Add scope-specific verification commands when the changed area needs more than the baseline.
- If required checks fail, stop and return `BLOCKED`.
4. Draft PR metadata
- Write the PR title in English using Conventional Commits and keep it within 72 characters.
- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
- Keep the PR body in English.
- Use the exact section headings from `.github/pull_request_template.md`.
- Fill non-applicable sections with `N/A`.
- Include verification commands in the PR description.
- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
5. Prepare reviewer context
- Summarize why the change exists.
- Summarize what was verified.
- Call out risks, rollout notes, config impact, and rollback notes when applicable.
- Mention assumptions or missing context instead of guessing.
6. Prepare CLI-safe output
- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
- Return a ready-to-save PR body plus a short title.
- If not ready, return blockers first and list the minimum steps needed to unblock.
## Output format
### Status
- `READY` or `BLOCKED`
### Title
- `<type>(<scope>): <summary>`
### PR Body
- Reproduce the repository template headings exactly.
- Fill every section.
- Omit local absolute paths unless explicitly required.
### Verification
- List each command run.
- State pass/fail.
### Risks
- List breaking changes, config changes, migration impact, or `N/A`.
## Blocker rules
- Return `BLOCKED` if `make pre-commit` has not passed.
- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
- Return `BLOCKED` if required template sections are missing.
- Return `BLOCKED` if the title/body is not in English.
- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
- Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
## Reference
- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.
@@ -1,4 +0,0 @@
interface:
display_name: "PR Creation Checker"
short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."
@@ -1,15 +0,0 @@
# PR Readiness Checklist
- Confirm the branch is based on current `main`.
- Confirm the diff matches the stated scope.
- Confirm no secrets, logs, temp files, or unrelated refactors are included.
- Confirm `make pre-commit` passed, or document why it could not run.
- Confirm extra verification commands are listed for risky changes.
- Confirm the PR title uses Conventional Commits and stays within 72 characters.
- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
- Confirm the PR body is in English.
- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
- Confirm non-applicable sections are filled with `N/A`.
- Confirm the PR body does not include local absolute paths unless explicitly required.
- Confirm multiline GitHub CLI commands use `--body-file`.
- Confirm new hardcoded string literals were not introduced for values already represented by existing constants/enums (including protocol labels, error identifiers, headers, and metric names), or record a justified exception.
@@ -1,124 +0,0 @@
---
name: rustfs-release-version-bump
description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation."
---
# RustFS Release Version Bump
Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).
Validated baseline: release pattern used in PR `#2957`.
## Required inputs
- Exact target version, for example `1.0.0-beta.4`.
- Delivery scope:
- Local only (`edit/verify`).
- Local + git (`commit/push`).
- Full GitHub flow (`commit/push/PR`).
If target version is missing or ambiguous, stop and ask before editing.
## Read before editing
- `AGENTS.md` (root and nearest path-specific files).
- `.github/pull_request_template.md`.
- Current branch status and diff against `origin/main`.
## Default release file scope
Treat the following file list as the default checklist for each release bump:
- `Cargo.toml`
- `Cargo.lock`
- `README.md`
- `README_ZH.md`
- `flake.nix`
- `helm/rustfs/Chart.yaml`
- `rustfs.spec`
Only drop a file when the current repository release process clearly no longer requires it.
## Hard release policy
- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
- Helm chart version mapping follows `beta.N -> 0.N.0`.
- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
- Do not change these rules without explicit confirmation.
## Step-by-step workflow
1. Confirm intent and isolate scope
- Confirm target version string exactly.
- Confirm whether user requested local-only or full GitHub flow.
- Inspect current branch and ensure only release-related files are touched for this task.
2. Update workspace versions
- Bump `[workspace.package].version` in `Cargo.toml`.
- Bump internal workspace crate dependency versions in `Cargo.toml`.
- Update `Cargo.lock` so workspace package versions match target version.
- Re-scan for partial leftovers.
3. Update release assets
- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
- `flake.nix`: update package version to target version.
- `helm/rustfs/Chart.yaml`:
- `appVersion` = target version.
- `version` follows chart mapping rule, for example:
- `1.0.0-beta.3` -> `0.3.0`
- `1.0.0-beta.4` -> `0.4.0`
- `rustfs.spec`:
- Set `Release` to prerelease suffix (example `beta.4`).
- Add/update top changelog entry with exact format:
- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
- `- Update RPM package to RustFS 1.0.0-beta.4`
- Changelog identity and time must come from current environment:
- `git config --get user.name`
- `git config --get user.email`
- `date '+%a %b %d %Y'`
- Changelog version text must match target release version exactly.
4. Verify before shipping
- Run:
- `cargo fmt --all`
- `cargo fmt --all --check`
- `make pre-commit`
- If verification passes, run `cargo clean`.
- If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
5. Commit strategy
- Preferred split when both parts changed:
- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
- `chore(release): align release assets for <version>` for docs and packaging files.
- If user asks for one commit, use one commit.
- Stage only intended release files; do not include unrelated working tree changes.
6. Push and PR
- Push branch:
- `git push -u origin <branch>` (first push), or `git push` (tracking already exists).
- Create PR with template headings unchanged:
- `gh pr create --base main --head <branch> --title ... --body-file ...`
- PR title/body must be English.
- Use `N/A` for non-applicable template sections.
- Include verification commands and any `BLOCKED` reason clearly.
## Recommended check commands
- `git status --short --branch`
- `git diff --name-only origin/main...HEAD`
- `git diff --stat origin/main...HEAD`
- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
- `cargo fmt --all`
- `cargo fmt --all --check`
- `make pre-commit`
- `cargo clean`
## Output contract
When using this skill, always report:
- Target version.
- Files changed.
- Any assumptions or uncertainties requiring confirmation.
- Verification result (`PASSED` or `BLOCKED`) with key evidence.
- Commit message(s) used.
- Push status and PR URL when GitHub flow is requested.
@@ -1,4 +0,0 @@
interface:
display_name: "RustFS Release Bump"
short_description: "Prepare RustFS release branches like PR #2957."
default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."
@@ -1,119 +0,0 @@
---
name: security-advisory-lessons
description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
---
# RustFS Security Advisory Lessons
Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
```bash
gh api repos/rustfs/rustfs/security-advisories --paginate \
--jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
```
Fetch full advisory details only when the live summary suggests a new or changed lesson:
```bash
gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
```
For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
## Workflow
### 1. Scope the change
- Identify touched routes, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/rio/`, and console preview/auth code.
### 2. Map to advisory classes
- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
### 3. Verify fail-closed behavior
- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
- Prefer exact action/permission checks over broad helper calls or inferred ownership.
- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
### 4. Require regression evidence
- For behavior changes, add focused negative tests that reproduce the advisory class.
- For sensitive fixes, include tests for the bypass form, not only the happy path.
- If a test is impractical, explain the residual risk and provide a manual verification command.
### 5. Report clearly
- Lead with concrete findings and file/line evidence.
- Separate proven vulnerabilities from hardening risks.
- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
## Advisory-Derived Guardrails
### Auth and admin authorization
- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
### IAM and service accounts
- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims.
- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
### S3 copy, multipart, and presigned POST
- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
### Paths, object keys, and filesystem access
- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
### Secrets, default credentials, and crypto
- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
- Defaults for internode/RPC auth must fail closed for network-reachable deployments or require explicit opt-in with loud warnings.
- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
- Plan key rotation and key IDs when removing exposed keys.
### Logging and debug output
- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
### RPC, parsing, and panic safety
- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
### Browser, CORS, and console surfaces
- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps.
### Profiling, debug, and health endpoints
- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
### Trusted proxy and network identity
- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
- Add tests for direct spoofed headers and trusted-proxy headers.
### SSE and storage invariants
- Encryption metadata is not proof that bytes were encrypted on disk.
- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
## Review Prompts
Use these prompts while reviewing a diff:
- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
- Does a public/default/empty config change security behavior from fail-closed to fail-open?
- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
- Does the test prove the exploit form is denied, or only that the intended form still works?
@@ -1,4 +0,0 @@
interface:
display_name: "Security Advisory Lessons"
short_description: "Apply advisory lessons in reviews."
default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes."
@@ -1,93 +0,0 @@
# RustFS Advisory Pattern Map
This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
When current advisory state, severity, URLs, or full text matters, fetch it live:
```bash
gh api repos/rustfs/rustfs/security-advisories --paginate \
--jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
```
Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
## Pattern Index
### Admin authorization and route exposure
- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
### IAM import, service accounts, and privilege boundaries
- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
### S3 copy, multipart, and upload policy validation
- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
### Filesystem paths and object key traversal
- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
### Secrets, defaults, and cryptographic misuse
- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
### Sensitive logging and debug output
- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
### RPC input validation and panic safety
- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
### Browser, CORS, and console isolation
- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
### Trusted proxy and source IP conditions
- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
### SSE and on-disk storage invariants
- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
## Useful Search Seeds
Use these targeted searches when a diff touches security-sensitive code:
```bash
rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
rg -n "PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
```
## Minimum Regression Test Expectations
- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
- Path fixes: include encoded traversal, absolute path, nested traversal, valid object keys that resemble traversal text but should be rejected, and canonical boundary checks.
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
@@ -1,66 +0,0 @@
---
name: test-coverage-improver
description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
---
# Test Coverage Improver
Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
## Usage assumptions
- Focus scope is either changed lines/files, a module, or the whole repository.
- Coverage artifact must be generated or provided in a supported format.
- If required context is missing, call out assumptions explicitly before proposing work.
## Workflow
1. Define scope and baseline
- Confirm target language, framework, and branch.
- Confirm whether the scope is changed files only or full-repo.
2. Produce coverage snapshot
- Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
- JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
- Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
- Collect total, per-file, and changed-line coverage.
3. Rank highest-risk gaps
- Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
- Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
- Keep shortlist to 58 gaps.
- For each gap, capture: file, lines, uncovered branches, and estimated risk score.
4. Propose high-impact tests
- For each shortlisted gap, output:
- Intent and expected behavior.
- Normal, edge, and failure scenarios.
- Assertions and side effects to verify.
- Setup needs (fixtures, mocks, integration dependencies).
- Estimated effort (`S/M/L`).
5. Close with validation plan
- State which gaps remain after proposals.
- Provide concrete verification command and acceptance threshold.
- List assumptions or blockers (environment, fixtures, flaky dependencies).
## Output template
### Coverage Snapshot
- total / branch coverage
- changed-file coverage
- top missing regions by size
### Top Gaps (ranked)
- `path:line-range` | risk score | why critical
### Test Proposals
- `path:line-range`
- Test name
- scenarios
- assertions
- effort
### Validation Plan
- command
- pass criteria
- remaining risk
@@ -1,4 +0,0 @@
interface:
display_name: "Test Coverage Improver"
short_description: "Find top uncovered risk areas and propose high-impact tests."
default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."
@@ -1,25 +0,0 @@
# Coverage Gap Prioritization Guide
Use this rubric for each uncovered area.
Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
- Criticality:
- 5: authz/authn, data-loss, payment/consistency path
- 4: state mutation, cache invalidation, scheduling
- 3: error handling + fallbacks in user-visible flows
- 2: parsing/format conversion paths
- 1: logging-only or low-impact utilities
- CoverageDebt:
- 0: 05 uncovered lines
- 1: 620 uncovered lines
- 2: 2140 uncovered lines
- 3: 41+ uncovered lines
- Volatility:
- 1: stable legacy code with few recent edits
- 2: changed in last 2 releases
- 3: touched in last 30 days or currently in active PR
Sort by score descending, then by business impact.
-26
View File
@@ -1,26 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# RustFS Cargo configuration
# Enable tokio_unstable cfg for dial9-tokio-telemetry support
# This allows dial9 to hook into Tokio's internal runtime events
[build]
# Enable Tokio unstable features required by dial9-tokio-telemetry for runtime tracing.
# See: https://docs.rs/tokio/latest/tokio/#unstable-features
rustflags = ["--cfg", "tokio_unstable"]
# Enable frame pointers for CPU profiling (Linux only, optional but recommended)
# Uncomment the following line for better CPU profiling data
# rustflags = ["--cfg", "tokio_unstable", "-C", "force-frame-pointers=yes"]
-64
View File
@@ -1,64 +0,0 @@
## —— Development/Source builds using direct buildx commands ---------------------------------------
.PHONY: docker-dev
docker-dev: ## Build dev multi-arch image (cannot load locally)
@echo "🏗️ Building multi-architecture development Docker images with buildx..."
@echo "💡 This builds from source code and is intended for local development and testing"
@echo "⚠️ Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
$(DOCKER_CLI) buildx build \
--platform linux/amd64,linux/arm64 \
--file $(DOCKERFILE_SOURCE) \
--tag rustfs:source-latest \
--tag rustfs:dev-latest \
.
.PHONY: docker-dev-local
docker-dev-local: ## Build dev single-arch image (local load)
@echo "🏗️ Building single-architecture development Docker image for local use..."
@echo "💡 This builds from source code for the current platform and loads locally"
$(DOCKER_CLI) buildx build \
--file $(DOCKERFILE_SOURCE) \
--tag rustfs:source-latest \
--tag rustfs:dev-latest \
--load \
.
.PHONY: docker-dev-push
docker-dev-push: ## Build and push multi-arch development image # e.g (make docker-dev-push REGISTRY=xxx)
@if [ -z "$(REGISTRY)" ]; then \
echo "❌ Error: Please specify registry, example: make docker-dev-push REGISTRY=ghcr.io/username"; \
exit 1; \
fi
@echo "🚀 Building and pushing multi-architecture development Docker images..."
@echo "💡 Pushing to registry: $(REGISTRY)"
$(DOCKER_CLI) buildx build \
--platform linux/amd64,linux/arm64 \
--file $(DOCKERFILE_SOURCE) \
--tag $(REGISTRY)/rustfs:source-latest \
--tag $(REGISTRY)/rustfs:dev-latest \
--push \
.
.PHONY: dev-env-start
dev-env-start: ## Start development container environment
@echo "🚀 Starting development environment..."
$(DOCKER_CLI) buildx build \
--file $(DOCKERFILE_SOURCE) \
--tag rustfs:dev \
--load \
.
$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) \
-p 9010:9010 -p 9000:9000 \
-v $(shell pwd):/workspace \
-it rustfs:dev
.PHONY: dev-env-stop
dev-env-stop: ## Stop development container environment
@echo "🛑 Stopping development environment..."
$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
.PHONY: dev-env-restart
dev-env-restart: dev-env-stop dev-env-start ## Restart development container environment
@@ -1,41 +0,0 @@
## —— Production builds using docker buildx (for CI/CD and production) -----------------------------
.PHONY: docker-buildx
docker-buildx: ## Build production multi-arch image (no push)
@echo "🏗️ Building multi-architecture production Docker images with buildx..."
./docker-buildx.sh
.PHONY: docker-buildx-push
docker-buildx-push: ## Build and push production multi-arch image
@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
./docker-buildx.sh --push
.PHONY: docker-buildx-version
docker-buildx-version: ## Build and version production multi-arch image # e.g (make docker-buildx-version VERSION=v1.0.0)
@if [ -z "$(VERSION)" ]; then \
echo "❌ Error: Please specify version, example: make docker-buildx-version VERSION=v1.0.0"; \
exit 1; \
fi
@echo "🏗️ Building multi-architecture production Docker images (version: $(VERSION))..."
./docker-buildx.sh --release $(VERSION)
.PHONY: docker-buildx-push-version
docker-buildx-push-version: ## Build and version and push production multi-arch image # e.g (make docker-buildx-push-version VERSION=v1.0.0)
@if [ -z "$(VERSION)" ]; then \
echo "❌ Error: Please specify version, example: make docker-buildx-push-version VERSION=v1.0.0"; \
exit 1; \
fi
@echo "🚀 Building and pushing multi-architecture production Docker images (version: $(VERSION))..."
./docker-buildx.sh --release $(VERSION) --push
.PHONY: docker-buildx-production-local
docker-buildx-production-local: ## Build production single-arch image locally
@echo "🏗️ Building single-architecture production Docker image locally..."
@echo "💡 Alternative to docker-buildx.sh for local testing"
$(DOCKER_CLI) buildx build \
--file $(DOCKERFILE_PRODUCTION) \
--tag rustfs:production-latest \
--tag rustfs:latest \
--load \
--build-arg RELEASE=latest \
.
-16
View File
@@ -1,16 +0,0 @@
## —— Single Architecture Docker Builds (Traditional) ----------------------------------------------
.PHONY: docker-build-production
docker-build-production: ## Build single-arch production image
@echo "🏗️ Building single-architecture production Docker image..."
@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
.PHONY: docker-build-source
docker-build-source: ## Build single-arch source image
@echo "🏗️ Building single-architecture source Docker image..."
@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
DOCKER_BUILDKIT=1 $(DOCKER_CLI) build \
--build-arg BUILDKIT_INLINE_CACHE=1 \
-f $(DOCKERFILE_SOURCE) -t rustfs:source .
-22
View File
@@ -1,22 +0,0 @@
## —— Docker-based build (alternative approach) ----------------------------------------------------
# Usage: make BUILD_OS=ubuntu22.04 build-docker
# Output: target/ubuntu22.04/release/rustfs
.PHONY: build-docker
build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) .
$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
.PHONY: docker-inspect-multiarch
docker-inspect-multiarch: ## Check image architecture support
@if [ -z "$(IMAGE)" ]; then \
echo "❌ Error: Please specify image, example: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
exit 1; \
fi
@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
docker buildx imagetools inspect $(IMAGE)
-55
View File
@@ -1,55 +0,0 @@
## —— Local Native Build using build-rustfs.sh script (Recommended) --------------------------------
.PHONY: build
build: ## Build RustFS binary (includes console by default)
@echo "🔨 Building RustFS using build-rustfs.sh script..."
./build-rustfs.sh
.PHONY: build-dev
build-dev: ## Build RustFS in Development mode
@echo "🔨 Building RustFS in development mode..."
./build-rustfs.sh --dev
.PHONY: build-musl
build-musl: ## Build x86_64 musl version
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
./build-rustfs.sh --platform x86_64-unknown-linux-musl
.PHONY: build-gnu
build-gnu: ## Build x86_64 GNU version
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
./build-rustfs.sh --platform x86_64-unknown-linux-gnu
.PHONY: build-musl-arm64
build-musl-arm64: ## Build aarch64 musl version
@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
./build-rustfs.sh --platform aarch64-unknown-linux-musl
.PHONY: build-gnu-arm64
build-gnu-arm64: ## Build aarch64 GNU version
@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
./build-rustfs.sh --platform aarch64-unknown-linux-gnu
.PHONY: build-cross-all
build-cross-all: core-deps ## Build binaries for all architectures
@echo "🔧 Building all target architectures..."
@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
@echo "🔨 Generating protobuf code..."
cargo run --bin gproto || true
@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
./build-rustfs.sh --platform x86_64-unknown-linux-musl
@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
./build-rustfs.sh --platform x86_64-unknown-linux-gnu
@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
./build-rustfs.sh --platform aarch64-unknown-linux-musl
@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
./build-rustfs.sh --platform aarch64-unknown-linux-gnu
-24
View File
@@ -1,24 +0,0 @@
## —— Check and Inform Dependencies ----------------------------------------------------------------
# Fatal check
# Checks all required dependencies and exits with error if not found
# (e.g., cargo, rustfmt)
check-%:
@command -v $* >/dev/null 2>&1 || { \
echo >&2 "❌ '$*' is not installed."; \
exit 1; \
}
# Warning-only check
# Checks for optional dependencies and issues a warning if not found
# (e.g., cargo-nextest for enhanced testing)
warn-%:
@command -v $* >/dev/null 2>&1 || { \
echo >&2 "⚠️ '$*' is not installed."; \
}
# For checking dependencies use check-<dep-name> or warn-<dep-name>
.PHONY: core-deps fmt-deps test-deps
core-deps: check-cargo ## Check core dependencies
fmt-deps: check-rustfmt ## Check lint and formatting dependencies
test-deps: warn-cargo-nextest ## Check tests dependencies
-6
View File
@@ -1,6 +0,0 @@
## —— Deploy using dev_deploy.sh script ------------------------------------------------------------
.PHONY: deploy-dev
deploy-dev: build-musl ## Deploy to dev server
@echo "🚀 Deploying to dev server: $${IP}"
./scripts/dev_deploy.sh $${IP}
-38
View File
@@ -1,38 +0,0 @@
## —— Help, Help Build and Help Docker -------------------------------------------------------------
.PHONY: help
help: ## Shows This Help Menu
echo -e "$$HEADER"
grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} /^## / {printf "\n${green}%s${reset}\n", $$0; next} {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}'
.PHONY: help-build
help-build: ## Shows RustFS build help
@echo ""
@echo "💡 build-rustfs.sh script provides more options, smart detection and binary verification"
@echo ""
@echo "🔧 Direct usage of build-rustfs.sh script:"
@echo ""
@echo " ./build-rustfs.sh --help # View script help"
@echo " ./build-rustfs.sh --no-console # Build without console resources"
@echo " ./build-rustfs.sh --force-console-update # Force update console resources"
@echo " ./build-rustfs.sh --dev # Development mode build"
@echo " ./build-rustfs.sh --sign # Sign binary files"
@echo " ./build-rustfs.sh --platform x86_64-unknown-linux-gnu # Specify target platform"
@echo " ./build-rustfs.sh --skip-verification # Skip binary verification"
@echo ""
.PHONY: help-docker
help-docker: ## Shows docker environment and suggestion help
@echo ""
@echo "📋 Environment Variables:"
@echo " REGISTRY Image registry address (required for push)"
@echo " DOCKERHUB_USERNAME Docker Hub username"
@echo " DOCKERHUB_TOKEN Docker Hub access token"
@echo " GITHUB_TOKEN GitHub access token"
@echo ""
@echo "💡 Suggestions:"
@echo " Production use: Use docker-buildx* commands (based on precompiled binaries)"
@echo " Local development: Use docker-dev* commands (build from source)"
@echo " Development environment: Use dev-env-* commands to manage dev containers"
@echo ""
-27
View File
@@ -1,27 +0,0 @@
## —— Code quality and Formatting ------------------------------------------------------------------
.PHONY: fmt
fmt: core-deps fmt-deps ## Format code
@echo "🔧 Formatting code..."
cargo fmt --all
.PHONY: fmt-check
fmt-check: core-deps fmt-deps ## Check code formatting
@echo "📝 Checking code formatting..."
cargo fmt --all --check
.PHONY: clippy-check
clippy-check: core-deps ## Run clippy checks
@echo "🔍 Running clippy checks..."
cargo clippy --fix --allow-dirty
cargo clippy --all-targets --all-features -- -D warnings
.PHONY: unsafe-code-check
unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
@echo "🔒 Checking unsafe_code allowances..."
./scripts/check_unsafe_code_allowances.sh
.PHONY: compilation-check
compilation-check: core-deps ## Run compilation check
@echo "🔨 Running compilation check..."
cargo check --all-targets
-11
View File
@@ -1,11 +0,0 @@
## —— Pre Commit Checks ----------------------------------------------------------------------------
.PHONY: setup-hooks
setup-hooks: ## Set up git hooks
@echo "🔧 Setting up git hooks..."
chmod +x .git/hooks/pre-commit
@echo "✅ Git hooks setup complete!"
.PHONY: pre-commit
pre-commit: fmt unsafe-code-check clippy-check compilation-check test ## Run pre-commit checks
@echo "✅ All pre-commit checks passed!"
-27
View File
@@ -1,27 +0,0 @@
## —— Tests and e2e test ---------------------------------------------------------------------------
TEST_THREADS ?= 1
.PHONY: script-tests
script-tests: ## Run shell script tests
@echo "Running script tests..."
./scripts/test_build_rustfs_options.sh
.PHONY: test
test: core-deps test-deps script-tests ## Run all tests
@echo "🧪 Running tests..."
@if command -v cargo-nextest >/dev/null 2>&1; then \
cargo nextest run --all --exclude e2e_test; \
else \
echo "️ cargo-nextest not found; falling back to 'cargo test'"; \
cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
fi
cargo test --all --doc
.PHONY: e2e-server
e2e-server: ## Run e2e-server tests
sh $(shell pwd)/scripts/run.sh
.PHONY: probe-e2e
probe-e2e: ## Probe e2e tests
sh $(shell pwd)/scripts/probe.sh
-131
View File
@@ -1,131 +0,0 @@
# RustFS Docker Infrastructure
This directory contains the complete Docker infrastructure for building, deploying, and monitoring RustFS. It provides ready-to-use configurations for development, testing, and production-grade observability.
## 📂 Directory Structure
| Directory | Description | Status |
| :--- | :--- | :--- |
| **[`observability/`](observability/README.md)** | **[RECOMMENDED]** Full-stack observability (Prometheus, Grafana, Tempo, Loki). | ✅ Production-Ready |
| **[`compose/`](compose/README.md)** | Specialized setups (e.g., 4-node distributed cluster testing). | ⚠️ Testing Only |
| **[`mqtt/`](mqtt/README.md)** | EMQX Broker configuration for MQTT integration testing. | 🧪 Development |
| **[`openobserve-otel/`](openobserve-otel/README.md)** | Alternative lightweight observability stack using OpenObserve. | 🔄 Alternative |
---
## 📄 Root Directory Files
The following files in the project root are essential for Docker operations:
### Build Scripts & Dockerfiles
| File | Description | Usage |
| :--- | :--- | :--- |
| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
### Docker Compose Configurations
| File | Description | Usage |
| :--- | :--- | :--- |
| **`docker-compose.yml`** | **Main Development Setup**<br>Comprehensive setup with profiles for development, observability, and proxying. | `docker compose up -d`<br>`docker compose --profile observability up -d` |
| **`docker-compose-simple.yml`** | **Quick Start Setup**<br>Minimal configuration running a single RustFS instance with 4 volumes. Perfect for first-time users. | `docker compose -f docker-compose-simple.yml up -d` |
---
## 🌟 Observability Stack (Recommended)
Located in: [`.docker/observability/`](observability/README.md)
We provide a comprehensive, industry-standard observability stack designed for deep insights into RustFS performance. This is the recommended setup for both development and production monitoring.
### Components
- **Metrics**: Prometheus (Collection) + Grafana (Visualization)
- **Traces**: Tempo (Storage) + Jaeger (UI)
- **Logs**: Loki
- **Ingestion**: OpenTelemetry Collector
### Key Features
- **Full Persistence**: All metrics, logs, and traces are saved to Docker volumes, ensuring no data loss on restarts.
- **Correlation**: Seamlessly jump between Logs, Traces, and Metrics in Grafana.
- **High Performance**: Optimized configurations for batching, compression, and memory management.
### Quick Start
```bash
cd .docker/observability
docker compose up -d
```
---
## 🧪 Specialized Environments
Located in: [`.docker/compose/`](compose/README.md)
These configurations are tailored for specific testing scenarios that require complex topologies.
### Distributed Cluster (4-Nodes)
Simulates a real-world distributed environment with 4 RustFS nodes running locally.
```bash
docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
```
### Integrated Observability Test
A self-contained environment running 4 RustFS nodes alongside the full observability stack. Useful for end-to-end testing of telemetry.
```bash
docker compose -f .docker/compose/docker-compose.observability.yaml up -d
```
---
## 📡 MQTT Integration
Located in: [`.docker/mqtt/`](mqtt/README.md)
Provides an EMQX broker for testing RustFS MQTT features.
### Quick Start
```bash
cd .docker/mqtt
docker compose up -d
```
- **Dashboard**: [http://localhost:18083](http://localhost:18083) (Default: `admin` / `public`)
- **MQTT Port**: `1883`
---
## 👁️ Alternative: OpenObserve
Located in: [`.docker/openobserve-otel/`](openobserve-otel/README.md)
For users preferring a lightweight, all-in-one solution, we support OpenObserve. It combines logs, metrics, and traces into a single binary and UI.
### Quick Start
```bash
cd .docker/openobserve-otel
docker compose up -d
```
---
## 🔧 Common Operations
### Cleaning Up
To stop all containers and remove volumes (**WARNING**: deletes all persisted data):
```bash
docker compose down -v
```
### Viewing Logs
To follow logs for a specific service:
```bash
docker compose logs -f [service_name]
```
### Checking Status
To see the status of all running containers:
```bash
docker compose ps
```
-102
View File
@@ -1,102 +0,0 @@
# Specialized Docker Compose Configurations
This directory contains specialized Docker Compose configurations for specific testing scenarios.
## ⚠️ Important Note
**For Observability:**
We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.
## 📁 Configuration Files
### Cluster Testing
- **`docker-compose.cluster.yaml`**
- **Purpose**: Simulates a 4-node RustFS distributed cluster.
- **Use Case**: Testing distributed storage logic, consensus, and failover.
- **Nodes**: 4 RustFS instances.
- **Storage**: Uses local HTTP endpoints.
### Legacy / Minimal Observability
- **`docker-compose.observability.yaml`**
- **Purpose**: A minimal observability setup.
- **Status**: **Deprecated**. Please use `../observability/docker-compose.yml` instead.
## 🚀 Usage Examples
### Cluster Testing
To start a 4-node cluster for distributed testing:
```bash
# From project root
docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
```
### Script-Based 4-Node Validation (Recommended)
Use the local validation script when you need local-source image build, failover checks,
and benchmark workflow in one command:
```bash
# Default mode: WAIT_PROBE_MODE=service
# This avoids false negatives where /health/ready remains 503 locally
# while the service path is already available.
./scripts/run_four_node_cluster_failover_bench.sh
```
Strict mode is available when you explicitly want `/health/ready == 200` as the gate:
```bash
WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
```
### Profiling + Trace Validation
The profiling-focused 4-node compose keeps profiling enabled and points RustFS
to an OTLP/HTTP collector endpoint:
```bash
docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
```
Important behavior notes:
- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
- Startup usually produces logs and metrics first. That does not guarantee
visible traces yet.
- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
`RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.
Minimal trace verification flow:
```bash
# 1. Start the profiling compose with richer span visibility.
RUSTFS_OBS_LOGGER_LEVEL=debug \
docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
# 2. Generate real request traffic after startup.
curl -I http://127.0.0.1:9000/health
curl -I http://127.0.0.1:9000/health/ready
# 3. Then inspect Tempo or Jaeger.
# Grafana: http://localhost:3000
# Jaeger: http://localhost:16686
```
If logs and metrics are present but traces are sparse, the most common cause is
"no real request traffic yet" or "`info` level filtered nested spans", not an
OTLP routing failure.
### (Deprecated) Minimal Observability
```bash
# From project root
docker compose -f .docker/compose/docker-compose.observability.yaml up -d
```
@@ -1,236 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Profiling-first 4-node local-build compose.
#
# Goals:
# - force linux/amd64 runtime/build on Apple Silicon hosts;
# - enable RustFS built-in CPU profiling;
# - keep all tuning knobs host-overridable via env.
#
# Observability notes:
# - `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS appends
# `/v1/traces`, `/v1/metrics`, and `/v1/logs` automatically.
# - Logs and metrics usually appear during startup. Traces mainly appear after
# real HTTP/S3/gRPC requests create spans.
# - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request trace span but
# filters many `debug`-level nested spans. Use `debug` when validating trace
# richness rather than collector reachability.
services:
node1:
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
build:
context: ../..
dockerfile: Dockerfile.source
hostname: node1
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
# should show richer nested spans during request-path verification.
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- node1_data_0:/data/rustfs0
- node1_data_1:/data/rustfs1
- node1_data_2:/data/rustfs2
- node1_data_3:/data/rustfs3
ports:
- "9000:9000"
networks:
- rustfs-cluster-net
node2:
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
build:
context: ../..
dockerfile: Dockerfile.source
hostname: node2
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
# should show richer nested spans during request-path verification.
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- node2_data_0:/data/rustfs0
- node2_data_1:/data/rustfs1
- node2_data_2:/data/rustfs2
- node2_data_3:/data/rustfs3
ports:
- "9001:9000"
networks:
- rustfs-cluster-net
node3:
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
build:
context: ../..
dockerfile: Dockerfile.source
hostname: node3
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
# should show richer nested spans during request-path verification.
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- node3_data_0:/data/rustfs0
- node3_data_1:/data/rustfs1
- node3_data_2:/data/rustfs2
- node3_data_3:/data/rustfs3
ports:
- "9002:9000"
networks:
- rustfs-cluster-net
node4:
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
build:
context: ../..
dockerfile: Dockerfile.source
hostname: node4
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
# `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
# should show richer nested spans during request-path verification.
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
- RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
- RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
- RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
- RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
- RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
- RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
- RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
- RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- node4_data_0:/data/rustfs0
- node4_data_1:/data/rustfs1
- node4_data_2:/data/rustfs2
- node4_data_3:/data/rustfs3
ports:
- "9003:9000"
networks:
- rustfs-cluster-net
volumes:
node1_data_0:
node1_data_1:
node1_data_2:
node1_data_3:
node2_data_0:
node2_data_1:
node2_data_2:
node2_data_3:
node3_data_0:
node3_data_1:
node3_data_2:
node3_data_3:
node4_data_0:
node4_data_1:
node4_data_2:
node4_data_3:
networks:
rustfs-cluster-net:
driver: bridge
@@ -1,164 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
services:
node1:
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
build:
context: ../..
dockerfile: Dockerfile.source
hostname: node1
environment:
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- node1_data_0:/data/rustfs0
- node1_data_1:/data/rustfs1
- node1_data_2:/data/rustfs2
- node1_data_3:/data/rustfs3
ports:
- "9000:9000"
networks:
- rustfs-cluster-net
node2:
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
build:
context: ../..
dockerfile: Dockerfile.source
hostname: node2
environment:
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- node2_data_0:/data/rustfs0
- node2_data_1:/data/rustfs1
- node2_data_2:/data/rustfs2
- node2_data_3:/data/rustfs3
ports:
- "9001:9000"
networks:
- rustfs-cluster-net
node3:
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
build:
context: ../..
dockerfile: Dockerfile.source
hostname: node3
environment:
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- node3_data_0:/data/rustfs0
- node3_data_1:/data/rustfs1
- node3_data_2:/data/rustfs2
- node3_data_3:/data/rustfs3
ports:
- "9002:9000"
networks:
- rustfs-cluster-net
node4:
image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
build:
context: ../..
dockerfile: Dockerfile.source
hostname: node4
environment:
- RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
- RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
- RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
- RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
- RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
- RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- node4_data_0:/data/rustfs0
- node4_data_1:/data/rustfs1
- node4_data_2:/data/rustfs2
- node4_data_3:/data/rustfs3
ports:
- "9003:9000"
networks:
- rustfs-cluster-net
volumes:
node1_data_0:
node1_data_1:
node1_data_2:
node1_data_3:
node2_data_0:
node2_data_1:
node2_data_2:
node2_data_3:
node3_data_0:
node3_data_1:
node3_data_2:
node3_data_3:
node4_data_0:
node4_data_1:
node4_data_2:
node4_data_3:
networks:
rustfs-cluster-net:
driver: bridge
@@ -1,56 +0,0 @@
services:
node1:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
- RUSTFS_RUNTIME_WORKER_THREADS=16
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
node2:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
- RUSTFS_RUNTIME_WORKER_THREADS=16
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
node3:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
- RUSTFS_RUNTIME_WORKER_THREADS=16
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
node4:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
- RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
- RUSTFS_RUNTIME_WORKER_THREADS=16
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
@@ -1,56 +0,0 @@
services:
node1:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
- RUSTFS_RUNTIME_WORKER_THREADS=24
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
node2:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
- RUSTFS_RUNTIME_WORKER_THREADS=24
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
node3:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
- RUSTFS_RUNTIME_WORKER_THREADS=24
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
node4:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
- RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
- RUSTFS_RUNTIME_WORKER_THREADS=24
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
@@ -1,56 +0,0 @@
services:
node1:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
- RUSTFS_RUNTIME_WORKER_THREADS=20
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
node2:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
- RUSTFS_RUNTIME_WORKER_THREADS=20
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
node3:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
- RUSTFS_RUNTIME_WORKER_THREADS=20
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
node4:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
- RUSTFS_RUNTIME_WORKER_THREADS=20
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
@@ -1,56 +0,0 @@
services:
node1:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
- RUSTFS_RUNTIME_WORKER_THREADS=12
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
node2:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
- RUSTFS_RUNTIME_WORKER_THREADS=12
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
node3:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
- RUSTFS_RUNTIME_WORKER_THREADS=12
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
node4:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
- RUSTFS_RUNTIME_WORKER_THREADS=12
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
@@ -1,56 +0,0 @@
services:
node1:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
- RUSTFS_RUNTIME_WORKER_THREADS=6
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
node2:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
- RUSTFS_RUNTIME_WORKER_THREADS=6
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
node3:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
- RUSTFS_RUNTIME_WORKER_THREADS=6
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
node4:
environment:
- RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
- RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
- RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
- RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
- RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
- RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
- RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
- RUSTFS_RUNTIME_WORKER_THREADS=6
- RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
- RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
- RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
@@ -1,82 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
services:
node0:
image: rustfs/rustfs:latest # Replace with your image name and label
container_name: node0
hostname: node0
environment:
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
platform: linux/amd64
ports:
- "9000:9000" # Map port 9001 of the host to port 9000 of the container
volumes:
- ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
command: "/app/rustfs"
node1:
image: rustfs/rustfs:latest
container_name: node1
hostname: node1
environment:
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
platform: linux/amd64
ports:
- "9001:9000" # Map port 9002 of the host to port 9000 of the container
volumes:
- ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
command: "/app/rustfs"
node2:
image: rustfs/rustfs:latest
container_name: node2
hostname: node2
environment:
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
platform: linux/amd64
ports:
- "9002:9000" # Map port 9003 of the host to port 9000 of the container
volumes:
- ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
command: "/app/rustfs"
node3:
image: rustfs/rustfs:latest
container_name: node3
hostname: node3
environment:
- RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
platform: linux/amd64
ports:
- "9003:9000" # Map port 9004 of the host to port 9000 of the container
volumes:
- ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
command: "/app/rustfs"
@@ -1,282 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
services:
# --- Observability Stack ---
tempo-init:
image: busybox:latest
command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
volumes:
- tempo-data:/var/tempo
user: root
networks:
- rustfs-network
restart: "no"
tempo:
image: grafana/tempo:2.10.5
user: "10001"
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ../../.docker/observability/tempo.yaml:/etc/tempo.yaml:ro
- tempo-data:/var/tempo
ports:
- "3200:3200" # tempo
- "4317" # otlp grpc
- "4318" # otlp http
- "7946" # memberlist
restart: unless-stopped
networks:
- rustfs-network
depends_on:
tempo-init:
condition: service_completed_successfully
healthcheck:
test: [ "CMD", "/tempo", "-version" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
environment:
- TZ=Asia/Shanghai
volumes:
- ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
ports:
- "1888:1888" # pprof
- "8888:8888" # Prometheus metrics for Collector
- "8889:8889" # Prometheus metrics for application indicators
- "13133:13133" # health check
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "55679:55679" # zpages
networks:
- rustfs-network
depends_on:
- tempo
- jaeger
- prometheus
- loki
restart: unless-stopped
healthcheck:
test: [ "CMD", "/otelcol-contrib", "--version" ]
interval: 10s
timeout: 5s
retries: 3
jaeger:
image: jaegertracing/jaeger:latest
environment:
- TZ=Asia/Shanghai
- SPAN_STORAGE_TYPE=badger
- BADGER_EPHEMERAL=false
- BADGER_DIRECTORY_VALUE=/badger/data
- BADGER_DIRECTORY_KEY=/badger/key
- COLLECTOR_OTLP_ENABLED=true
volumes:
- ../../.docker/observability/jaeger.yaml:/etc/jaeger/config.yml:ro
- jaeger-data:/badger
ports:
- "16686:16686" # Web UI
- "14269:14269" # Admin/Metrics
- "4317" # otlp grpc
- "4318" # otlp http
command: [ "--config", "/etc/jaeger/config.yml" ]
networks:
- rustfs-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
prometheus:
image: prom/prometheus:latest
environment:
- TZ=Asia/Shanghai
volumes:
- ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ../../.docker/observability/prometheus-rules:/etc/prometheus/rules:ro
- prometheus-data:/prometheus
ports:
- "9090:9090"
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--web.enable-otlp-receiver'
- '--web.enable-remote-write-receiver'
- '--enable-feature=promql-experimental-functions'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention.time=30d'
networks:
- rustfs-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
interval: 10s
timeout: 5s
retries: 3
loki:
image: grafana/loki:latest
environment:
- TZ=Asia/Shanghai
volumes:
- ../../.docker/observability/loki.yaml:/etc/loki/loki.yaml:ro
- loki-data:/loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/loki.yaml
networks:
- rustfs-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
interval: 15s
timeout: 10s
retries: 5
start_period: 60s
pyroscope:
image: grafana/pyroscope:latest
ports:
- "4040:4040"
command:
- -self-profiling.disable-push=true
networks:
- rustfs-network
restart: unless-stopped
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000" # Web UI
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_SECURITY_ADMIN_USER=admin
- TZ=Asia/Shanghai
- GF_INSTALL_PLUGINS=grafana-pyroscope-datasource
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
networks:
- rustfs-network
volumes:
- ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
- ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
- grafana-data:/var/lib/grafana
depends_on:
- prometheus
- tempo
- loki
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
interval: 10s
timeout: 5s
retries: 3
# --- RustFS Cluster ---
node1:
build:
context: ../..
dockerfile: Dockerfile.source
container_name: node1
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_LOGGER_LEVEL=debug
platform: linux/amd64
ports:
- "9001:9000"
networks:
- rustfs-network
depends_on:
- otel-collector
node2:
build:
context: ../..
dockerfile: Dockerfile.source
container_name: node2
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_LOGGER_LEVEL=debug
platform: linux/amd64
ports:
- "9002:9000"
networks:
- rustfs-network
depends_on:
- otel-collector
node3:
build:
context: ../..
dockerfile: Dockerfile.source
container_name: node3
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_LOGGER_LEVEL=debug
platform: linux/amd64
ports:
- "9003:9000"
networks:
- rustfs-network
depends_on:
- otel-collector
node4:
build:
context: ../..
dockerfile: Dockerfile.source
container_name: node4
environment:
- RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
- RUSTFS_ADDRESS=:9000
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_LOGGER_LEVEL=debug
platform: linux/amd64
ports:
- "9004:9000"
networks:
- rustfs-network
depends_on:
- otel-collector
volumes:
prometheus-data:
tempo-data:
loki-data:
jaeger-data:
grafana-data:
networks:
rustfs-network:
driver: bridge
name: "network_rustfs_config"
driver_opts:
com.docker.network.enable_ipv6: "true"
-30
View File
@@ -1,30 +0,0 @@
# MQTT Broker (EMQX)
This directory contains the configuration for running an EMQX MQTT broker, which can be used for testing RustFS's MQTT integration.
## 🚀 Quick Start
To start the EMQX broker:
```bash
docker compose up -d
```
## 📊 Access
- **Dashboard**: [http://localhost:18083](http://localhost:18083)
- **Default Credentials**: `admin` / `public`
- **MQTT Port**: `1883`
- **WebSocket Port**: `8083`
## 🛠️ Configuration
The `docker-compose.yml` file sets up a single-node EMQX instance.
- **Persistence**: Data is not persisted by default (for testing).
- **Network**: Uses the default bridge network.
## 📝 Notes
- This setup is intended for development and testing purposes.
- For production deployments, please refer to the official [EMQX Documentation](https://www.emqx.io/docs/en/latest/).
-37
View File
@@ -1,37 +0,0 @@
# 节点配置
node.name = "emqx@127.0.0.1"
node.cookie = "aBcDeFgHiJkLmNoPqRsTuVwXyZ012345"
node.data_dir = "/opt/emqx/data"
# 日志配置
log.console = {level = info, enable = true}
log.file = {path = "/opt/emqx/log/emqx.log", enable = true, level = info}
# MQTT TCP 监听器
listeners.tcp.default = {bind = "0.0.0.0:1883", max_connections = 1000000, enable = true}
# MQTT SSL 监听器
listeners.ssl.default = {bind = "0.0.0.0:8883", enable = false}
# MQTT WebSocket 监听器
listeners.ws.default = {bind = "0.0.0.0:8083", enable = true}
# MQTT WebSocket SSL 监听器
listeners.wss.default = {bind = "0.0.0.0:8084", enable = false}
# 管理控制台
dashboard.listeners.http = {bind = "0.0.0.0:18083", enable = true}
# HTTP API
management.listeners.http = {bind = "0.0.0.0:8081", enable = true}
# 认证配置
authentication = [
{enable = true, mechanism = password_based, backend = built_in_database, user_id_type = username}
]
# 授权配置
authorization.sources = [{type = built_in_database, enable = true}]
# 持久化消息存储
message.storage.backend = built_in_database
-9
View File
@@ -1,9 +0,0 @@
-name emqx@127.0.0.1
-setcookie aBcDeFgHiJkLmNoPqRsTuVwXyZ012345
+P 2097152
+t 1048576
+zdbbl 32768
-kernel inet_dist_listen_min 6000
-kernel inet_dist_listen_max 6100
-smp enable
-mnesia dir "/opt/emqx/data/mnesia"
-74
View File
@@ -1,74 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
services:
emqx:
image: emqx/emqx:latest
container_name: emqx
restart: unless-stopped
environment:
- EMQX_NODE__NAME=emqx@127.0.0.1
- EMQX_NODE__COOKIE=aBcDeFgHiJkLmNoPqRsTuVwXyZ012345
- EMQX_NODE__DATA_DIR=/opt/emqx/data
- EMQX_LOG__CONSOLE__LEVEL=info
- EMQX_LOG__CONSOLE__ENABLE=true
- EMQX_LOG__FILE__PATH=/opt/emqx/log/emqx.log
- EMQX_LOG__FILE__LEVEL=info
- EMQX_LOG__FILE__ENABLE=true
- EMQX_LISTENERS__TCP__DEFAULT__BIND=0.0.0.0:1883
- EMQX_LISTENERS__TCP__DEFAULT__MAX_CONNECTIONS=1000000
- EMQX_LISTENERS__TCP__DEFAULT__ENABLE=true
- EMQX_LISTENERS__SSL__DEFAULT__BIND=0.0.0.0:8883
- EMQX_LISTENERS__SSL__DEFAULT__ENABLE=false
- EMQX_LISTENERS__WS__DEFAULT__BIND=0.0.0.0:8083
- EMQX_LISTENERS__WS__DEFAULT__ENABLE=true
- EMQX_LISTENERS__WSS__DEFAULT__BIND=0.0.0.0:8084
- EMQX_LISTENERS__WSS__DEFAULT__ENABLE=false
- EMQX_DASHBOARD__LISTENERS__HTTP__BIND=0.0.0.0:18083
- EMQX_DASHBOARD__LISTENERS__HTTP__ENABLE=true
- EMQX_MANAGEMENT__LISTENERS__HTTP__BIND=0.0.0.0:8081
- EMQX_MANAGEMENT__LISTENERS__HTTP__ENABLE=true
- EMQX_AUTHENTICATION__1__ENABLE=true
- EMQX_AUTHENTICATION__1__MECHANISM=password_based
- EMQX_AUTHENTICATION__1__BACKEND=built_in_database
- EMQX_AUTHENTICATION__1__USER_ID_TYPE=username
- EMQX_AUTHORIZATION__SOURCES__1__TYPE=built_in_database
- EMQX_AUTHORIZATION__SOURCES__1__ENABLE=true
ports:
- "1883:1883" # MQTT TCP
- "8883:8883" # MQTT SSL
- "8083:8083" # MQTT WebSocket
- "8084:8084" # MQTT WebSocket SSL
- "18083:18083" # Web 管理控制台
- "8081:8081" # HTTP API
volumes:
- ./data:/opt/emqx/data
- ./log:/opt/emqx/log
- ./config:/opt/emqx/etc
networks:
- mqtt-net
healthcheck:
test: [ "CMD", "/opt/emqx/bin/emqx_ctl", "status" ]
interval: 30s
timeout: 10s
retries: 3
logging:
driver: "json-file"
options:
max-size: "100m"
max-file: "3"
networks:
mqtt-net:
driver: bridge
-29
View File
@@ -1,29 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
services:
emqx:
image: emqx/emqx:latest
container_name: emqx
ports:
- "1883:1883"
- "8083:8083"
- "8084:8084"
- "8883:8883"
- "18083:18083"
restart: unless-stopped
networks:
default:
driver: bridge
-93
View File
@@ -1,93 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
worker_processes auto;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log warn;
sendfile on;
keepalive_timeout 65;
# RustFS Server Block
server {
listen 80;
server_name localhost;
# Redirect HTTP to HTTPS (optional, uncomment if SSL is configured)
# return 301 https://$host$request_uri;
location / {
proxy_pass http://rustfs:9000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# S3 specific headers
proxy_set_header X-Amz-Date $http_x_amz_date;
proxy_set_header Authorization $http_authorization;
# Disable buffering for large uploads
proxy_request_buffering off;
client_max_body_size 0;
}
location /rustfs/console {
proxy_pass http://rustfs:9001;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
# SSL Configuration (Example)
# server {
# listen 443 ssl;
# server_name localhost;
#
# ssl_certificate /etc/nginx/ssl/server.crt;
# ssl_certificate_key /etc/nginx/ssl/server.key;
#
# # Restrict to modern TLS versions and ciphers. Operators copying this
# # example must keep at least these directives — without them, nginx
# # may negotiate older protocol versions that have known weaknesses.
# ssl_protocols TLSv1.2 TLSv1.3;
# ssl_prefer_server_ciphers on;
# ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
# ssl_session_timeout 1d;
# ssl_session_cache shared:SSL:10m;
# ssl_session_tickets off;
# # add_header Strict-Transport-Security "max-age=63072000" always;
#
# location / {
# proxy_pass http://rustfs:9000;
# ...
# }
# }
}
View File
-5
View File
@@ -1,5 +0,0 @@
jaeger-data/*
loki-data/*
prometheus-data/*
tempo-data/*
grafana-data/*
-145
View File
@@ -1,145 +0,0 @@
# RustFS Observability Stack
This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.
## Components
The stack is composed of the following best-in-class open-source components:
- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
- **Grafana** (v11.1.0): The leading platform for observability visualization.
- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
## Architecture
1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
2. **Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
- **Traces** -> **Tempo** (Primary) & **Jaeger** (Secondary/Optional)
- **Metrics** -> **Prometheus** (via scraping the Collector's exporter)
- **Logs** -> **Loki**
3. **Visualization**: **Grafana** connects to all backends (Prometheus, Tempo, Loki, Jaeger) to provide a unified dashboard experience.
## Features
- **Full Persistence**: All data (Metrics, Logs, Traces) is persisted to Docker volumes, ensuring no data loss on restart.
- **Correlation**: Seamless navigation between Metrics, Logs, and Traces in Grafana.
- Jump from a Metric spike to relevant Traces.
- Jump from a Trace to relevant Logs.
- **High Performance**: Optimized configurations for batching, compression, and memory management.
- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
## Quick Start
### Prerequisites
- Docker
- Docker Compose
### Deploy
Run the following command to start the entire stack:
```bash
docker compose up -d
```
### High Availability Tempo
The default `docker-compose.yml` is the single-node stack.
If you need the Kafka-backed HA Tempo configuration, start it with:
```bash
docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
```
### Access Dashboards
| Service | URL | Credentials | Description |
| :------------- | :----------------------------------------------- | :---------------- | :----------------------------- |
| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | Main visualization hub. |
| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | Metric queries and status. |
| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | Secondary trace visualization. |
| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo status/metrics. |
## Configuration
### Data Persistence
Data is stored in the following Docker volumes:
- `prometheus-data`: Prometheus metrics
- `tempo-data`: Tempo traces (WAL and Blocks)
- `loki-data`: Loki logs (Chunks and Rules)
- `jaeger-data`: Jaeger traces (Badger DB)
To clear all data:
```bash
docker compose down -v
```
### Customization
- **Prometheus**: Edit `prometheus.yml` to add scrape targets or alerting rules.
- **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
- **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
### Verifying RustFS Traces
When RustFS points `RUSTFS_OBS_ENDPOINT` at this stack, treat the value as the
OTLP/HTTP base URL, for example:
```bash
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
```
RustFS automatically expands that base URL to:
- `/v1/traces`
- `/v1/metrics`
- `/v1/logs`
Important behavior notes:
- Logs and metrics usually appear during startup, so seeing those two signals
first is expected.
- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
startup, because request-path spans are created on demand.
- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
many nested `debug` spans. If Tempo or Jaeger looks sparse, retry with
`RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting collector or Tempo issues.
Minimal validation flow:
```bash
# 1. Start this observability stack.
docker compose up -d
# 2. Start RustFS with OTLP/HTTP export and richer span visibility.
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
export RUSTFS_OBS_LOGGER_LEVEL=debug
# 3. Generate real request traffic.
curl -I http://127.0.0.1:9000/health
curl -I http://127.0.0.1:9000/health/ready
# 4. Inspect Grafana or Jaeger.
# Grafana: http://localhost:3000
# Jaeger: http://localhost:16686
```
If logs and metrics are present but traces are sparse, the most common cause is
"no real request traffic yet" or "`info` level filtered nested spans", not an
OTLP routing failure.
## Troubleshooting
- **Service Health**: Check the health of services using `docker compose ps`.
- **Logs**: View logs for a specific service using `docker compose logs -f <service_name>`.
- **Otel Collector**: Check `http://localhost:13133` for health status and `http://localhost:1888/debug/pprof/` for profiling.
-141
View File
@@ -1,141 +0,0 @@
# RustFS 可观测性技术栈
本目录包含 RustFS 的全面可观测性技术栈,旨在提供对应用程序性能、日志和追踪的深入洞察。
## 组件
该技术栈由以下一流的开源组件组成:
- **Prometheus** (v2.53.1): 行业标准的指标收集和告警工具。
- **Grafana** (v11.1.0): 领先的可观测性可视化平台。
- **Loki** (v3.1.0): 水平可扩展、高可用、多租户的日志聚合系统。
- **Tempo** (v2.5.0): 高吞吐量、最小依赖的分布式追踪后端。
- **Jaeger** (v1.59.0): 分布式追踪系统(配置为辅助 UI/存储)。
- **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
默认情况下,这套技术栈使用 Tempo 单二进制模式,不依赖 Kafka/Redpanda。
如果需要基于 Kafka 的 HA Tempo 路径,请使用 `docker-compose-example-for-rustfs.yml` 配合 `docker-compose-tempo-ha-override.yml`
## 架构
1. **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据(指标、日志、追踪)发送到 **OpenTelemetry Collector**
2. **处理与导出**: Collector 处理数据(批处理、内存限制)并将其导出到相应的后端:
- **追踪** -> **Tempo** (主要) & **Jaeger** (辅助/可选)
- **指标** -> **Prometheus** (通过抓取 Collector 的导出器)
- **日志** -> **Loki**
3. **可视化**: **Grafana** 连接到所有后端(Prometheus, Tempo, Loki, Jaeger),提供统一的仪表盘体验。
## 特性
- **完全持久化**: 所有数据(指标、日志、追踪)都持久化到 Docker 卷,确保重启后无数据丢失。
- **关联性**: 在 Grafana 中实现指标、日志和追踪之间的无缝导航。
- 从指标峰值跳转到相关追踪。
- 从追踪跳转到相关日志。
- **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
- **标准化协议**: 完全基于 OpenTelemetry 标准构建。
## 快速开始
### 前置条件
- Docker
- Docker Compose
### 部署
运行以下命令启动整个技术栈:
```bash
docker compose up -d
```
### Tempo 高可用模式
默认的 `docker-compose.yml` 对应单机栈。
如果需要基于 Kafka 的 HA Tempo 配置,请使用:
```bash
docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
```
### 访问仪表盘
| 服务 | URL | 凭据 | 描述 |
| :--- | :--- | :--- | :--- |
| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | 主要可视化中心。 |
| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | 指标查询和状态。 |
| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | 辅助追踪可视化。 |
| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo 状态/指标。 |
## 配置
### 数据持久化
数据存储在以下 Docker 卷中:
- `prometheus-data`: Prometheus 指标
- `tempo-data`: Tempo 追踪 (WAL 和 Blocks)
- `loki-data`: Loki 日志 (Chunks 和 Rules)
- `jaeger-data`: Jaeger 追踪 (Badger DB)
要清除所有数据:
```bash
docker compose down -v
```
### 自定义
- **Prometheus**: 编辑 `prometheus.yml` 以添加抓取目标或告警规则。
- **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
- **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
### 验证 RustFS Trace
当 RustFS 将 `RUSTFS_OBS_ENDPOINT` 指向这套技术栈时,应将该值视为
OTLP/HTTP 的基础 URL,例如:
```bash
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
```
RustFS 会自动在该基础 URL 后补全:
- `/v1/traces`
- `/v1/metrics`
- `/v1/logs`
需要注意:
- 启动阶段通常会先看到日志和指标,因此“先有日志/指标、后有 trace”是正常现象。
- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量,因为请求路径上的 span 是按需创建的。
- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span,但会过滤掉很多 `debug` 级别的嵌套 span。
如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏,建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`,再判断是否是 collector 或 Tempo 问题。
最小验证流程:
```bash
# 1. 启动本目录下的可观测性技术栈。
docker compose up -d
# 2. 以 OTLP/HTTP 导出方式启动 RustFS,并提高 span 可见性。
export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
export RUSTFS_OBS_LOGGER_LEVEL=debug
# 3. 产生真实请求流量。
curl -I http://127.0.0.1:9000/health
curl -I http://127.0.0.1:9000/health/ready
# 4. 到 Grafana 或 Jaeger 中检查。
# Grafana: http://localhost:3000
# Jaeger: http://localhost:16686
```
如果日志和指标已经正常,但 trace 仍然稀疏,最常见的原因通常是
“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”,而不是 OTLP 路由失败。
## 故障排除
- **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
- **日志**: 使用 `docker compose logs -f <service_name>` 查看特定服务的日志。
- **Otel Collector**: 检查 `http://localhost:13133` 获取健康状态,检查 `http://localhost:1888/debug/pprof/` 进行性能分析。
@@ -1,268 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
services:
rustfs:
security_opt:
- "no-new-privileges:true"
image: rustfs/rustfs:latest
container_name: rustfs-server
ports:
- "9000:9000" # S3 API port
- "9001:9001" # Console port
environment:
- RUSTFS_VOLUMES=/data/rustfs
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CORS_ALLOWED_ORIGINS=*
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
- RUSTFS_ACCESS_KEY=rustfsadmin
- RUSTFS_SECRET_KEY=rustfsadmin
- RUSTFS_OBS_LOGGER_LEVEL=info
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
volumes:
- rustfs-data:/data/rustfs
networks:
- otel-network
restart: unless-stopped
healthcheck:
test:
[
"CMD",
"sh",
"-c",
"curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
depends_on:
otel-collector:
condition: service_started
rustfs-init:
image: alpine
container_name: rustfs-init
volumes:
- rustfs-data:/data
networks:
- otel-network
command: >
sh -c "
chown -R 10001:10001 /data &&
echo 'Volume Permissions fixed' &&
exit 0
"
restart: no
# --- Tracing ---
tempo:
image: grafana/tempo:latest
container_name: tempo
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ./tempo.yaml:/etc/tempo.yaml:ro
- tempo-data:/var/tempo
ports:
- "3200:3200" # tempo
- "4317" # otlp grpc
- "4318" # otlp http
networks:
- otel-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "/tempo", "-version" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
redpanda:
image: redpandadata/redpanda:latest # for tempo ingest
container_name: redpanda
ports:
- "9092:9092"
networks:
- otel-network
restart: unless-stopped
command: >
redpanda start --overprovisioned
--mode=dev-container
--kafka-addr=PLAINTEXT://0.0.0.0:9092
--advertise-kafka-addr=PLAINTEXT://redpanda:9092
jaeger:
image: jaegertracing/jaeger:latest
container_name: jaeger
environment:
- SPAN_STORAGE_TYPE=badger
- BADGER_EPHEMERAL=false
- BADGER_DIRECTORY_VALUE=/badger/data
- BADGER_DIRECTORY_KEY=/badger/key
- COLLECTOR_OTLP_ENABLED=true
volumes:
- ./jaeger.yaml:/etc/jaeger/config.yml
- jaeger-data:/badger
ports:
- "16686:16686" # Web UI
- "14269:14269" # Admin/Metrics
- "4317" # otlp grpc
- "4318" # otlp http
command: [ "--config", "/etc/jaeger/config.yml" ]
networks:
- otel-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
# --- Metrics ---
prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
ports:
- "9090:9090"
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--web.enable-otlp-receiver" # Enable OTLP
- "--web.enable-remote-write-receiver" # Enable remote write
- "--enable-feature=promql-experimental-functions" # Enable info()
- "--storage.tsdb.retention.time=30d"
restart: unless-stopped
networks:
- otel-network
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
interval: 10s
timeout: 5s
retries: 3
# --- Logging ---
loki:
image: grafana/loki:latest
container_name: loki
volumes:
- ./loki.yaml:/etc/loki/loki.yaml:ro
- loki-data:/loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/loki.yaml
networks:
- otel-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
interval: 15s
timeout: 10s
retries: 5
start_period: 60s
# --- Collection ---
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
volumes:
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
ports:
- "1888:1888" # pprof
- "8888:8888" # Prometheus metrics for Collector
- "8889:8889" # Prometheus metrics for application indicators
- "13133:13133" # health check
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "55679:55679" # zpages
networks:
- otel-network
restart: unless-stopped
depends_on:
- tempo
- jaeger
- prometheus
- loki
healthcheck:
test: [ "CMD", "/otelcol-contrib", "--version" ]
interval: 10s
timeout: 5s
retries: 3
# --- Profiles ---
pyroscope:
image: grafana/pyroscope:latest
container_name: pyroscope
ports:
- "4040:4040"
command:
- -self-profiling.disable-push=true
networks:
- otel-network
restart: unless-stopped
# --- Visualization ---
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_SECURITY_ADMIN_USER=admin
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/etc/grafana/dashboards:ro
- grafana-data:/var/lib/grafana
networks:
- otel-network
restart: unless-stopped
depends_on:
- prometheus
- tempo
- loki
healthcheck:
test:
[ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
interval: 10s
timeout: 5s
retries: 3
volumes:
rustfs-data:
tempo-data:
jaeger-data:
prometheus-data:
loki-data:
grafana-data:
networks:
otel-network:
driver: bridge
name: "network_otel"
ipam:
config:
- subnet: 172.28.0.0/16
driver_opts:
com.docker.network.enable_ipv6: "true"
@@ -1,62 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Compose override file for High Availability Tempo setup
#
# Usage:
# docker-compose -f docker-compose-example-for-rustfs.yml \
# -f docker-compose-tempo-ha-override.yml up
services:
# Override Tempo to use high-availability configuration
tempo:
volumes:
- ./tempo-ha.yaml:/etc/tempo.yaml:ro
- tempo-data:/var/tempo
ports:
- "3200:3200" # Tempo HTTP
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "7946:7946" # Memberlist
- "14250:14250" # Jaeger gRPC
- "14268:14268" # Jaeger Thrift HTTP
- "9411:9411" # Zipkin
environment:
- TEMPO_MEMBERLIST_BIND_PORT=7946
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
depends_on:
- redpanda
volumes:
tempo-data:
driver: local
driver_opts:
type: tmpfs
device: tmpfs
o: "size=4g" # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
# Network configuration remains the same
# networks:
# otel-network:
# driver: bridge
# name: "network_otel"
# ipam:
# config:
# - subnet: 172.28.0.0/16
-211
View File
@@ -1,211 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
services:
# --- Tracing ---
tempo:
image: grafana/tempo:2.10.5
container_name: tempo
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ./tempo.yaml:/etc/tempo.yaml:ro
- tempo-data:/var/tempo
ports:
- "3200:3200" # tempo
- "4317" # otlp grpc
- "4318" # otlp http
- "7946" # memberlist
networks:
- otel-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "/tempo", "-version" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
vulture:
image: grafana/tempo-vulture:latest
restart: always
command:
[
"-prometheus-listen-address=:8080",
"-tempo-query-url=http://tempo:3200",
"-tempo-push-url=http://tempo:4317",
]
depends_on:
- tempo
jaeger:
image: jaegertracing/jaeger:latest
container_name: jaeger
environment:
- SPAN_STORAGE_TYPE=badger
- BADGER_EPHEMERAL=false
- BADGER_DIRECTORY_VALUE=/badger/data
- BADGER_DIRECTORY_KEY=/badger/key
- COLLECTOR_OTLP_ENABLED=true
volumes:
- ./jaeger.yaml:/etc/jaeger/config.yml
- jaeger-data:/badger
ports:
- "16686:16686" # Web UI
- "14269:14269" # Admin/Metrics
- "4317" # otlp grpc
- "4318" # otlp http
command: [ "--config", "/etc/jaeger/config.yml" ]
networks:
- otel-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
interval: 10s
timeout: 5s
retries: 3
start_period: 15s
# --- Metrics ---
prometheus:
image: prom/prometheus:latest
container_name: prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./prometheus-rules:/etc/prometheus/rules:ro
- prometheus-data:/prometheus
ports:
- "9090:9090"
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--web.enable-otlp-receiver" # Enable OTLP
- "--web.enable-remote-write-receiver" # Enable remote write
- "--enable-feature=promql-experimental-functions" # Enable info()
- "--storage.tsdb.retention.time=30d"
restart: unless-stopped
networks:
- otel-network
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
interval: 10s
timeout: 5s
retries: 3
# --- Logging ---
loki:
image: grafana/loki:latest
container_name: loki
volumes:
- ./loki.yaml:/etc/loki/loki.yaml:ro
- loki-data:/loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/loki.yaml
networks:
- otel-network
restart: unless-stopped
healthcheck:
test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
interval: 15s
timeout: 10s
retries: 5
start_period: 60s
# --- Collection ---
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
volumes:
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
ports:
- "1888:1888" # pprof
- "8888:8888" # Prometheus metrics for Collector
- "8889:8889" # Prometheus metrics for application indicators
- "13133:13133" # health check
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "55679:55679" # zpages
networks:
- otel-network
restart: unless-stopped
depends_on:
- tempo
- jaeger
- prometheus
- loki
healthcheck:
test: [ "CMD", "/otelcol-contrib", "--version" ]
interval: 10s
timeout: 5s
retries: 3
# --- Profiles ---
pyroscope:
image: grafana/pyroscope:latest
container_name: pyroscope
ports:
- "4040:4040"
command:
- -self-profiling.disable-push=true
networks:
- otel-network
restart: unless-stopped
# --- Visualization ---
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_SECURITY_ADMIN_USER=admin
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning:ro
- ./grafana/dashboards:/etc/grafana/dashboards:ro
- grafana-data:/var/lib/grafana
networks:
- otel-network
restart: unless-stopped
depends_on:
- prometheus
- tempo
- loki
healthcheck:
test:
[ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
interval: 10s
timeout: 5s
retries: 3
volumes:
tempo-data:
jaeger-data:
prometheus-data:
loki-data:
grafana-data:
networks:
otel-network:
driver: bridge
name: "network_otel"
ipam:
config:
- subnet: 172.28.0.0/16
driver_opts:
com.docker.network.enable_ipv6: "true"
File diff suppressed because it is too large Load Diff
@@ -1,25 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: 1
providers:
- name: "default"
orgId: 1
folder: ""
type: file
disableDeletion: false
updateIntervalSeconds: 10
options:
path: /etc/grafana/dashboards
@@ -1,98 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
uid: prometheus
url: http://prometheus:9090
access: proxy
isDefault: true
editable: false
jsonData:
httpMethod: GET
exemplarTraceIdDestinations:
- name: trace_id
datasourceUid: tempo
- name: Tempo
type: tempo
uid: tempo
access: proxy
url: http://tempo:3200
isDefault: false
editable: false
jsonData:
httpMethod: GET
serviceMap:
datasourceUid: prometheus
tracesToLogs:
datasourceUid: loki
tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
mappedTags: [ { key: 'service.name', value: 'app' } ]
spanStartTimeShift: '-1h'
spanEndTimeShift: '1h'
filterByTraceID: true
filterBySpanID: false
tracesToMetrics:
datasourceUid: prometheus
tags: [ { key: 'service.name' }, { key: 'job' } ]
queries:
- name: 'Service-Level Latency'
query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
- name: 'Service-Level Calls'
query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
- name: 'Service-Level Errors'
query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
nodeGraph:
enabled: true
- name: Loki
type: loki
uid: loki
url: http://loki:3100
basicAuth: false
isDefault: false
editable: false
jsonData:
derivedFields:
- datasourceUid: tempo
matcherRegex: 'trace_id=(\w+)'
name: 'TraceID'
url: '$${__value.raw}'
- name: Jaeger
type: jaeger
uid: jaeger
url: http://jaeger:16686
access: proxy
isDefault: false
editable: false
jsonData:
tracesToLogs:
datasourceUid: loki
tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
mappedTags: [ { key: 'service.name', value: 'app' } ]
spanStartTimeShift: '1s'
spanEndTimeShift: '-1s'
filterByTraceID: true
filterBySpanID: false
- name: Pyroscope
type: grafana-pyroscope-datasource
url: http://pyroscope:4040
jsonData:
minStep: '15s'
-74
View File
@@ -1,74 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
service:
extensions: [jaeger_storage, jaeger_query]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [jaeger_storage_exporter, spanmetrics]
metrics/spanmetrics:
receivers: [spanmetrics]
exporters: [prometheus]
telemetry:
resource:
service.name: jaeger
metrics:
level: detailed
readers:
- pull:
exporter:
prometheus:
host: 0.0.0.0
port: 8888
logs:
level: DEBUG
extensions:
jaeger_query:
storage:
traces: some_storage
metrics: some_metrics_storage
jaeger_storage:
backends:
some_storage:
memory:
max_traces: 100000
metric_backends:
some_metrics_storage:
prometheus:
endpoint: http://prometheus:9090
normalize_calls: true
normalize_duration: true
connectors:
spanmetrics:
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
processors:
batch:
exporters:
jaeger_storage_exporter:
trace_storage: some_storage
prometheus:
endpoint: "0.0.0.0:8889"
-63
View File
@@ -1,63 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9095
log_level: info
grpc_server_max_concurrent_streams: 1000
common:
instance_addr: 127.0.0.1
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
allow_structured_metadata: true
max_line_size: 256KB
pattern_ingester:
enabled: true
metric_aggregation:
loki_address: localhost:3100
frontend:
encoding: protobuf
@@ -1,116 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 1s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_mib: 1024
spike_limit_mib: 256
transform/logs:
log_statements:
- context: log
statements:
- set(attributes["message"], body.string)
- set(attributes["log.body"], body.string)
exporters:
otlp/tempo:
endpoint: "tempo:4317"
tls:
insecure: true
compression: gzip
retry_on_failure:
enabled: true
initial_interval: 1s
max_interval: 30s
max_elapsed_time: 300s
sending_queue:
enabled: true
num_consumers: 10
queue_size: 5000
otlp/jaeger:
endpoint: "jaeger:4317"
tls:
insecure: true
compression: gzip
retry_on_failure:
enabled: true
initial_interval: 1s
max_interval: 30s
max_elapsed_time: 300s
sending_queue:
enabled: true
num_consumers: 10
queue_size: 5000
prometheus:
endpoint: "0.0.0.0:8889"
send_timestamps: true
metric_expiration: 5m
resource_to_telemetry_conversion:
enabled: true
otlphttp/loki:
endpoint: "http://loki:3100/otlp"
tls:
insecure: true
compression: gzip
extensions:
health_check:
endpoint: 0.0.0.0:13133
pprof:
endpoint: 0.0.0.0:1888
zpages:
endpoint: 0.0.0.0:55679
service:
extensions: [ health_check, pprof, zpages ]
pipelines:
traces:
receivers: [ otlp ]
processors: [ memory_limiter, batch ]
exporters: [ otlp/tempo, otlp/jaeger ]
metrics:
receivers: [ otlp ]
processors: [ batch ]
exporters: [ prometheus ]
logs:
receivers: [ otlp ]
processors: [ batch, transform/logs ]
exporters: [ otlphttp/loki ]
telemetry:
logs:
level: "info"
encoding: "json"
metrics:
level: "normal"
readers:
- pull:
exporter:
prometheus:
host: '0.0.0.0'
port: 8888
@@ -1,53 +0,0 @@
groups:
- name: rustfs-dashboard
interval: 30s
rules:
- record: rustfs:http_server_requests:rate5m
expr: sum by (job) (rate(rustfs_http_server_requests_total[5m]))
- record: rustfs:http_server_request_duration_seconds:p50_5m
expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
- record: rustfs:http_server_request_duration_seconds:p95_5m
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
- record: rustfs:http_server_request_duration_seconds:p99_5m
expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
- record: rustfs:http_server_response_body_size_bytes:p50_5m
expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
- record: rustfs:http_server_response_body_size_bytes:p95_5m
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
- record: rustfs:http_server_response_body_size_bytes:p99_5m
expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
- record: rustfs:log_cleaner_runs:rate15m
expr: sum by (job) (rate(rustfs_log_cleaner_runs_total[15m]))
- record: rustfs:log_cleaner_failure_ratio:rate5m
expr: sum by (job) (rate(rustfs_log_cleaner_run_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_runs_total[5m])), 1e-9)
- record: rustfs:log_cleaner_rotation_failure_ratio:rate5m
expr: sum by (job) (rate(rustfs_log_cleaner_rotation_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_rotation_total[5m])), 1e-9)
- record: rustfs:log_cleaner_rotation_duration_seconds:p95_5m
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_rotation_duration_seconds_bucket[5m])))
- record: rustfs:log_cleaner_compress_duration_seconds:p95_5m
expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])))
- record: rustfs:scanner_objects_scanned:rate5m
expr: sum by (job) (rate(rustfs_scanner_objects_scanned_total[5m]))
- record: rustfs:scanner_directories_scanned:rate5m
expr: sum by (job) (rate(rustfs_scanner_directories_scanned_total[5m]))
- record: rustfs:scanner_buckets_scanned:rate5m
expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
- record: rustfs:scanner_cycles_success:rate5m
expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))
- record: rustfs:log_chain_op_event_mismatch:rate5m
expr: sum by (job) (rate(rustfs_log_chain_op_event_mismatch_total[5m]))
- alert: RustFSLogChainOpEventMismatchDetected
expr: rustfs:log_chain_op_event_mismatch:rate5m > 0
for: 10m
labels:
severity: warning
component: s3-log-chain
annotations:
summary: "RustFS log-chain op/event mismatch detected"
description: "job={{ $labels.job }} has non-zero rustfs_log_chain_op_event_mismatch_total rate for more than 10m. Check s3 op/event mapping changes."
-84
View File
@@ -1,84 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
global:
scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
evaluation_interval: 15s
external_labels:
cluster: 'rustfs-dev' # Label to identify the cluster
replica: '1' # Replica identifier
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
- job_name: 'otel-collector'
static_configs:
- targets: [ 'otel-collector:8888' ] # Scrape metrics from Collector
scrape_interval: 10s
- job_name: 'rustfs-app-metrics'
static_configs:
- targets: [ 'otel-collector:8889' ] # Application indicators
scrape_interval: 15s
metric_relabel_configs:
- source_labels: [ __name__ ]
regex: 'go_.*'
action: drop # Drop Go runtime metrics if not needed
- job_name: 'tempo'
static_configs:
- targets: [ 'tempo:3200' ] # Scrape metrics from Tempo
- job_name: 'jaeger'
static_configs:
- targets: [ 'jaeger:14269' ] # Jaeger admin port (14269 is standard for admin/metrics)
- job_name: 'loki'
static_configs:
- targets: [ 'loki:3100' ]
- job_name: 'prometheus'
static_configs:
- targets: [ 'localhost:9090' ]
- job_name: 'vulture'
static_configs:
- targets:
- 'vulture:8080'
otlp:
promote_resource_attributes:
- service.instance.id
- service.name
- service.namespace
- cloud.availability_zone
- cloud.region
- container.name
- deployment.environment.name
- k8s.cluster.name
- k8s.container.name
- k8s.cronjob.name
- k8s.daemonset.name
- k8s.deployment.name
- k8s.job.name
- k8s.namespace.name
- k8s.pod.name
- k8s.replicaset.name
- k8s.statefulset.name
translation_strategy: NoUTF8EscapingWithSuffixes
storage:
tsdb:
out_of_order_time_window: 30m
@@ -1,3 +0,0 @@
kafka:
brokers:
- redpanda:9092
-286
View File
@@ -1,286 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# High Availability Tempo Configuration for docker-compose-example-for-rustfs.yml
# Features:
# - Distributed architecture with multiple components
# - Kafka-based ingestion for fault tolerance
# - Replication factor of 3 for data resilience
# - Query frontend for load balancing
# - Metrics generation from traces
# - WAL for durability
partition_ring_live_store: true
stream_over_http_enabled: true
server:
http_listen_port: 3200
http_server_read_timeout: 30s
http_server_write_timeout: 30s
grpc_server_max_recv_msg_size: 4194304 # 4MB
grpc_server_max_send_msg_size: 4194304
log_level: info
log_format: json
# Memberlist configuration for distributed mode
memberlist:
node_name: tempo
bind_port: 7946
join_members:
- tempo:7946
retransmit_factor: 4
node_timeout: 15s
retransmit_interval: 300ms
dead_node_reclaim_time: 30s
# Distributor configuration - receives traces and routes to ingesters
distributor:
ingester_write_path_enabled: true
kafka_write_path_enabled: true
rate_limit_bytes: 10MB
rate_limit_enabled: true
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
max_concurrent_streams: 0
max_receive_message_size: 4194304
http:
endpoint: "0.0.0.0:4318"
cors:
allowed_origins:
- "*"
max_age: 86400
jaeger:
protocols:
grpc:
endpoint: "0.0.0.0:14250"
thrift_http:
endpoint: "0.0.0.0:14268"
zipkin:
endpoint: "0.0.0.0:9411"
ring:
kvstore:
store: memberlist
heartbeat_timeout: 5s
replication_factor: 3
heartbeat_interval: 5s
# Ingester configuration - stores traces and querying
ingester:
lifecycler:
address: tempo
ring:
kvstore:
store: memberlist
replication_factor: 3
max_cache_freshness_per_sec: 10s
heartbeat_interval: 5s
heartbeat_timeout: 5s
num_tokens: 128
tokens_file_path: /var/tempo/tokens.json
claim_on_rollout: true
trace_idle_period: 20s
max_block_bytes: 10_000_000
max_block_duration: 10m
chunk_size_bytes: 1_000_000
chunk_encoding: snappy
wal:
checkpoint_duration: 5s
max_wal_blocks: 4
metrics:
enabled: true
level: block
target_info_duration: 15m
# WAL configuration for data durability
wal:
checkpoint_duration: 5s
flush_on_shutdown: true
path: /var/tempo/wal
# Kafka ingestion configuration - for high throughput scenarios
ingest:
enabled: true
kafka:
brokers: [ redpanda:9092 ]
topic: tempo-ingest
encoding: protobuf
consumer_group: tempo-ingest-consumer
session_timeout: 10s
rebalance_timeout: 1m
partition: auto
verbosity: 2
# Query frontend configuration - distributed querying
query_frontend:
compression: gzip
downstream_url: http://localhost:3200
log_queries_longer_than: 5s
cache_uncompressed_bytes: 100MB
max_outstanding_requests_per_tenant: 100
max_query_length: 48h
max_query_lookback: 30d
default_result_cache_ttl: 1m
result_cache:
cache:
enable_fifocache: true
default_validity: 1m
rf1_after: "1999-01-01T00:00:00Z"
mcp_server:
enabled: true
# Querier configuration - queries traces
querier:
frontend_worker:
frontend_address: localhost:3200
grpc_client_config:
max_recv_msg_size: 104857600
max_concurrent_queries: 20
max_metric_bytes_per_trace: 1MB
# Query scheduler configuration - for distributed querying
query_scheduler:
use_scheduler_ring: false
# Metrics generator configuration - generates metrics from traces
metrics_generator:
enabled: true
registry:
enabled: true
external_labels:
source: tempo
cluster: rustfs-docker-ha
environment: production
storage:
path: /var/tempo/generator/wal
remote_write:
- url: http://prometheus:9090/api/v1/write
send_exemplars: true
resource_to_telemetry_conversion:
enabled: true
processor:
batch:
timeout: 10s
send_batch_size: 1024
memory_limiter:
check_interval: 5s
limit_mib: 512
spike_limit_mib: 128
processors:
- span-metrics
- local-blocks
- service-graphs
generate_native_histograms: both
# Backend worker configuration
backend_worker:
backend_scheduler_addr: localhost:3200
compaction:
block_retention: 24h
compacted_block_retention: 1h
ring:
kvstore:
store: memberlist
# Backend scheduler configuration
backend_scheduler:
enabled: true
provider:
compaction:
compaction:
block_retention: 24h
compacted_block_retention: 1h
concurrency: 25
v2_out_path: /var/tempo/blocks/compaction
# Storage configuration - local backend with proper retention
storage:
trace:
backend: local
wal:
path: /var/tempo/wal
checkpoint_duration: 5s
flush_on_shutdown: true
local:
path: /var/tempo/blocks
bloom_filter_false_positive: 0.05
bloom_shift: 4
index:
downsample_bytes: 1000000
page_size_bytes: 0
cache_size_bytes: 0
pool:
max_workers: 400
queue_depth: 10000
# Compactor configuration - manages block compaction
compactor:
compaction:
block_retention: 168h # 7 days
compacted_block_retention: 1h
concurrency: 25
v2_out_path: /var/tempo/blocks/compaction
shard_count: 32
max_block_bytes: 107374182400 # 100GB
max_compaction_objects: 6000000
max_time_per_tenant: 5m
block_size_bytes: 107374182400
ring:
kvstore:
store: memberlist
heartbeat_interval: 5s
heartbeat_timeout: 5s
# Limits configuration - rate limiting and quotas
limits:
max_traces_per_user: 10000
max_bytes_per_trace: 10485760 # 10MB
max_search_bytes_per_trace: 0
forgiving_oversize_traces: true
rate_limit_bytes: 10MB
rate_limit_enabled: true
ingestion_burst_size_bytes: 20MB
ingestion_rate_limit_bytes: 10MB
max_bytes_per_second: 10485760
metrics_generator_max_active_series: 10000
metrics_generator_max_churned_series: 10000
metrics_generator_forta_out_of_order_ttl: 5m
# Override configuration
overrides:
defaults:
metrics_generator:
processors:
- span-metrics
- local-blocks
- service-graphs
generate_native_histograms: both
max_active_series: 10000
max_churned_series: 10000
# Usage reporting configuration
usage_report:
reporting_enabled: false
# Tracing configuration for debugging
tracing:
enabled: true
jaeger:
sampler:
name: probabilistic
param: 0.1
reporter_log_spans: false
-72
View File
@@ -1,72 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
stream_over_http_enabled: true
server:
http_listen_port: 3200
log_level: info
memberlist:
node_name: tempo
bind_port: 7946
join_members:
- tempo:7946
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
ingester:
max_block_duration: 5m
metrics_generator:
registry:
external_labels:
source: tempo
cluster: docker-compose
traces_storage:
path: /var/tempo/generator/traces
storage:
path: /var/tempo/generator/wal
remote_write:
- url: http://prometheus:9090/api/v1/write
send_exemplars: true
query_frontend:
rf1_after: "1999-01-01T00:00:00Z"
mcp_server:
enabled: true
storage:
trace:
backend: local
wal:
path: /var/tempo/wal # where to store the wal locally
local:
path: /var/tempo/blocks # where to store the traces locally
overrides:
defaults:
metrics_generator:
processors: [ "span-metrics", "service-graphs", "local-blocks" ]
generate_native_histograms: both
usage_report:
reporting_enabled: false
-61
View File
@@ -1,61 +0,0 @@
# OpenObserve + OpenTelemetry Collector
[![OpenObserve](https://img.shields.io/badge/OpenObserve-OpenSource-blue.svg)](https://openobserve.org)
[![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-Collector-green.svg)](https://opentelemetry.io/)
English | [中文](README_ZH.md)
This directory contains the configuration for an **alternative** observability stack using OpenObserve.
## ⚠️ Note
For the **recommended** observability stack (Prometheus, Grafana, Tempo, Loki), please see `../observability/`.
## 🌟 Overview
OpenObserve is a lightweight, all-in-one observability platform that handles logs, metrics, and traces in a single binary. This setup is ideal for:
- Resource-constrained environments.
- Quick setup and testing.
- Users who prefer a unified UI.
## 🚀 Quick Start
### 1. Start Services
```bash
cd .docker/openobserve-otel
docker compose up -d
```
### 2. Access Dashboard
- **URL**: [http://localhost:5080](http://localhost:5080)
- **Username**: `root@rustfs.com`
- **Password**: `rustfs123`
## 🛠️ Configuration
### OpenObserve
- **Persistence**: Data is persisted to a Docker volume.
- **Ports**:
- `5080`: HTTP API and UI
- `5081`: OTLP gRPC
### OpenTelemetry Collector
- **Receivers**: OTLP (gRPC `4317`, HTTP `4318`)
- **Exporters**: Sends data to OpenObserve.
## 🔗 Integration
Configure your application to send OTLP data to the collector:
- **Endpoint**: `http://localhost:4318` (HTTP) or `localhost:4317` (gRPC)
Example for RustFS:
```bash
export RUSTFS_OBS_ENDPOINT=http://localhost:4318
export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
```
-61
View File
@@ -1,61 +0,0 @@
# OpenObserve + OpenTelemetry Collector
[![OpenObserve](https://img.shields.io/badge/OpenObserve-OpenSource-blue.svg)](https://openobserve.org)
[![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-Collector-green.svg)](https://opentelemetry.io/)
[English](README.md) | 中文
本目录包含使用 OpenObserve 的**替代**可观测性技术栈配置。
## ⚠️ 注意
对于**推荐**的可观测性技术栈(Prometheus, Grafana, Tempo, Loki),请参阅 `../observability/`
## 🌟 概览
OpenObserve 是一个轻量级、一体化的可观测性平台,在一个二进制文件中处理日志、指标和追踪。此设置非常适合:
- 资源受限的环境。
- 快速设置和测试。
- 喜欢统一 UI 的用户。
## 🚀 快速开始
### 1. 启动服务
```bash
cd .docker/openobserve-otel
docker compose up -d
```
### 2. 访问仪表盘
- **URL**: [http://localhost:5080](http://localhost:5080)
- **用户名**: `root@rustfs.com`
- **密码**: `rustfs123`
## 🛠️ 配置
### OpenObserve
- **持久化**: 数据持久化到 Docker 卷。
- **端口**:
- `5080`: HTTP API 和 UI
- `5081`: OTLP gRPC
### OpenTelemetry Collector
- **接收器**: OTLP (gRPC `4317`, HTTP `4318`)
- **导出器**: 将数据发送到 OpenObserve。
## 🔗 集成
配置您的应用程序将 OTLP 数据发送到收集器:
- **端点**: `http://localhost:4318` (HTTP) 或 `localhost:4317` (gRPC)
RustFS 示例:
```bash
export RUSTFS_OBS_ENDPOINT=http://localhost:4318
export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
```
@@ -1,87 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
services:
openobserve:
image: public.ecr.aws/zinclabs/openobserve:latest
restart: unless-stopped
environment:
ZO_ROOT_USER_EMAIL: "root@rustfs.com"
ZO_ROOT_USER_PASSWORD: "rustfs123"
ZO_TRACING_HEADER_KEY: "Authorization"
ZO_TRACING_HEADER_VALUE: "Basic cm9vdEBydXN0ZnMuY29tOmQ4SXlCSEJTUkk3RGVlcEQ="
ZO_DATA_DIR: "/data"
ZO_MEMORY_CACHE_ENABLED: "true"
ZO_MEMORY_CACHE_MAX_SIZE: "256"
RUST_LOG: "info"
TZ: Asia/Shanghai
ports:
- "5080:5080"
- "5081:5081"
volumes:
- ./data:/data
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:5080/health" ]
start_period: 60s
interval: 10s
timeout: 5s
retries: 6
networks:
- otel-network
deploy:
resources:
limits:
memory: 1024M
reservations:
memory: 512M
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
restart: unless-stopped
environment:
- TZ=Asia/Shanghai
volumes:
- ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
- "13133:13133" # Health check
- "1777:1777" # pprof
- "55679:55679" # zpages
- "1888:1888" # Metrics
- "8888:8888" # Prometheus metrics
- "8889:8889" # Additional metrics endpoint
depends_on:
- openobserve
networks:
- otel-network
deploy:
resources:
limits:
memory: 10240M
reservations:
memory: 512M
networks:
otel-network:
driver: bridge
name: otel-network
ipam:
config:
- subnet: 172.28.0.0/16
gateway: 172.28.0.1
labels:
com.example.description: "Network for OpenObserve and OpenTelemetry Collector"
volumes:
data:
@@ -1,92 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
filelog:
include: [ "/var/log/app/*.log" ]
start_at: end
processors:
batch:
timeout: 1s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_mib: 400
spike_limit_mib: 100
exporters:
otlphttp/openobserve:
endpoint: http://openobserve:5080/api/default # http://127.0.0.1:5080/api/default
headers:
Authorization: "Basic cm9vdEBydXN0ZnMuY29tOmQ4SXlCSEJTUkk3RGVlcEQ="
stream-name: default
organization: default
compression: gzip
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
timeout: 10s
otlp/openobserve:
endpoint: openobserve:5081 # http://127.0.0.1:5080/api/default
headers:
Authorization: "Basic cm9vdEBydXN0ZnMuY29tOmQ4SXlCSEJTUkk3RGVlcEQ="
stream-name: default
organization: default
compression: gzip
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
timeout: 10s
tls:
insecure: true
extensions:
health_check:
endpoint: 0.0.0.0:13133
pprof:
endpoint: 0.0.0.0:1777
zpages:
endpoint: 0.0.0.0:55679
service:
extensions: [ health_check, pprof, zpages ]
pipelines:
traces:
receivers: [ otlp ]
processors: [ memory_limiter, batch ]
exporters: [ otlp/openobserve ]
metrics:
receivers: [ otlp ]
processors: [ memory_limiter, batch ]
exporters: [ otlp/openobserve ]
logs:
receivers: [ otlp, filelog ]
processors: [ memory_limiter, batch ]
exporters: [ otlp/openobserve ]
telemetry:
logs:
level: "info" # Collector 日志级别
metrics:
address: "0.0.0.0:8888" # Collector 自身指标暴露
@@ -1,52 +0,0 @@
services:
rustfs:
image: rustfs/rustfs:1.0.0-alpha.99-glibc
container_name: rustfs-issue-2715-test
security_opt:
- "no-new-privileges:true"
ports:
- "19000:9000"
- "19001:9001"
environment:
- RUSTFS_VOLUMES=/data/rustfs{0...8}
- RUSTFS_ADDRESS=0.0.0.0:9000
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
- RUSTFS_CONSOLE_ENABLE=true
- RUSTFS_CORS_ALLOWED_ORIGINS=*
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
- RUSTFS_ACCESS_KEY=admin
- RUSTFS_SECRET_KEY=admin
- RUSTFS_OBS_LOGGER_LEVEL=info
- RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
- RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
- RUSTFS_STORAGE_CLASS_STANDARD=EC:2
- RUSTFS_STORAGE_CLASS_RRS=EC:1
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
- RUSTFS_OBS_LOG_DIRECTORY=/opt/rustfs/logs
extra_hosts:
- "otel-collector:host-gateway"
- "pyroscope:host-gateway"
volumes:
- ./deploy/data/issue-2715/rustfs0:/data/rustfs0
- ./deploy/data/issue-2715/rustfs1:/data/rustfs1
- ./deploy/data/issue-2715/rustfs2:/data/rustfs2
- ./deploy/data/issue-2715/rustfs3:/data/rustfs3
- ./deploy/data/issue-2715/rustfs4:/data/rustfs4
- ./deploy/data/issue-2715/rustfs5:/data/rustfs5
- ./deploy/data/issue-2715/rustfs6:/data/rustfs6
- ./deploy/data/issue-2715/rustfs7:/data/rustfs7
- ./deploy/data/issue-2715/rustfs8:/data/rustfs8
- ./deploy/logs/issue-2715:/opt/rustfs/logs
restart: unless-stopped
healthcheck:
test:
[
"CMD",
"sh",
"-c",
"curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
-1
View File
@@ -1 +0,0 @@
data/
-106
View File
@@ -1,106 +0,0 @@
# Issue 2815 Local Docker Verification
## Purpose
This directory contains the local distributed Docker verification assets used to validate issue `#2815` against the current source build.
The target behavior is:
- 4-node distributed cluster starts successfully
- `/health/ready` becomes reachable on each node
- logs no longer contain `storage_info failed: Io error: wrong msgpack marker FixArray(1)`
- internode RPC authentication succeeds with an explicit non-default RPC secret
## Files
- `docker-compose.yml`: 4-node distributed cluster using a locally built image
## Data Directories
Create the bind-mount directories before `docker compose up`:
```bash
mkdir -p .docker/test/issues-2815/data/rustfs{1..4}-disk{0..3}
```
## Build
Apple Silicon / arm64 host:
```bash
docker build --platform linux/arm64 -f Dockerfile.source -t rustfs-issue-2815-local .
```
If you intentionally want amd64 emulation:
```bash
docker build --platform linux/amd64 -f Dockerfile.source -t rustfs-issue-2815-local .
```
## Run
```bash
docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
```
If the image platform is not `linux/arm64`, align compose explicitly:
```bash
RUSTFS_DOCKER_PLATFORM=linux/amd64 docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
```
## Health Checks
Container-level healthcheck is now included and probes:
```bash
curl -fsS http://127.0.0.1:9000/health
```
Manual checks:
```bash
curl -i http://127.0.0.1:9101/health/ready
curl -i http://127.0.0.1:9102/health/ready
curl -i http://127.0.0.1:9103/health/ready
curl -i http://127.0.0.1:9104/health/ready
```
## RPC Secret Requirement
The current source build no longer reproduces the original `FixArray(1)` decode error from issue `#2815`.
Earlier local Docker attempts failed during erasure bootstrap with:
```text
No valid auth token
store init failed to load formats after 10 retries: erasure read quorum
```
Root cause:
- RPC authentication rejects the default secret `rustfsadmin`
- distributed local Docker validation therefore needs an explicit non-default secret
This compose now sets both:
- `RUSTFS_SECRET_KEY=issue-2815-secret`
- `RUSTFS_RPC_SECRET=issue-2815-rpc-secret`
With those values in place, the current 4-node local Docker cluster reaches healthy state and `/health/ready` returns `200`.
In other words:
- `RUSTFS_ACCESS_KEY` may still be `rustfsadmin` for local service credentials if desired
- `RUSTFS_SECRET_KEY` can still be used for service credentials
- but RPC authentication must not resolve to the default secret value `rustfsadmin`
- if `RUSTFS_RPC_SECRET` is unset, the code falls back to `RUSTFS_SECRET_KEY`
- so at least one of them must provide a non-default shared secret for internode RPC signing
## Suggested Debug Commands
```bash
docker compose -f .docker/test/issues-2815/docker-compose.yml ps
docker compose -f .docker/test/issues-2815/docker-compose.yml logs --no-color --tail=200
docker compose -f .docker/test/issues-2815/docker-compose.yml down -v
```
-120
View File
@@ -1,120 +0,0 @@
services:
rustfs1:
image: rustfs-issue-2815-local
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
hostname: rustfs1
container_name: rustfs-issue-2815-rustfs1
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: "rustfsadmin"
RUSTFS_SECRET_KEY: "issue-2815-secret"
RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
RUSTFS_CONSOLE_ENABLE: "false"
RUST_LOG: "info"
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
volumes:
- ./data/rustfs1-disk0:/data/rustfs0
- ./data/rustfs1-disk1:/data/rustfs1
- ./data/rustfs1-disk2:/data/rustfs2
- ./data/rustfs1-disk3:/data/rustfs3
networks: [rustfs-issue-2815-net]
ports:
- "9101:9000"
healthcheck:
test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
interval: 15s
timeout: 5s
retries: 8
start_period: 30s
rustfs2:
image: rustfs-issue-2815-local
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
hostname: rustfs2
container_name: rustfs-issue-2815-rustfs2
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: "rustfsadmin"
RUSTFS_SECRET_KEY: "issue-2815-secret"
RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
RUSTFS_CONSOLE_ENABLE: "false"
RUST_LOG: "info"
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
volumes:
- ./data/rustfs2-disk0:/data/rustfs0
- ./data/rustfs2-disk1:/data/rustfs1
- ./data/rustfs2-disk2:/data/rustfs2
- ./data/rustfs2-disk3:/data/rustfs3
networks: [rustfs-issue-2815-net]
ports:
- "9102:9000"
healthcheck:
test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
interval: 15s
timeout: 5s
retries: 8
start_period: 30s
rustfs3:
image: rustfs-issue-2815-local
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
hostname: rustfs3
container_name: rustfs-issue-2815-rustfs3
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: "rustfsadmin"
RUSTFS_SECRET_KEY: "issue-2815-secret"
RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
RUSTFS_CONSOLE_ENABLE: "false"
RUST_LOG: "info"
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
volumes:
- ./data/rustfs3-disk0:/data/rustfs0
- ./data/rustfs3-disk1:/data/rustfs1
- ./data/rustfs3-disk2:/data/rustfs2
- ./data/rustfs3-disk3:/data/rustfs3
networks: [rustfs-issue-2815-net]
ports:
- "9103:9000"
healthcheck:
test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
interval: 15s
timeout: 5s
retries: 8
start_period: 30s
rustfs4:
image: rustfs-issue-2815-local
platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
hostname: rustfs4
container_name: rustfs-issue-2815-rustfs4
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: "rustfsadmin"
RUSTFS_SECRET_KEY: "issue-2815-secret"
RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
RUSTFS_CONSOLE_ENABLE: "false"
RUST_LOG: "info"
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
volumes:
- ./data/rustfs4-disk0:/data/rustfs0
- ./data/rustfs4-disk1:/data/rustfs1
- ./data/rustfs4-disk2:/data/rustfs2
- ./data/rustfs4-disk3:/data/rustfs3
networks: [rustfs-issue-2815-net]
ports:
- "9104:9000"
healthcheck:
test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
interval: 15s
timeout: 5s
retries: 8
start_period: 30s
networks:
rustfs-issue-2815-net:
name: rustfs-issue-2815-net
-1
View File
@@ -1 +0,0 @@
target
-1
View File
@@ -1 +0,0 @@
use flake
-36
View File
@@ -1,36 +0,0 @@
# GitHub Workflow Instructions
Applies to `.github/` and repository pull-request operations.
## Pull Requests
- PR titles and descriptions must be in English.
- Use `.github/pull_request_template.md` for every PR body.
- Keep all template section headings.
- Use `N/A` for non-applicable sections.
- Include verification commands in the PR details.
- For `gh pr create` and `gh pr edit`, always write markdown body to a file and pass `--body-file`.
- Do not use multiline inline `--body`; backticks and shell expansion can corrupt content or trigger unintended commands.
- Recommended pattern:
- `cat > /tmp/pr_body.md <<'EOF'`
- `...markdown...`
- `EOF`
- `gh pr create ... --body-file /tmp/pr_body.md`
## CI Alignment
When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`.
Current `test-and-lint` gate includes:
- `cargo nextest run --all --exclude e2e_test`
- `cargo test --all --doc`
- `cargo test -p rustfs get_object_chunk_fast_path`
- `cargo test -p rustfs materialize_chunk_stream_before_commit`
- `touch rustfs/build.rs`
- `cargo build -p rustfs --bins --jobs 2`
- `cargo test -p e2e_test archive_multipart_roundtrip_preserves_bytes`
- `cargo test -p e2e_test presigned_get_and_reverse_proxy_preserve_multipart_bytes_with_fast_path`
- `cargo fmt --all --check`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `./scripts/check_layer_dependencies.sh`
-38
View File
@@ -1,38 +0,0 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
**Additional context**
Add any other context about the problem here.
-20
View File
@@ -1,20 +0,0 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
-94
View File
@@ -1,94 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: "Setup Rust Environment"
description: "Setup Rust development environment with caching for RustFS"
inputs:
rust-version:
description: "Rust version to install"
required: false
default: "stable"
cache-shared-key:
description: "Shared cache key for Rust dependencies"
required: false
default: "rustfs-deps"
cache-save-if:
description: "Condition for saving cache"
required: false
default: "true"
install-cross-tools:
description: "Install cross-compilation tools"
required: false
default: "false"
target:
description: "Target architecture to add"
required: false
default: ""
github-token:
description: "GitHub token for API access"
required: false
default: ""
runs:
using: "composite"
steps:
- name: Install system dependencies (Ubuntu)
if: runner.os == 'Linux'
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y \
musl-tools \
build-essential \
pkg-config \
libssl-dev \
protobuf-compiler
- name: Install protoc
uses: rustfs/setup-protoc@v3.0.1
with:
version: "34.1"
repo-token: ${{ github.token }}
- name: Install flatc
uses: Nugine/setup-flatc@v1
with:
version: "25.12.19"
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: ${{ inputs.rust-version }}
targets: ${{ inputs.target }}
components: rustfmt, clippy
- name: Install Zig
if: inputs.install-cross-tools == 'true'
uses: mlugg/setup-zig@v2
- name: Install cargo-zigbuild
if: inputs.install-cross-tools == 'true'
uses: taiki-e/install-action@cargo-zigbuild
- name: Install cargo-nextest
uses: taiki-e/install-action@cargo-nextest
- name: Setup Rust cache
uses: Swatinem/rust-cache@v2
with:
cache-all-crates: true
cache-on-failure: true
shared-key: ${{ inputs.cache-shared-key }}
save-if: ${{ inputs.cache-save-if }}
-17
View File
@@ -1,17 +0,0 @@
enabled: true
document:
version: v1
url: https://github.com/rustfs/cla/blob/main/cla/v1.md
signing:
mode: comment
comment_pattern: I have read and agree to the CLA.
registry:
type: json-repo
repository: rustfs/cla
path_prefix: signatures
status:
check_name: CLA Check
-1
View File
@@ -1 +0,0 @@
../AGENTS.md
-53
View File
@@ -1,53 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
version: 2
updates:
- package-ecosystem: "cargo" # See documentation for possible values
directory: "/" # Location of package manifests
schedule:
interval: "weekly"
day: "monday"
timezone: "Asia/Shanghai"
time: "08:00"
assignees:
- "houseme"
reviewers:
- "overtrue"
- "majinghe"
ignore:
- dependency-name: "object_store"
versions: [ "0.13.x" ]
- dependency-name: "libunftp"
versions: [ "0.23.x" ]
- dependency-name: "ratelimit"
versions: [ "1.x" ]
- dependency-name: "ratelimit"
versions: [ "2.x" ]
groups:
s3s:
update-types:
- "minor"
- "patch"
patterns:
- "s3s"
- "s3s-*"
dependencies:
patterns:
- "*"
-36
View File
@@ -1,36 +0,0 @@
<!--
Pull Request Template for RustFS
-->
## Related Issues
<!--
List related issues, e.g. Fixes #123.
Use N/A when there is no related issue.
-->
## Summary of Changes
<!--
Briefly explain what changed and why reviewers should accept it.
Focus on behavior, compatibility, and review-relevant context.
-->
## Verification
<!--
List the commands or checks you ran, for example:
- `make pre-commit`
Use N/A only when verification is not applicable.
-->
## Impact
<!--
Describe user-facing, compatibility, API, deployment, configuration, or
documentation impact. Use N/A when there is no expected impact.
-->
## Additional Notes
<!-- Any extra information for reviewers, or N/A. -->
---
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
-103
View File
@@ -1,103 +0,0 @@
# S3 Compatibility Tests Configuration
This directory contains the configuration for running [Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
## Configuration File
The `s3tests.conf` file is based on the official `s3tests.conf.SAMPLE` from the ceph/s3-tests repository. It uses environment variable substitution via `envsubst` to configure the endpoint and credentials.
### Key Configuration Points
- **Host**: Set via `${S3_HOST}` environment variable (e.g., `rustfs-single` for single-node, `lb` for multi-node)
- **Port**: 9000 (standard RustFS port)
- **Credentials**: Uses `${S3_ACCESS_KEY}` and `${S3_SECRET_KEY}` from workflow environment
- **TLS**: Disabled (`is_secure = False`)
## Test Execution Strategy
### Network Connectivity Fix
Tests run inside a Docker container on the `rustfs-net` network, which allows them to resolve and connect to the RustFS container hostnames. This fixes the "Temporary failure in name resolution" error that occurred when tests ran on the GitHub runner host.
### Performance Optimizations
1. **Parallel Execution**: Uses `pytest-xdist` with `-n 4` to run tests in parallel across 4 workers
2. **Load Distribution**: Uses `--dist=loadgroup` to distribute test groups across workers
3. **Fail-Fast**: Uses `--maxfail=50` to stop after 50 failures, saving time on catastrophic failures
### Feature Filtering
Tests are filtered using pytest markers (`-m`) to skip features not yet supported by RustFS:
- `lifecycle` - Bucket lifecycle policies
- `versioning` - Object versioning
- `s3website` - Static website hosting
- `bucket_logging` - Bucket logging
- `encryption` / `sse_s3` - Server-side encryption
- `cloud_transition` / `cloud_restore` - Cloud storage transitions
- `lifecycle_expiration` / `lifecycle_transition` - Lifecycle operations
This filtering:
1. Reduces test execution time significantly (from 1+ hour to ~10-15 minutes)
2. Focuses on features RustFS currently supports
3. Avoids hundreds of expected failures
## Running Tests Locally
### Single-Node Test
```bash
# Set credentials
export S3_ACCESS_KEY=rustfsadmin
export S3_SECRET_KEY=rustfsadmin
# Start RustFS container
docker run -d --name rustfs-single \
--network rustfs-net \
-e RUSTFS_ADDRESS=0.0.0.0:9000 \
-e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
-e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
-e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
rustfs-ci
# Generate config
export S3_HOST=rustfs-single
envsubst < .github/s3tests/s3tests.conf > /tmp/s3tests.conf
# Run tests
docker run --rm \
--network rustfs-net \
-v /tmp/s3tests.conf:/etc/s3tests.conf:ro \
python:3.12-slim \
bash -c '
apt-get update -qq && apt-get install -y -qq git
git clone --depth 1 https://github.com/ceph/s3-tests.git /s3-tests
cd /s3-tests
pip install -q -r requirements.txt pytest-xdist
S3TEST_CONF=/etc/s3tests.conf pytest -v -n 4 \
s3tests/functional/test_s3.py \
-m "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption and not sse_s3"
'
```
## Test Results Interpretation
- **PASSED**: Test succeeded, feature works correctly
- **FAILED**: Test failed, indicates a potential bug or incompatibility
- **ERROR**: Test setup failed (e.g., network issues, missing dependencies)
- **SKIPPED**: Test skipped due to marker filtering
## Adding New Feature Support
When adding support for a new S3 feature to RustFS:
1. Remove the corresponding marker from the filter in `.github/workflows/e2e-s3tests.yml`
2. Run the tests to verify compatibility
3. Fix any failing tests
4. Update this README to reflect the newly supported feature
## References
- [Ceph S3 Tests Repository](https://github.com/ceph/s3-tests)
- [S3 API Compatibility](https://docs.aws.amazon.com/AmazonS3/latest/API/)
- [pytest-xdist Documentation](https://pytest-xdist.readthedocs.io/)
-193
View File
@@ -1,193 +0,0 @@
# RustFS s3-tests configuration
# Based on: https://github.com/ceph/s3-tests/blob/master/s3tests.conf.SAMPLE
#
# Usage:
# Single-node: S3_HOST=rustfs-single envsubst < s3tests.conf > /tmp/s3tests.conf
# Multi-node: S3_HOST=lb envsubst < s3tests.conf > /tmp/s3tests.conf
[DEFAULT]
## this section is just used for host, port and bucket_prefix
# host set for RustFS - will be substituted via envsubst
host = ${S3_HOST}
# port for RustFS
port = 9000
## say "False" to disable TLS
is_secure = False
## say "False" to disable SSL Verify
ssl_verify = False
[fixtures]
## all the buckets created will start with this prefix;
## {random} will be filled with random characters to pad
## the prefix to 30 characters long, and avoid collisions
bucket prefix = rustfs-{random}-
# all the iam account resources (users, roles, etc) created
# will start with this name prefix
iam name prefix = s3-tests-
# all the iam account resources (users, roles, etc) created
# will start with this path prefix
iam path prefix = /s3-tests/
[s3 main]
# main display_name
display_name = RustFS Tester
# main user_id
user_id = rustfsadmin
# main email
email = tester@rustfs.local
# zonegroup api_name for bucket location
api_name = default
## main AWS access key
access_key = ${S3_ACCESS_KEY}
## main AWS secret key
secret_key = ${S3_SECRET_KEY}
## replace with key id obtained when secret is created, or delete if KMS not tested
#kms_keyid = 01234567-89ab-cdef-0123-456789abcdef
## Storage classes
#storage_classes = "LUKEWARM, FROZEN"
## Lifecycle debug interval (default: 10)
#lc_debug_interval = 20
## Restore debug interval (default: 100)
#rgw_restore_debug_interval = 60
#rgw_restore_processor_period = 60
[s3 alt]
# alt display_name
display_name = RustFS Alt Tester
## alt email
email = alt@rustfs.local
# alt user_id
user_id = rustfsalt
# alt AWS access key (must be different from s3 main for many tests)
access_key = ${S3_ALT_ACCESS_KEY}
# alt AWS secret key
secret_key = ${S3_ALT_SECRET_KEY}
#[s3 cloud]
## to run the testcases with "cloud_transition" for transition
## and "cloud_restore" for restore attribute.
## Note: the waiting time may have to tweaked depending on
## the I/O latency to the cloud endpoint.
## host set for cloud endpoint
# host = localhost
## port set for cloud endpoint
# port = 8001
## say "False" to disable TLS
# is_secure = False
## cloud endpoint credentials
# access_key = 0555b35654ad1656d804
# secret_key = h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
## storage class configured as cloud tier on local rgw server
# cloud_storage_class = CLOUDTIER
## Below are optional -
## Above configured cloud storage class config options
# retain_head_object = false
# allow_read_through = false # change it to enable read_through
# read_through_restore_days = 2
# target_storage_class = Target_SC
# target_path = cloud-bucket
## another regular storage class to test multiple transition rules,
# storage_class = S1
[s3 tenant]
# tenant display_name
display_name = RustFS Tenant Tester
# tenant user_id
# Note: Using same user_id as main to avoid teardown failures.
# RustFS does not currently support multi-tenancy, so the tenant client
# effectively operates as the main user. This ensures nuke_prefixed_buckets()
# in s3-tests teardown can successfully clean up resources.
user_id = rustfsadmin
# tenant AWS access key
access_key = ${S3_ACCESS_KEY}
# tenant AWS secret key
secret_key = ${S3_SECRET_KEY}
# tenant email
email = tenant@rustfs.local
# tenant name
# Note: Empty tenant name to avoid multi-tenant path issues during teardown.
# When s3-tests calls get_tenant_client(), it uses this tenant value in requests.
# An empty value makes the tenant client behave like the main client, preventing
# "bucket not found" errors when teardown tries to clean up test buckets.
tenant =
#following section needs to be added for all sts-tests
[iam]
#used for iam operations in sts-tests
#email
email = s3@rustfs.local
#user_id
user_id = rustfsiam
#access_key
access_key = ${S3_ACCESS_KEY}
#secret_key
secret_key = ${S3_SECRET_KEY}
#display_name
display_name = RustFS IAM User
# iam account root user for iam_account tests
[iam root]
access_key = ${S3_ACCESS_KEY}
secret_key = ${S3_SECRET_KEY}
user_id = RGW11111111111111111
email = account1@rustfs.local
# iam account root user in a different account than [iam root]
[iam alt root]
access_key = ${S3_ACCESS_KEY}
secret_key = ${S3_SECRET_KEY}
user_id = RGW22222222222222222
email = account2@rustfs.local
#following section needs to be added when you want to run Assume Role With Webidentity test
[webidentity]
#used for assume role with web identity test in sts-tests
#all parameters will be obtained from ceph/qa/tasks/keycloak.py
#token=<access_token>
#aud=<obtained after introspecting token>
#sub=<obtained after introspecting token>
#azp=<obtained after introspecting token>
#user_token=<access token for a user, with attribute Department=[Engineering, Marketing>]
#thumbprint=<obtained from x509 certificate>
#KC_REALM=<name of the realm>
-81
View File
@@ -1,81 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Security Audit
on:
push:
branches: [ main ]
paths:
- '**/Cargo.toml'
- '**/Cargo.lock'
- '.github/workflows/audit.yml'
pull_request:
branches: [ main ]
paths:
- '**/Cargo.toml'
- '**/Cargo.lock'
- '.github/workflows/audit.yml'
schedule:
- cron: '0 0 * * 0' # Weekly on Sunday at midnight UTC
workflow_dispatch:
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
jobs:
security-audit:
name: Security Audit
runs-on: ubicloud-standard-2
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Install cargo-audit
uses: taiki-e/install-action@v2
with:
tool: cargo-audit
- name: Run security audit
run: |
cargo audit -D warnings --json | tee audit-results.json
- name: Upload audit results
if: always()
uses: actions/upload-artifact@v6
with:
name: security-audit-results-${{ github.run_number }}
path: audit-results.json
retention-days: 30
dependency-review:
name: Dependency Review
runs-on: ubicloud-standard-2
if: github.event_name == 'pull_request'
permissions:
contents: read
pull-requests: write
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Dependency Review
uses: actions/dependency-review-action@v5
with:
fail-on-severity: moderate
comment-summary-in-pr: always
-867
View File
@@ -1,867 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Build and Release Workflow
#
# This workflow builds RustFS binaries and automatically triggers Docker image builds.
#
# Flow:
# 1. Build binaries for multiple platforms
# 2. Upload binaries to OSS storage
# 3. Trigger docker.yml to build and push images using the uploaded binaries
#
# Manual Parameters:
# - build_docker: Build and push Docker images (default: true)
# - platforms: Comma-separated platform IDs or 'all' (default: all)
name: Build and Release
on:
push:
tags: [ "*.*.*" ]
branches: [ main ]
paths-ignore:
- "**.md"
- "**.txt"
- ".github/**"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "LICENSE*"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".gitignore"
- ".dockerignore"
schedule:
- cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
workflow_dispatch:
inputs:
build_docker:
description: "Build and push Docker images after binary build"
required: false
default: true
type: boolean
platforms:
description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
required: false
default: "all"
type: string
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
# Optimize build performance
CARGO_INCREMENTAL: 0
jobs:
# Build strategy check - determine build type based on trigger
build-check:
name: Build Strategy Check
runs-on: ubicloud-standard-2
outputs:
should_build: ${{ steps.check.outputs.should_build }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Determine build strategy
id: check
run: |
should_build=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
# Get short SHA for all builds
short_sha=$(git rev-parse --short HEAD)
# Determine build type based on trigger
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
# Tag push - release or prerelease
should_build=true
tag_name="${GITHUB_REF#refs/tags/}"
version="${tag_name}"
# Check if this is a prerelease
if [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
echo "🚀 Prerelease build detected: $tag_name"
else
build_type="release"
echo "📦 Release build detected: $tag_name"
fi
elif [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
# Main branch push - development build
should_build=true
build_type="development"
version="dev-${short_sha}"
echo "🛠️ Development build detected"
elif [[ "${{ github.event_name }}" == "schedule" ]] || \
[[ "${{ github.event_name }}" == "workflow_dispatch" ]] || \
[[ "${{ contains(github.event.head_commit.message, '--build') }}" == "true" ]]; then
# Scheduled or manual build
should_build=true
build_type="development"
version="dev-${short_sha}"
echo "⚡ Manual/scheduled build detected"
fi
echo "should_build=$should_build" >> $GITHUB_OUTPUT
echo "build_type=$build_type" >> $GITHUB_OUTPUT
echo "version=$version" >> $GITHUB_OUTPUT
echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
echo "📊 Build Summary:"
echo " - Should build: $should_build"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
# Build RustFS binaries
prepare-platform-matrix:
name: Prepare Platform Matrix
runs-on: ubicloud-standard-2
outputs:
matrix: ${{ steps.select.outputs.matrix }}
selected: ${{ steps.select.outputs.selected }}
steps:
- name: Select target platforms
id: select
shell: bash
run: |
set -euo pipefail
selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
fi
all='{"include":[
{"target_id":"linux-x86_64-musl","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
{"target_id":"linux-aarch64-musl","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
{"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
{"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
{"target_id":"macos-x86_64","os":"macos-15-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
]}'
if [[ "${selected}" == "all" ]]; then
matrix="$(jq -c . <<<"${all}")"
else
unknown="$(jq -rn --arg selected "${selected}" --argjson all "${all}" '
($selected | split(",") | map(select(length > 0))) as $req
| ($all.include | map(.target_id)) as $known
| [$req[] | select(( $known | index(.) ) == null)]
')"
if [[ "$(jq 'length' <<<"${unknown}")" -gt 0 ]]; then
echo "Unknown platforms: $(jq -r 'join(\",\")' <<<"${unknown}")" >&2
echo "Allowed: $(jq -r '.include[].target_id' <<<"${all}" | paste -sd ',' -)" >&2
exit 1
fi
matrix="$(jq -c --arg selected "${selected}" '
($selected | split(",") | map(select(length > 0))) as $req
| .include |= map(select(.target_id as $id | ($req | index($id))))
' <<<"${all}")"
fi
echo "selected=${selected}" >> "$GITHUB_OUTPUT"
echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
echo "Selected platforms: ${selected}"
build-rustfs:
name: Build RustFS
needs: [ build-check, prepare-platform-matrix ]
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
runs-on: ${{ matrix.os }}
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
# Always enable Tokio unstable features (required by dial9-tokio-telemetry).
# The RUSTFLAGS env var takes precedence over .cargo/config.toml [build] rustflags,
# so we must include --cfg tokio_unstable here explicitly; otherwise an empty
# RUSTFLAGS value would shadow the config-file flag and silently break tracing.
RUSTFLAGS: "--cfg tokio_unstable ${{ matrix.rustflags }}"
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}-${{ hashFiles('**/Cargo.lock') }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
install-cross-tools: ${{ matrix.cross }}
- name: Download static console assets
shell: bash
run: |
mkdir -p ./rustfs/static
if [[ "${{ matrix.platform }}" == "windows" ]]; then
if curl.exe --fail -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300; then
unzip -o console.zip -d ./rustfs/static
rm console.zip
else
echo "Warning: Failed to download console assets, continuing without them"
echo "// Static assets not available" > ./rustfs/static/empty.txt
fi
else
chmod +w ./rustfs/static/LICENSE || true
if curl --fail -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
-o console.zip --retry 3 --retry-delay 5 --max-time 300; then
unzip -o console.zip -d ./rustfs/static
rm console.zip
else
echo "Warning: Failed to download console assets, continuing without them"
echo "// Static assets not available" > ./rustfs/static/empty.txt
fi
fi
- name: Build RustFS
shell: bash
run: |
# Force rebuild by touching build.rs
touch rustfs/build.rs
if [[ "${{ matrix.cross }}" == "true" ]]; then
if [[ "${{ matrix.platform }}" == "windows" ]]; then
# Use cross for Windows ARM64
cargo install cross --git https://github.com/cross-rs/cross
cross build --release --target ${{ matrix.target }} -p rustfs --bins
else
# Use zigbuild for other cross-compilation
cargo zigbuild --release --target ${{ matrix.target }} -p rustfs --bins
fi
else
cargo build --release --target ${{ matrix.target }} -p rustfs --bins
fi
- name: Create release package
id: package
shell: bash
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
# Extract platform and arch from target
TARGET="${{ matrix.target }}"
PLATFORM="${{ matrix.platform }}"
# Map target to architecture and variant
case "$TARGET" in
*x86_64*musl*)
ARCH="x86_64"
VARIANT="musl"
;;
*x86_64*gnu*)
ARCH="x86_64"
VARIANT="gnu"
;;
*x86_64*)
ARCH="x86_64"
VARIANT=""
;;
*aarch64*musl*|*arm64*musl*)
ARCH="aarch64"
VARIANT="musl"
;;
*aarch64*gnu*|*arm64*gnu*)
ARCH="aarch64"
VARIANT="gnu"
;;
*aarch64*|*arm64*)
ARCH="aarch64"
VARIANT=""
;;
*armv7*)
ARCH="armv7"
VARIANT=""
;;
*)
ARCH="unknown"
VARIANT=""
;;
esac
# Normalize version used for package filenames
PACKAGE_VERSION="${VERSION}"
if [[ "$PACKAGE_VERSION" == v* ]]; then
PACKAGE_VERSION="${PACKAGE_VERSION#v}"
fi
# Generate package name based on build type
if [[ -n "$VARIANT" ]]; then
ARCH_WITH_VARIANT="${ARCH}-${VARIANT}"
else
ARCH_WITH_VARIANT="${ARCH}"
fi
PACKAGE_BASENAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}"
if [[ "$BUILD_TYPE" == "development" ]]; then
# Development build: rustfs-${platform}-${arch}-${variant}-dev-${short_sha}.zip
PACKAGE_NAME="rustfs-${PLATFORM}-${ARCH_WITH_VARIANT}-dev-${SHORT_SHA}"
else
# Release/Prerelease build: rustfs-${platform}-${arch}-${variant}-v${version}.zip
PACKAGE_NAME="${PACKAGE_BASENAME}-v${PACKAGE_VERSION}"
fi
# Create zip packages for all platforms
# Ensure zip is available
if ! command -v zip &> /dev/null; then
if [[ "${{ matrix.os }}" == "ubuntu-latest" ]]; then
sudo apt-get update && sudo apt-get install -y zip
fi
fi
cd target/${{ matrix.target }}/release
# Determine the binary name based on platform
if [[ "${{ matrix.platform }}" == "windows" ]]; then
BINARY_NAME="rustfs.exe"
else
BINARY_NAME="rustfs"
fi
# Verify the binary exists before packaging
if [[ ! -f "$BINARY_NAME" ]]; then
echo "❌ Binary $BINARY_NAME not found in $(pwd)"
if [[ "${{ matrix.platform }}" == "windows" ]]; then
dir
else
ls -la
fi
exit 1
fi
# Universal packaging function
package_zip() {
local src=$1
local dst=$2
if [[ "${{ matrix.platform }}" == "windows" ]]; then
# Windows uses PowerShell Compress-Archive
powershell -Command "Compress-Archive -Path '$src' -DestinationPath '$dst' -Force"
elif command -v zip &> /dev/null; then
# Unix systems use zip command
zip "$dst" "$src"
else
echo "❌ No zip utility available"
exit 1
fi
}
# Create the zip package
echo "Start packaging: $BINARY_NAME -> ../../../${PACKAGE_NAME}.zip"
package_zip "$BINARY_NAME" "../../../${PACKAGE_NAME}.zip"
cd ../../..
# Verify the package was created
if [[ -f "${PACKAGE_NAME}.zip" ]]; then
echo "✅ Package created successfully: ${PACKAGE_NAME}.zip"
if [[ "${{ matrix.platform }}" == "windows" ]]; then
dir
else
ls -lh ${PACKAGE_NAME}.zip
fi
else
echo "❌ Failed to create package: ${PACKAGE_NAME}.zip"
exit 1
fi
# Create latest version files right after the main package
LATEST_FILES=""
if [[ "$BUILD_TYPE" == "release" ]] || [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Create latest version filename
# Convert from rustfs-linux-x86_64-musl-v1.0.0 to rustfs-linux-x86_64-musl-latest
LATEST_FILE="${PACKAGE_BASENAME}-latest.zip"
echo "🔄 Creating latest version: ${PACKAGE_NAME}.zip -> $LATEST_FILE"
cp "${PACKAGE_NAME}.zip" "$LATEST_FILE"
if [[ -f "$LATEST_FILE" ]]; then
echo "✅ Latest version created: $LATEST_FILE"
LATEST_FILES="$LATEST_FILE"
fi
elif [[ "$BUILD_TYPE" == "development" ]]; then
# Development builds (only main branch triggers development builds)
# Create main-latest version filename
# Convert from rustfs-linux-x86_64-dev-abc123 to rustfs-linux-x86_64-main-latest
MAIN_LATEST_FILE="${PACKAGE_BASENAME}-main-latest.zip"
echo "🔄 Creating main-latest version: ${PACKAGE_NAME}.zip -> $MAIN_LATEST_FILE"
cp "${PACKAGE_NAME}.zip" "$MAIN_LATEST_FILE"
if [[ -f "$MAIN_LATEST_FILE" ]]; then
echo "✅ Main-latest version created: $MAIN_LATEST_FILE"
LATEST_FILES="$MAIN_LATEST_FILE"
# Also create a generic main-latest for Docker builds (Linux only)
if [[ "${{ matrix.platform }}" == "linux" ]]; then
DOCKER_MAIN_LATEST_FILE="rustfs-linux-${ARCH_WITH_VARIANT}-main-latest.zip"
echo "🔄 Creating Docker main-latest version: ${PACKAGE_NAME}.zip -> $DOCKER_MAIN_LATEST_FILE"
cp "${PACKAGE_NAME}.zip" "$DOCKER_MAIN_LATEST_FILE"
if [[ -f "$DOCKER_MAIN_LATEST_FILE" ]]; then
echo "✅ Docker main-latest version created: $DOCKER_MAIN_LATEST_FILE"
LATEST_FILES="$LATEST_FILES $DOCKER_MAIN_LATEST_FILE"
fi
fi
fi
fi
echo "package_name=${PACKAGE_NAME}" >> $GITHUB_OUTPUT
echo "package_file=${PACKAGE_NAME}.zip" >> $GITHUB_OUTPUT
echo "latest_files=${LATEST_FILES}" >> $GITHUB_OUTPUT
echo "build_type=${BUILD_TYPE}" >> $GITHUB_OUTPUT
echo "version=${VERSION}" >> $GITHUB_OUTPUT
echo "📦 Package created: ${PACKAGE_NAME}.zip"
if [[ -n "$LATEST_FILES" ]]; then
echo "📦 Latest files created: $LATEST_FILES"
fi
echo "🔧 Build type: ${BUILD_TYPE}"
echo "📊 Version: ${VERSION}"
- name: Upload to GitHub artifacts
uses: actions/upload-artifact@v6
with:
name: ${{ steps.package.outputs.package_name }}
path: "rustfs-*.zip"
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
- name: Upload to Aliyun OSS
if: env.OSS_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
env:
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-accelerate.aliyuncs.com
shell: bash
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
# Install ossutil (platform-specific)
OSSUTIL_VERSION="2.1.1"
case "${{ matrix.platform }}" in
linux)
if [[ "$(uname -m)" == "arm64" ]]; then
ARCH="arm64"
else
ARCH="amd64"
fi
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-${ARCH}.zip"
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-${ARCH}"
curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
unzip "$OSSUTIL_ZIP"
mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x /usr/local/bin/ossutil
OSSUTIL_BIN=ossutil
;;
macos)
if [[ "$(uname -m)" == "arm64" ]]; then
ARCH="arm64"
else
ARCH="amd64"
fi
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-mac-${ARCH}.zip"
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-mac-${ARCH}"
curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
unzip "$OSSUTIL_ZIP"
mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x /usr/local/bin/ossutil
OSSUTIL_BIN=ossutil
;;
windows)
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-windows-amd64.zip"
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-windows-amd64"
curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
unzip "$OSSUTIL_ZIP"
mv "${OSSUTIL_DIR}/ossutil.exe" ./ossutil.exe
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
OSSUTIL_BIN=./ossutil.exe
;;
esac
# Determine upload path based on build type
if [[ "$BUILD_TYPE" == "development" ]]; then
OSS_PATH="oss://rustfs-artifacts/artifacts/rustfs/dev/"
echo "📤 Uploading development build to OSS dev directory"
else
OSS_PATH="oss://rustfs-artifacts/artifacts/rustfs/release/"
echo "📤 Uploading release build to OSS release directory"
fi
# Upload all rustfs zip files to OSS using glob pattern
echo "📤 Uploading all rustfs-*.zip files to $OSS_PATH..."
for zip_file in rustfs-*.zip; do
if [[ -f "$zip_file" ]]; then
echo "Uploading: $zip_file to $OSS_PATH..."
$OSSUTIL_BIN cp "$zip_file" "$OSS_PATH" --force
echo "✅ Uploaded: $zip_file"
fi
done
echo "✅ Upload completed successfully"
# Build summary
build-summary:
name: Build Summary
needs: [ build-check, build-rustfs ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubicloud-standard-2
steps:
- name: Build completion summary
shell: bash
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
echo "🎉 Build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo ""
# Check build status
BUILD_STATUS="${{ needs.build-rustfs.result }}"
echo "📊 Build Results:"
echo " 📦 All platforms: $BUILD_STATUS"
echo ""
case "$BUILD_TYPE" in
"development")
echo "🛠️ Development build artifacts have been uploaded to OSS dev directory"
echo "⚠️ This is a development build - not suitable for production use"
;;
"release")
echo "🚀 Release build artifacts have been uploaded to OSS release directory"
echo "✅ This build is ready for production use"
echo "🏷️ GitHub Release will be created in this workflow"
;;
"prerelease")
echo "🧪 Prerelease build artifacts have been uploaded to OSS release directory"
echo "⚠️ This is a prerelease build - use with caution"
echo "🏷️ GitHub Release will be created in this workflow"
;;
esac
echo ""
echo "🐳 Docker Images:"
if [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
echo "⏭️ Docker image build was skipped (binary only build)"
elif [[ "$BUILD_STATUS" == "success" ]]; then
echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
else
echo "❌ Docker image build will be skipped due to build failure"
fi
# Create GitHub Release (only for tag pushes)
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubicloud-standard-2
permissions:
contents: write
outputs:
release_id: ${{ steps.create.outputs.release_id }}
release_url: ${{ steps.create.outputs.release_url }}
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Create GitHub Release
id: create
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
# Determine release type for title
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
RELEASE_TYPE="release"
fi
# Check if release already exists
if gh release view "$TAG" >/dev/null 2>&1; then
echo "Release $TAG already exists"
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
else
# Get release notes from tag message
RELEASE_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
RELEASE_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
RELEASE_NOTES="Release ${VERSION}"
fi
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
TITLE="RustFS $VERSION"
fi
# Create the release
PRERELEASE_FLAG=""
if [[ "$IS_PRERELEASE" == "true" ]]; then
PRERELEASE_FLAG="--prerelease"
fi
gh release create "$TAG" \
--title "$TITLE" \
--notes "$RELEASE_NOTES" \
$PRERELEASE_FLAG \
--draft
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
fi
echo "release_id=$RELEASE_ID" >> $GITHUB_OUTPUT
echo "release_url=$RELEASE_URL" >> $GITHUB_OUTPUT
echo "Created release: $RELEASE_URL"
# Prepare and upload release assets
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubicloud-standard-2
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Download all build artifacts
uses: actions/download-artifact@v7
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Prepare release assets
id: prepare
shell: bash
run: |
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
mkdir -p ./release-assets
# Copy and verify artifacts (including latest files created during build)
ASSETS_COUNT=0
for file in ./artifacts/*.zip; do
if [[ -f "$file" ]]; then
cp "$file" ./release-assets/
ASSETS_COUNT=$((ASSETS_COUNT + 1))
fi
done
if [[ $ASSETS_COUNT -eq 0 ]]; then
echo "❌ No artifacts found!"
exit 1
fi
cd ./release-assets
# Generate checksums for all files (including latest versions)
if ls *.zip >/dev/null 2>&1; then
sha256sum *.zip > SHA256SUMS
sha512sum *.zip > SHA512SUMS
fi
# Create signature placeholder files
for file in *.zip; do
echo "# Signature for $file" > "${file}.asc"
echo "# GPG signature will be added in future versions" >> "${file}.asc"
done
echo "📦 Prepared assets:"
ls -la
echo "🔢 Total asset count: $ASSETS_COUNT"
- name: Upload to GitHub Release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
cd ./release-assets
# Upload all files
for file in *; do
if [[ -f "$file" ]]; then
echo "📤 Uploading $file..."
gh release upload "$TAG" "$file" --clobber
fi
done
echo "✅ All assets uploaded successfully"
# Update latest.json for stable releases only
update-latest-version:
name: Update Latest Version
needs: [ build-check, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubicloud-standard-2
steps:
- name: Update latest.json
env:
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
shell: bash
run: |
if [[ -z "$OSS_ACCESS_KEY_ID" ]]; then
echo "⚠️ OSS credentials not available, skipping latest.json update"
exit 0
fi
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
# Install ossutil
OSSUTIL_VERSION="2.1.1"
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-amd64"
curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
unzip "$OSSUTIL_ZIP"
mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x /usr/local/bin/ossutil
# Create latest.json
cat > latest.json << EOF
{
"version": "${VERSION}",
"tag": "${TAG}",
"release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"release_type": "stable",
"download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
}
EOF
# Upload to OSS
ossutil cp latest.json oss://rustfs-version/latest.json --force
echo "✅ Updated latest.json for stable release $VERSION"
# Publish release (remove draft status)
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubicloud-standard-2
permissions:
contents: write
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Update release notes and publish
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
# Determine release type
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
RELEASE_TYPE="release"
fi
# Get original release notes from tag
ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
ORIGINAL_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
ORIGINAL_NOTES="Release ${VERSION}"
fi
fi
# Publish the release (remove draft status)
gh release edit "$TAG" --draft=false
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
-256
View File
@@ -1,256 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Continuous Integration
on:
push:
branches: [ main ]
paths-ignore:
- "**.md"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "scripts/probe.sh"
- "LICENSE*"
- ".gitignore"
- ".dockerignore"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- ".github/workflows/performance.yml"
pull_request:
branches: [ main ]
paths-ignore:
- "**.md"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "scripts/probe.sh"
- "LICENSE*"
- ".gitignore"
- ".dockerignore"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- ".github/workflows/performance.yml"
merge_group:
types: [checks_requested]
schedule:
- cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
CARGO_BUILD_JOBS: 2
jobs:
skip-check:
name: Skip Duplicate Actions
permissions:
actions: write
contents: read
runs-on: ubicloud-standard-2
outputs:
should_skip: ${{ steps.skip_check.outputs.should_skip }}
steps:
- name: Skip duplicate actions
id: skip_check
uses: fkirc/skip-duplicate-actions@v5
with:
concurrent_skipping: "same_content_newer"
cancel_others: true
paths_ignore: '["*.md", "docs/**", "deploy/**"]'
do_not_skip: '["workflow_dispatch", "schedule", "merge_group", "release", "push"]'
typos:
name: Typos
needs: skip-check
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubicloud-standard-2
steps:
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@stable
- name: Typos check with custom config file
uses: crate-ci/typos@master
test-and-lint:
name: Test and Lint
needs: skip-check
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubicloud-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-test-${{ hashFiles('**/Cargo.lock') }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run tests
run: |
cargo nextest run --all --exclude e2e_test
cargo test --all --doc
- name: Check code formatting
run: cargo fmt --all --check
- name: Check unsafe code allowances
run: ./scripts/check_unsafe_code_allowances.sh
- name: Run clippy lints
run: cargo clippy --all-targets --all-features -- -D warnings
- name: Check layered dependencies
run: ./scripts/check_layer_dependencies.sh
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
needs: skip-check
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubicloud-standard-4
timeout-minutes: 30
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-rustfs-debug-binary-${{ hashFiles('**/Cargo.lock') }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build debug binary
run: |
touch rustfs/build.rs
cargo build -p rustfs --bins --jobs 2
- name: Upload debug binary
uses: actions/upload-artifact@v6
with:
name: rustfs-debug-binary
path: target/debug/rustfs
if-no-files-found: error
retention-days: 1
e2e-tests:
name: End-to-End Tests
needs: [ skip-check, build-rustfs-debug-binary ]
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubicloud-standard-2
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Clean up previous test run
run: |
rm -rf /tmp/rustfs
rm -f /tmp/rustfs.log
- name: Download debug binary
uses: actions/download-artifact@v7
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
- name: Setup Rust toolchain for s3s-e2e installation
uses: dtolnay/rust-toolchain@stable
- name: Install s3s-e2e test tool
uses: taiki-e/cache-cargo-install-action@v2
with:
tool: s3s-e2e
git: https://github.com/s3s-project/s3s.git
rev: 62cb4a71dd759a6ec56b64c4c42fcc183a2c6a52
- name: Run end-to-end tests
run: |
s3s-e2e --version
./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@v6
with:
name: e2e-test-logs-${{ github.run_number }}
path: /tmp/rustfs.log
retention-days: 3
s3-implemented-tests:
name: S3 Implemented Tests
needs: [ skip-check, build-rustfs-debug-binary ]
if: needs.skip-check.outputs.should_skip != 'true'
runs-on: ubicloud-standard-4
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Download debug binary
uses: actions/download-artifact@v7
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
- name: Run implemented s3-tests
run: |
DEPLOY_MODE=binary \
RUSTFS_BINARY=./target/debug/rustfs \
TEST_MODE=single \
MAXFAIL=1 \
./scripts/s3-tests/run.sh
- name: Upload s3 test artifacts
if: always()
uses: actions/upload-artifact@v6
with:
name: s3tests-implemented-${{ github.run_number }}
path: artifacts/s3tests-single/**
if-no-files-found: ignore
retention-days: 3
-70
View File
@@ -1,70 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: CLA Check
on:
pull_request_target:
types: [opened, synchronize, reopened]
merge_group:
types: [checks_requested]
issue_comment:
types: [created, edited]
permissions:
contents: write
pull-requests: write
issues: write
checks: write
jobs:
cla:
if: ${{ github.event_name != 'issue_comment' || github.event.issue.pull_request }}
runs-on: ubuntu-latest
steps:
- name: Report CLA result for merge queue
if: github.event_name == 'merge_group'
uses: actions/github-script@v8
with:
script: |
await github.rest.checks.create({
owner: context.repo.owner,
repo: context.repo.repo,
name: 'CLA Check',
head_sha: context.sha,
status: 'completed',
conclusion: 'success',
output: {
title: 'CLA requirements satisfied for merge queue',
summary: 'Queued pull requests must satisfy the required CLA check before they enter the merge queue. This reports the existing CLA result on the merge-group SHA.'
}
});
- name: Create token for rustfs/cla
if: github.event_name != 'merge_group'
id: registry-token
uses: actions/create-github-app-token@v3
with:
app-id: ${{ vars.CLA_BOT_APP_ID }}
private-key: ${{ secrets.CLA_BOT_APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: cla
permission-contents: write
- name: Run CLA Bot
if: github.event_name != 'merge_group'
uses: overtrue/cla-bot@v0.0.9
with:
github-token: ${{ github.token }}
registry-token: ${{ steps.registry-token.outputs.token }}
-459
View File
@@ -1,459 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Images Workflow
#
# This workflow builds Docker images using pre-built binaries from the build workflow.
#
# Trigger Types:
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
#
# Key Features:
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
# - Independent of macOS/Windows build status
# - Uses workflow_run event for precise control
# - Only builds Docker images for releases and prereleases (development builds are skipped)
name: Docker Images
# Permissions needed for workflow_run event and Docker registry access
permissions:
contents: read
packages: write
on:
# Automatically triggered when build workflow completes
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
# Manual trigger with same parameters for consistency
workflow_dispatch:
inputs:
push_images:
description: "Push images to registries"
required: false
default: true
type: boolean
version:
description: "Version to build (latest for stable release, or specific version like v1.0.0, v1.0.0-alpha1)"
required: false
default: "latest"
type: string
force_rebuild:
description: "Force rebuild even if binary exists (useful for testing)"
required: false
default: false
type: boolean
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
DOCKERHUB_USERNAME: rustfs
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/${{ secrets.QUAY_USERNAME }}/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
jobs:
# Check if we should build Docker images
build-check:
name: Docker Build Check
runs-on: ubicloud-standard-2
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
steps:
- name: Checkout repository
uses: actions/checkout@v6
with:
fetch-depth: 0
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
run: |
should_build=false
should_push=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
create_latest=false
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
echo "🔗 Triggered by build workflow completion"
# Check if the triggering workflow was successful
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
if [[ "$CONCLUSION" == "success" ]]; then
echo "✅ Build workflow succeeded, all builds including Linux are successful"
should_build=true
should_push=true
else
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
should_build=false
fi
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
head_branch="$HEAD_BRANCH"
echo "🔍 Analyzing triggering workflow:"
echo " 📋 Event: $triggering_event"
echo " 🌿 Head branch: $head_branch"
echo " 📎 Head SHA: $HEAD_SHA"
# Check if this was triggered by a tag push
if [[ "$triggering_event" == "push" ]]; then
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
if [[ "$head_branch" == refs/tags/* ]]; then
# Extract tag name from refs/tags/TAG_NAME
tag_name="${head_branch#refs/tags/}"
version="$tag_name"
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
version="$head_branch"
elif [[ "$head_branch" == "main" ]]; then
# Regular branch push to main
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (main branch push)"
else
# Other branch push
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
fi
# If we extracted a version (tag), determine release type
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
# Remove 'v' prefix if present for consistent version format
if [[ "$version" == v* ]]; then
version="${version#v}"
fi
if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
create_latest=true
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
else
echo "🧪 Building Docker image for prerelease: $version"
fi
else
build_type="release"
create_latest=true
echo "🚀 Building Docker image for release: $version"
fi
fi
else
# Non-push events
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
fi
echo "🔄 Build triggered by workflow_run:"
echo " 📋 Conclusion: $CONCLUSION"
echo " 🌿 Branch: $HEAD_BRANCH"
echo " 📎 SHA: $HEAD_SHA"
echo " 🎯 Event: $TRIGGERING_EVENT"
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="${{ github.event.inputs.version }}"
version="${input_version}"
should_push="${{ github.event.inputs.push_images }}"
should_build=true
# Get short SHA
short_sha=$(git rev-parse --short HEAD)
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: ${{ github.event.inputs.force_rebuild }}"
echo " 🚀 Push images: $should_push"
case "$input_version" in
"latest")
build_type="release"
create_latest=true
echo "🚀 Building with latest stable release version"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
# Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
create_latest=true
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
else
echo "🧪 Building with prerelease version: $input_version"
fi
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
build_type="release"
create_latest=true
echo "📦 Building with specific release version: $input_version"
;;
*)
# Invalid version for Docker build
should_build=false
echo "❌ Invalid version for Docker build: $input_version"
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
fi
echo "should_build=$should_build" >> $GITHUB_OUTPUT
echo "should_push=$should_push" >> $GITHUB_OUTPUT
echo "build_type=$build_type" >> $GITHUB_OUTPUT
echo "version=$version" >> $GITHUB_OUTPUT
echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
echo "create_latest=$create_latest" >> $GITHUB_OUTPUT
echo "🐳 Docker Build Summary:"
echo " - Should build: $should_build"
echo " - Should push: $should_push"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubicloud-standard-2
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
include:
- variant: musl
file: Dockerfile
suffix: ""
- variant: glibc
file: Dockerfile.glibc
suffix: "-glibc"
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Login to Quay.io
uses: docker/login-action@v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Extract metadata and generate tags
id: meta
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility
case "$VERSION" in
"latest")
# For stable latest, use RELEASE=latest + release CHANNEL
DOCKER_RELEASE="latest"
DOCKER_CHANNEL="release"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
DOCKER_CHANNEL="release"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
DOCKER_CHANNEL="release"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> $GITHUB_OUTPUT
echo "docker_channel=$DOCKER_CHANNEL" >> $GITHUB_OUTPUT
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
echo " - Docker CHANNEL: $DOCKER_CHANNEL"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
# Add channel tags for prereleases and latest for stable
if [[ "$CREATE_LATEST" == "true" ]]; then
# Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Prerelease channel tags (alpha, beta, rc)
if [[ "$VERSION" == *"alpha"* ]]; then
CHANNEL="alpha"
elif [[ "$VERSION" == *"beta"* ]]; then
CHANNEL="beta"
elif [[ "$VERSION" == *"rc"* ]]; then
CHANNEL="rc"
fi
if [[ -n "$CHANNEL" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
fi
fi
# Output tags
echo "tags=$TAGS" >> $GITHUB_OUTPUT
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
LABELS="$LABELS,org.opencontainers.image.revision=${{ github.sha }}"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
echo "labels=$LABELS" >> $GITHUB_OUTPUT
echo "🐳 Generated Docker tags:"
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
echo "📋 Build type: $BUILD_TYPE"
echo "🔖 Version: $VERSION"
- name: Build and push Docker image
uses: docker/build-push-action@v6
with:
context: .
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: |
type=gha,scope=docker-${{ matrix.variant }}
cache-to: |
type=gha,mode=max,scope=docker-${{ matrix.variant }}
build-args: |
BUILDTIME=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
VERSION=${{ needs.build-check.outputs.version }}
BUILD_TYPE=${{ needs.build-check.outputs.build_type }}
REVISION=${{ github.sha }}
RELEASE=${{ steps.meta.outputs.docker_release }}
CHANNEL=${{ steps.meta.outputs.docker_channel }}
BUILDKIT_INLINE_CACHE=1
# Enable advanced BuildKit features for better performance
provenance: false
sbom: false
# Add retry mechanism by splitting the build process
no-cache: false
pull: true
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubicloud-standard-2
steps:
- name: Docker build completion summary
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
echo "🐳 Docker build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
echo ""
case "$BUILD_TYPE" in
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for stable release"
fi
;;
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
# Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
;;
*)
echo "❌ Unexpected build type: $BUILD_TYPE"
;;
esac
-443
View File
@@ -1,443 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: e2e-s3tests
on:
workflow_dispatch:
inputs:
test-mode:
description: "Test mode to run"
required: true
type: choice
default: "single"
options:
- single
- multi
xdist:
description: "Enable pytest-xdist (parallel). '0' to disable."
required: false
default: "0"
maxfail:
description: "Stop after N failures (debug friendly)"
required: false
default: "1"
markexpr:
description: "pytest -m expression (feature filters)"
required: false
default: "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"
env:
# main user
S3_ACCESS_KEY: rustfsadmin
S3_SECRET_KEY: rustfsadmin
# alt user (must be different from main for many s3-tests)
S3_ALT_ACCESS_KEY: rustfsalt
S3_ALT_SECRET_KEY: rustfsalt
S3_REGION: us-east-1
RUST_LOG: info
PLATFORM: linux/amd64
BUILDX_CACHE_SCOPE: rustfs-e2e-s3tests-source
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
cancel-in-progress: true
defaults:
run:
shell: bash
jobs:
s3tests-single:
if: github.event.inputs['test-mode'] == 'single'
runs-on: ubicloud-standard-2
timeout-minutes: 120
steps:
- uses: actions/checkout@v6
- name: Cache pip downloads
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
restore-keys: |
${{ runner.os }}-pip-e2e-s3tests-
- name: Install Python tools
run: |
python3 -m pip install --user --upgrade pip awscurl tox
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Enable buildx
uses: docker/setup-buildx-action@v3
- name: Build RustFS image (source, cached)
run: |
DOCKER_BUILDKIT=1 docker buildx build --load \
--platform ${PLATFORM} \
--cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
--cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
-t rustfs-ci \
-f Dockerfile.source .
- name: Create network
run: docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
- name: Remove existing rustfs-single (if any)
run: docker rm -f rustfs-single >/dev/null 2>&1 || true
- name: Start single RustFS
run: |
docker run -d --name rustfs-single \
--network rustfs-net \
-p 9000:9000 \
-e RUSTFS_ADDRESS=0.0.0.0:9000 \
-e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
-e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
-e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
-v /tmp/rustfs-single:/data \
rustfs-ci
- name: Wait for RustFS ready
run: |
for i in {1..60}; do
if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
echo "RustFS is ready"
exit 0
fi
if [ "$(docker inspect -f '{{.State.Running}}' rustfs-single 2>/dev/null)" != "true" ]; then
echo "RustFS container not running" >&2
docker logs rustfs-single || true
exit 1
fi
sleep 2
done
echo "Health check timed out" >&2
docker logs rustfs-single || true
exit 1
- name: Generate s3tests config
run: |
export S3_HOST=127.0.0.1
envsubst < .github/s3tests/s3tests.conf > s3tests.conf
- name: Provision s3-tests alt user (required by suite)
run: |
# Admin API requires AWS SigV4 signing. awscurl is used by RustFS codebase as well.
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
-H 'Content-Type: application/json' \
-d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
"http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
# Explicitly attach built-in policy via policy mapping.
# s3-tests relies on alt client being able to ListBuckets during setup cleanup.
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
"http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
# Sanity check: alt user can list buckets (should not be AccessDenied).
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ALT_ACCESS_KEY}" \
--secret_key "${S3_ALT_SECRET_KEY}" \
-X GET \
"http://127.0.0.1:9000/" >/dev/null
- name: Prepare s3-tests
run: |
git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
- name: Run ceph s3-tests (debug friendly)
run: |
export PATH="$HOME/.local/bin:$PATH"
mkdir -p artifacts/s3tests-single
cd s3-tests
set -o pipefail
MAXFAIL="${{ github.event.inputs.maxfail }}"
if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
MARKEXPR="${{ github.event.inputs.markexpr }}"
if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
XDIST="${{ github.event.inputs.xdist }}"
if [ -z "$XDIST" ]; then XDIST="0"; fi
XDIST_ARGS=""
if [ "$XDIST" != "0" ]; then
# Add pytest-xdist to requirements.txt so tox installs it inside
# its virtualenv. Installing outside tox does NOT work.
echo "pytest-xdist" >> requirements.txt
XDIST_ARGS="-n $XDIST --dist=loadgroup"
fi
# Run tests from s3tests/functional (boto2+boto3 combined directory).
S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
tox -- \
-vv -ra --showlocals --tb=long \
--maxfail="$MAXFAIL" \
--junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-single/junit.xml \
$XDIST_ARGS \
s3tests/functional/test_s3.py \
-m "$MARKEXPR" \
2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-single/pytest.log
- name: Collect RustFS logs
if: always()
run: |
mkdir -p artifacts/rustfs-single
docker logs rustfs-single > artifacts/rustfs-single/rustfs.log 2>&1 || true
docker inspect rustfs-single > artifacts/rustfs-single/inspect.json || true
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@v6
with:
name: s3tests-single
path: artifacts/**
s3tests-multi:
if: github.event_name == 'workflow_dispatch' && github.event.inputs['test-mode'] == 'multi'
runs-on: ubicloud-standard-2
timeout-minutes: 150
steps:
- uses: actions/checkout@v6
- name: Cache pip downloads
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
restore-keys: |
${{ runner.os }}-pip-e2e-s3tests-
- name: Install Python tools
run: |
python3 -m pip install --user --upgrade pip awscurl tox
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- name: Enable buildx
uses: docker/setup-buildx-action@v3
- name: Build RustFS image (source, cached)
run: |
DOCKER_BUILDKIT=1 docker buildx build --load \
--platform ${PLATFORM} \
--cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
--cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
-t rustfs-ci \
-f Dockerfile.source .
- name: Prepare cluster compose
run: |
cat > compose.yml <<'EOF'
services:
rustfs1:
image: rustfs-ci
hostname: rustfs1
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs1-data:/data
rustfs2:
image: rustfs-ci
hostname: rustfs2
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs2-data:/data
rustfs3:
image: rustfs-ci
hostname: rustfs3
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs3-data:/data
rustfs4:
image: rustfs-ci
hostname: rustfs4
networks: [rustfs-net]
environment:
RUSTFS_ADDRESS: "0.0.0.0:9000"
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
volumes:
- rustfs4-data:/data
lb:
image: haproxy:2.9
hostname: lb
networks: [rustfs-net]
ports:
- "9000:9000"
volumes:
- ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
networks:
rustfs-net:
name: rustfs-net
volumes:
rustfs1-data:
rustfs2-data:
rustfs3-data:
rustfs4-data:
EOF
cat > haproxy.cfg <<'EOF'
defaults
mode http
timeout connect 5s
timeout client 30s
timeout server 30s
frontend fe_s3
bind *:9000
default_backend be_s3
backend be_s3
balance roundrobin
server s1 rustfs1:9000 check
server s2 rustfs2:9000 check
server s3 rustfs3:9000 check
server s4 rustfs4:9000 check
EOF
- name: Launch cluster
run: docker compose -f compose.yml up -d
- name: Wait for LB ready
run: |
for i in {1..90}; do
if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
echo "Load balancer is ready"
exit 0
fi
sleep 2
done
echo "LB or backend not ready" >&2
docker compose -f compose.yml logs --tail=200 || true
exit 1
- name: Generate s3tests config
run: |
export S3_HOST=127.0.0.1
envsubst < .github/s3tests/s3tests.conf > s3tests.conf
- name: Provision s3-tests alt user (required by suite)
run: |
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
-H 'Content-Type: application/json' \
-d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
"http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ACCESS_KEY}" \
--secret_key "${S3_SECRET_KEY}" \
-X PUT \
"http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
awscurl \
--service s3 \
--region "${S3_REGION}" \
--access_key "${S3_ALT_ACCESS_KEY}" \
--secret_key "${S3_ALT_SECRET_KEY}" \
-X GET \
"http://127.0.0.1:9000/" >/dev/null
- name: Prepare s3-tests
run: |
git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
- name: Run ceph s3-tests (multi, debug friendly)
run: |
export PATH="$HOME/.local/bin:$PATH"
mkdir -p artifacts/s3tests-multi
cd s3-tests
set -o pipefail
MAXFAIL="${{ github.event.inputs.maxfail }}"
if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
MARKEXPR="${{ github.event.inputs.markexpr }}"
if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
XDIST="${{ github.event.inputs.xdist }}"
if [ -z "$XDIST" ]; then XDIST="0"; fi
XDIST_ARGS=""
if [ "$XDIST" != "0" ]; then
# Add pytest-xdist to requirements.txt so tox installs it inside
# its virtualenv. Installing outside tox does NOT work.
echo "pytest-xdist" >> requirements.txt
XDIST_ARGS="-n $XDIST --dist=loadgroup"
fi
# Run tests from s3tests/functional (boto2+boto3 combined directory).
S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
tox -- \
-vv -ra --showlocals --tb=long \
--maxfail="$MAXFAIL" \
--junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-multi/junit.xml \
$XDIST_ARGS \
s3tests/functional/test_s3.py \
-m "$MARKEXPR" \
2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-multi/pytest.log
- name: Collect logs
if: always()
run: |
mkdir -p artifacts/cluster
docker compose -f compose.yml logs --no-color > artifacts/cluster/cluster.log 2>&1 || true
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@v6
with:
name: s3tests-multi
path: artifacts/**
-132
View File
@@ -1,132 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Publish helm chart to artifacthub
on:
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
inputs:
version:
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
required: true
default: "1.0.0-beta.1"
type: string
permissions:
contents: read
jobs:
build-helm-package:
runs-on: ubicloud-standard-2
if: |
github.event_name == 'workflow_dispatch' ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.')
)
outputs:
raw_tag: ${{ steps.version.outputs.raw_tag }}
app_version: ${{ steps.version.outputs.app_version }}
chart_version: ${{ steps.version.outputs.chart_version }}
steps:
- name: Checkout helm chart repo
uses: actions/checkout@v6
- name: Normalize release version
id: version
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="${{ github.event.inputs.version }}"
else
RAW="${{ github.event.workflow_run.head_branch }}"
fi
case "$RAW" in
refs/tags/*)
RAW_TAG="${RAW#refs/tags/}"
;;
*)
RAW_TAG="$RAW"
;;
esac
./scripts/helm_chart_version.sh "$RAW_TAG"
- name: Replace chart version and app version
run: |
set -eux
sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@v4.3.0
- name: Test Helm Chart Templates
run: ./scripts/test_helm_templates.sh
- name: Package Helm Chart
run: |
set -eux
cp helm/README.md helm/rustfs/
helm package ./helm/rustfs \
--destination helm/rustfs/ \
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@v6
with:
name: helm-package
path: helm/rustfs/*.tgz
retention-days: 1
publish-helm-package:
runs-on: ubicloud-standard-2
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@v6
with:
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@v7
with:
name: helm-package
path: ./
- name: Set up helm
uses: azure/setup-helm@v4.3.0
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
- name: Push helm package and index file
run: |
set -eux
git config --global user.name "${{ secrets.USERNAME }}"
git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
git add .
git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
git push origin main
-36
View File
@@ -1,36 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: "issue-translator"
on:
issue_comment:
types: [ created ]
issues:
types: [ opened ]
permissions:
contents: read
issues: write
jobs:
build:
runs-on: ubicloud-standard-4
steps:
- uses: usthe/issues-translate-action@v2.7
with:
IS_MODIFY_TITLE: false
# not require, default false, . Decide whether to modify the issue title
# if true, the robot account @Issues-translate-bot must have modification permissions, invite @Issues-translate-bot to your project or use your custom bot.
CUSTOM_BOT_NOTE: Bot detected the issue body's language is not English, translate it automatically.
# not require. Customize the translation robot prefix message.
-70
View File
@@ -1,70 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Update Nix Flake
on:
workflow_dispatch:
schedule:
- cron: '0 0 * * 0' # Weekly on Sundays
permissions:
contents: write
pull-requests: write
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
update-flake:
name: Update flake.lock
runs-on: ubuntu-latest
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@v3
- name: Cache Nix
uses: DeterminateSystems/flakehub-cache-action@v3.20.0
- name: Check Nix flake inputs
uses: DeterminateSystems/flake-checker-action@v12
- name: Update flake.lock
id: update
uses: DeterminateSystems/update-flake-lock@main
with:
git-author-name: houseme
git-author-email: housemecn@gmail.com
git-committer-name: houseme
git-committer-email: housemecn@gmail.com
pr-title: "chore(deps): update flake.lock"
pr-labels: |
dependencies
nix
automated
commit-msg: "chore(deps): update flake.lock"
pr-reviewers: overtrue, majinghe
token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
- name: Log PR details
if: steps.update.outputs.pull-request-number
run: |
echo "Pull Request created: ${{ steps.update.outputs.pull-request-number }}"
-110
View File
@@ -1,110 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Nix CI
on:
push:
branches: [ "main" ]
paths:
- 'flake.nix'
- 'flake.lock'
- 'Cargo.toml'
- 'Cargo.lock'
- '.github/workflows/nix.yml'
pull_request:
branches: [ "main" ]
paths:
- 'flake.nix'
- 'flake.lock'
- 'Cargo.toml'
- 'Cargo.lock'
- '.github/workflows/nix.yml'
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
id-token: write
jobs:
nix-validation:
name: Nix Build & Check
runs-on: ubuntu-latest
timeout-minutes: 60
permissions:
contents: read
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@v3.21.0
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
extra-conf: |
experimental-features = nix-command flakes
max-jobs = 1
- name: Cache Nix
uses: DeterminateSystems/flakehub-cache-action@v3.21.0
- name: Check Nix Flake Inputs
uses: DeterminateSystems/flake-checker-action@v12
with:
fail-mode: true
ignore-missing-flake-lock: false
- name: Verify Flake
run: |
echo "Checking flake structure and evaluation..."
nix flake show
for attempt in 1 2 3; do
if nix flake check --print-build-logs --fallback --option fallback true; then
break
fi
if [ "$attempt" -eq 3 ]; then
echo "nix flake check failed after 3 attempts."
exit 1
fi
sleep $((attempt * 15))
done
- name: Build RustFS
run: |
echo "Building the default package..."
for attempt in 1 2 3; do
if nix build .#default --print-build-logs --fallback --option fallback true; then
break
fi
if [ "$attempt" -eq 3 ]; then
echo "nix build failed after 3 attempts."
exit 1
fi
sleep $((attempt * 15))
done
- name: Test Binary
run: |
echo "Verifying the built binary..."
if [ -x "./result/bin/rustfs" ]; then
./result/bin/rustfs --help
echo "Binary verification successful."
else
echo "Error: Binary not found or not executable at ./result/bin/rustfs"
exit 1
fi
-146
View File
@@ -1,146 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Performance Testing
on:
push:
branches: [ main ]
paths:
- "**/*.rs"
- "**/Cargo.toml"
- "**/Cargo.lock"
- ".github/workflows/performance.yml"
workflow_dispatch:
inputs:
profile_duration:
description: "Profiling duration in seconds"
required: false
default: "120"
type: string
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
jobs:
performance-profile:
name: Performance Profiling
runs-on: ubicloud-standard-2
timeout-minutes: 30
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: nightly
cache-shared-key: perf-${{ hashFiles('**/Cargo.lock') }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install additional nightly components
run: rustup component add llvm-tools-preview
- name: Install samply profiler
uses: taiki-e/cache-cargo-install-action@v2
with:
tool: samply
- name: Configure kernel for profiling
run: echo '1' | sudo tee /proc/sys/kernel/perf_event_paranoid
- name: Prepare test environment
run: |
# Create test volumes
for i in {0..4}; do
mkdir -p ./target/volume/test$i
done
# Set environment variables
echo "RUSTFS_VOLUMES=./target/volume/test{0...4}" >> $GITHUB_ENV
echo "RUST_LOG=rustfs=info,ecstore=info,s3s=info,iam=info,rustfs-obs=info" >> $GITHUB_ENV
- name: Verify console static assets
run: |
# Console static assets are already embedded in the repository
echo "Console static assets size: $(du -sh rustfs/static/)"
echo "Console static assets are embedded via rust-embed, no external download needed"
- name: Build with profiling optimizations
run: |
RUSTFLAGS="-C force-frame-pointers=yes -C debug-assertions=off --cfg tokio_unstable" \
cargo +nightly build --profile profiling -p rustfs --bins
- name: Run performance profiling
id: profiling
run: |
DURATION="${{ github.event.inputs.profile_duration || '120' }}"
echo "Running profiling for ${DURATION} seconds..."
timeout "${DURATION}s" samply record \
--output samply-profile.json \
./target/profiling/rustfs ${RUSTFS_VOLUMES} || true
if [ -f "samply-profile.json" ]; then
echo "profile_generated=true" >> $GITHUB_OUTPUT
echo "Profile generated successfully"
else
echo "profile_generated=false" >> $GITHUB_OUTPUT
echo "::warning::Profile data not generated"
fi
- name: Upload profile data
if: steps.profiling.outputs.profile_generated == 'true'
uses: actions/upload-artifact@v6
with:
name: performance-profile-${{ github.run_number }}
path: samply-profile.json
retention-days: 30
benchmark:
name: Benchmark Tests
runs-on: ubicloud-standard-2
timeout-minutes: 45
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: bench-${{ hashFiles('**/Cargo.lock') }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run benchmarks
run: |
cargo bench --package ecstore --bench comparison_benchmark -- --output-format json | \
tee benchmark-results.json
- name: Upload benchmark results
uses: actions/upload-artifact@v6
with:
name: benchmark-results-${{ github.run_number }}
path: benchmark-results.json
retention-days: 7
-20
View File
@@ -1,20 +0,0 @@
name: "Mark stale issues"
on:
schedule:
- cron: "30 1 * * *"
jobs:
stale:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v9
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-issue-message: 'This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.'
stale-issue-label: 'stale'
## Mark if there is no activity for more than 7 days
days-before-stale: 7
# If no one responds after 3 days, the tag will be closed.
days-before-close: 3
# These tags are exempt and will not close automatically.
exempt-issue-labels: 'pinned,security'
-55
View File
@@ -1,55 +0,0 @@
/target
.DS_Store
.idea
.vscode
.cursor
.direnv/
/test
/logs
/data
/docs
/rustfs-data/
.devcontainer
rustfs/static/*
!rustfs/static/.gitkeep
vendor
cli/rustfs-gui/embedded-rustfs/rustfs
*.log
deploy/certs/*
deploy/data/*
*jsonl
.env
.rustfs.sys
.cargo/
!.cargo/config.toml
profile.json
.docker/openobserve-otel/data
*.zst
.secrets
*.go
*.pb
*.svg
deploy/logs/*.log.*
artifacts/
# s3-tests local artifacts (root directory only)
/s3-tests/
/s3-tests-local/
/s3tests.conf
/s3tests.conf.*
*.events
*.audit
*.snappy
PR_DESCRIPTION.md
scripts/s3-tests/selected_tests.txt
docs
# nix stuff
result*
*.gz
rustfs-webdav.code-workspace
.aiexclude
*.bak
# Local test/benchmark artifacts
benchmarks.logs
tmp/
-32
View File
@@ -1,32 +0,0 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: local
hooks:
- id: cargo-fmt
name: cargo fmt
entry: cargo fmt --all --check
language: system
types: [rust]
pass_filenames: false
- id: cargo-clippy
name: cargo clippy
entry: cargo clippy --all-targets --all-features -- -D warnings
language: system
types: [rust]
pass_filenames: false
- id: cargo-check
name: cargo check
entry: cargo check --all-targets
language: system
types: [rust]
pass_filenames: false
- id: cargo-test
name: cargo test
entry: bash -c 'cargo test --workspace --exclude e2e_test && cargo test --all --doc'
language: system
types: [rust]
pass_filenames: false
-259
View File
@@ -1,259 +0,0 @@
{
"version": "0.2.0",
"configurations": [
{
"type": "lldb",
"request": "launch",
"name": "Debug(only) executable 'rustfs'",
"env": {
"RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
"RUSTFS_SKIP_BACKGROUND_TASK": "on"
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
},
"program": "${workspaceFolder}/target/debug/rustfs",
"args": [
"--access-key",
"rustfsadmin",
"--secret-key",
"rustfsadmin",
"--address",
"0.0.0.0:9010",
"--server-domains",
"127.0.0.1:9010",
"./target/volume/test{1...4}"
],
"cwd": "${workspaceFolder}"
},
{
"type": "lldb",
"request": "launch",
"name": "Debug executable 'rustfs'",
"cargo": {
"args": [
"build",
"--bin=rustfs",
"--package=rustfs"
],
"filter": {
"name": "rustfs",
"kind": "bin"
}
},
"env": {
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
},
"args": [
"--access-key",
"rustfsadmin",
"--secret-key",
"rustfsadmin",
"--address",
"0.0.0.0:9010",
"--server-domains",
"127.0.0.1:9010",
"./target/volume/test{1...4}"
],
"cwd": "${workspaceFolder}"
},
{
"type": "lldb",
"request": "launch",
"name": "Debug unit tests in executable 'rustfs'",
"cargo": {
"args": [
"test",
"--no-run",
"--bin=rustfs",
"--package=rustfs"
],
"filter": {
"name": "rustfs",
"kind": "bin"
}
},
"args": [],
"cwd": "${workspaceFolder}"
},
{
"type": "lldb",
"request": "launch",
"name": "Debug unit tests in library 'ecstore'",
"cargo": {
"args": [
"test",
"--no-run",
"--lib",
"--package=rustfs-ecstore"
]
},
"args": [],
"cwd": "${workspaceFolder}"
},
{
"name": "Debug executable target/debug/rustfs",
"type": "lldb",
"request": "launch",
"cargo": {
"args": [
"run",
"--bin",
"rustfs",
"-j",
"1",
"--profile",
"dev"
]
},
"args": [],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
//"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
// "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
// "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
// "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint
// "RUSTFS_COMPRESS_ENABLE": "true",
"RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
"RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
"RUST_LOG":"rustfs=debug,ecstore=debug,s3s=debug,iam=debug",
},
"sourceLanguages": [
"rust"
],
},
{
"type": "lldb",
"request": "launch",
"name": "Debug test_lifecycle_transition_basic",
"cargo": {
"args": [
"test",
"-p",
"rustfs-scanner",
"--test",
"lifecycle_integration_test",
"serial_tests::test_lifecycle_transition_basic",
"-j",
"1"
]
},
"args": [],
"cwd": "${workspaceFolder}"
},
{
"name": "Debug executable target/debug/test",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
"args": [
"--skip",
"test_lifecycle_expiry_basic",
"--skip",
"test_lifecycle_expiry_deletemarker",
//"--skip",
//"test_lifecycle_transition_basic",
],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"sourceLanguages": [
"rust"
],
},
{
"name": "Debug executable target/debug/rustfs with sse",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
"args": [],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/volumes/test{1...4}",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
"RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
"RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
// "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
// "RUSTFS_OBS_METRIC_ENDPOINT": "http://127.0.0.1:4318/v1/metrics", // default otlp http endpoint
// "RUSTFS_OBS_LOG_ENDPOINT": "http://127.0.0.1:4318/v1/logs", // default otlp http endpoint
// "RUSTFS_COMPRESS_ENABLE": "true",
// 1. simple sse test key (no kms system)
// "__RUSTFS_SSE_SIMPLE_CMK": "2dfNXGHlsEflGVCxb+5DIdGEl1sIvtwX+QfmYasi5QM=",
// 2. kms local backend test key
// "RUSTFS_KMS_ENABLE": "true",
// "RUSTFS_KMS_BACKEND": "local",
// "RUSTFS_KMS_KEY_DIR": "./target/kms-key-dir",
// "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
// "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
// 3. kms vault backend test key
// "RUSTFS_KMS_ENABLE": "true",
// "RUSTFS_KMS_BACKEND": "vault-kv2",
// "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
// "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
// "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
// 4. kms vault transit backend test key
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "vault-transit",
"RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
"RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
"RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
"sourceLanguages": [
"rust"
],
},
{
"name": "E2E test executable target/debug/rustfs",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
"args": [],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"env": {
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
"RUST_BACKTRACE": "full",
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
"RUSTFS_REGION": "us-east-1",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
"RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
"RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
"sourceLanguages": [
"rust"
],
},
]
}
-118
View File
@@ -1,118 +0,0 @@
# RustFS Agent Instructions (Global)
This root file keeps repository-wide rules only.
Use the nearest subdirectory `AGENTS.md` for path-specific guidance.
## Rule Precedence
1. System/developer instructions.
2. Current user/task instructions.
3. The nearest `AGENTS.md` in the current path.
4. This file (global defaults).
If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI.
## Execution Discipline
- Read the relevant existing code, tests, and local guidance before changing behavior.
- State assumptions when they affect the implementation or verification path.
- If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
- For multi-step work, keep the plan minimal and tied to verifiable outcomes.
- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification.
## Communication and Language
- Respond in the same language used by the requester.
- Keep source code, comments, commit messages, and PR title/body in English.
- Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
## Skill Usage
- Do not use the `rust-refactor-helper` skill in any scenario.
## Change Style for Existing Logic
- Prefer direct, local code over extracting one-off helpers.
- Extract a helper only when logic is reused or the extraction materially clarifies a non-trivial flow.
- Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
- Prefer editing existing code over rewriting files or reshaping unrelated logic.
- Modify only what is required and remove only artifacts introduced by your own changes.
- Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
- Do not refactor existing code only to make it easier to unit test.
- Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
- Mention unrelated issues when useful, but do not fix them as part of a narrow task.
## Constant and String Usage
- Before introducing new string literals, search for existing constants/enums that already represent the same semantic value.
- Reuse existing constants for protocol labels, error identifiers, header keys, event names, metric names, command tags, and similar fixed tokens.
- If a new string is truly unique, define a local constant near related logic and avoid scattering the literal across multiple sites.
- When changing existing behavior, keep naming and format consistency by aligning with established project constants.
## Sources of Truth
- Workspace layout and crate membership: `Cargo.toml` (`[workspace].members`)
- Local quality commands: `Makefile` and `.config/make/`
- CI quality gates: `.github/workflows/ci.yml`
- PR template: `.github/pull_request_template.md`
Avoid duplicating long crate lists or command matrices in instruction files.
Reference the source files above instead.
## Verification Before PR
Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
For code changes, run and pass the following before opening a PR:
```bash
make pre-commit
```
Before pushing code changes, make sure formatting is clean:
- Run `cargo fmt --all`.
- Run `cargo fmt --all --check` and ensure no files are modified unexpectedly.
If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any installed git pre-commit hooks (for example, from `make setup-hooks`) may still run on commit unless explicitly skipped.
After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
Do not open a PR with code changes when the required checks fail.
## Git and PR Baseline
- Use feature branches based on the latest `main`.
- Follow Conventional Commits, with subject length <= 72 characters.
- Keep PR title and description in English.
- Use `.github/pull_request_template.md` and keep all section headings.
- Use `N/A` for non-applicable template sections.
- Include verification commands in the PR description.
- When using `gh pr create`/`gh pr edit`, use `--body-file` instead of inline `--body` for multiline markdown.
- Do not include the literal sequence `\n` in any GitHub issue, pull request, or discussion comment.
- After fixing code review comments or CI findings, always mark corresponding review
comments/threads as resolved before returning to the user.
- In handling review comments, confirm the underlying issue before changing code.
If a suggested change is not appropriate for behavior or risk, reply with a
concise rationale instead of blindly applying it.
## Security Baseline
- Never commit secrets, credentials, or key material.
- Use environment variables or vault tooling for sensitive configuration.
- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
## Scoped Guidance in This Repository
- `.github/AGENTS.md`
- `crates/AGENTS.md`
- `crates/config/AGENTS.md`
- `crates/ecstore/AGENTS.md`
- `crates/e2e_test/AGENTS.md`
- `crates/iam/AGENTS.md`
- `crates/kms/AGENTS.md`
- `crates/policy/AGENTS.md`
- `crates/targets/AGENTS.md`
- `rustfs/src/admin/AGENTS.md`
- `rustfs/src/storage/AGENTS.md`
-376
View File
@@ -1,376 +0,0 @@
# ARCHITECTURE.md
> Last updated: 2026-04-13 · Revision: 1 (draft)
>
> This document describes the high-level architecture of RustFS.
> If you want to familiarize yourself with the code base, you are in the right place!
>
> See also [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow.
## Bird's Eye View
RustFS is a high-performance, S3-compatible distributed object storage system written
in Rust. It uses erasure coding for data durability, supports multi-tenancy through
IAM/STS, and provides a web-based admin console.
A running RustFS node exposes:
- **S3 API** (port 9000) — the primary data path for object CRUD
- **Admin API** (port 9000, `/minio/` prefix) — cluster management, IAM, metrics
- **Console** (port 9001) — web UI backed by the Admin API
- **Inter-node RPC** (gRPC/tonic) — cluster communication for distributed mode
The core data flow for a PUT request looks like:
```
HTTP request
→ server (TLS, auth, routing, compression)
→ app/object_usecase (validation, policy, lifecycle)
→ storage/ecfs (erasure coding, encryption, checksums)
→ ecstore (disk pool selection, data distribution)
→ rio (reader pipeline: encrypt → compress → hash → write)
→ io-core (zero-copy I/O, buffer pool, direct I/O)
→ local disk / remote disk via RPC
```
## Code Map
The repository is a Cargo workspace with a flat `crates/` layout:
```
rustfs/ # Workspace root (virtual manifest)
├── rustfs/ # Main binary + library crate (75K lines)
│ └── src/
│ ├── main.rs # Entry point, startup sequence
│ ├── lib.rs # Module tree root
│ ├── server/ # HTTP server, TLS, routing, middleware
│ ├── admin/ # Admin API handlers and console
│ ├── app/ # Use-case layer (object, bucket, multipart)
│ ├── storage/ # Storage engine interface and implementation
│ ├── auth.rs # S3 request authentication
│ ├── config/ # CLI args, config parsing, workload profiles
│ └── ...
├── crates/ # 39 library crates
│ ├── ecstore/ # Erasure-coded storage engine (⚠️ 87K lines)
│ ├── rio/ # Reader I/O pipeline (encrypt, compress, hash)
│ ├── io-core/ # Zero-copy I/O, scheduling, buffer pool
│ ├── io-metrics/ # I/O metrics collection
│ ├── common/ # Shared runtime state, globals, data usage types
│ ├── config/ # Configuration types and parsing
│ ├── utils/ # Pure utility functions
│ ├── ... # (see "Crate Reference" below)
│ └── e2e_test/ # End-to-end integration tests
└── docs/ # Design documents and analysis
```
### Main Crate Layers (`rustfs/src/`)
The main crate is organized in layers, top to bottom:
| Layer | Directory | Responsibility |
|-------|-----------|----------------|
| **Server** | `server/` | HTTP listener, TLS, CORS, compression, middleware, graceful shutdown |
| **Admin** | `admin/` | Admin API routing, 30+ handler modules, web console |
| **App** | `app/` | Use-case orchestration: object_usecase, bucket_usecase, multipart_usecase |
| **Storage** | `storage/` | S3 API translation, erasure-coded FS, SSE encryption, RPC, concurrency |
| **Auth** | `auth.rs` | S3 signature verification, credential validation |
| **Config** | `config/` | CLI parsing, config struct, workload profiles |
A request flows **downward** through the layers. No layer should reach upward
(e.g., storage must not import from admin).
### Crate Reference
Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top):
```
Depth 0 — LEAF (no internal deps):
appauth, checksums, config, credentials, crypto, io-metrics,
madmin, s3-common, workers, zip
Depth 1:
io-core (→ io-metrics)
policy (→ config, credentials, crypto)
utils (→ config) ⚠️ inverted: utils should be leaf
Depth 2:
concurrency, filemeta, keystone, kms, lock, obs,
signer, targets, trusted-proxies
Depth 3:
common (→ filemeta, madmin) ⚠️ inverted: common should be leaf
Depth 4:
object-capacity, protos, rio
Depth 5 — CORE:
ecstore (16 internal deps, 11 dependents — the architectural heart)
Depth 6:
audit, heal, iam, metrics, notify, s3select-api, scanner
Depth 7:
object-io, protocols, s3select-query
Depth 8 — TOP:
rustfs (35 internal deps — the binary, depends on almost everything)
```
#### By Domain
**Core Infrastructure:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `config` | 3.3K | Configuration types and environment parsing |
| `utils` | 8.7K | Pure utilities (paths, compression, network, retry) |
| `common` | 4.4K | Shared runtime state, globals, data usage types, metrics |
| `madmin` | 5.5K | Admin API request/response types |
**I/O Pipeline:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `io-core` | 6.5K | Zero-copy I/O, buffer pool, direct I/O, scheduling, backpressure |
| `io-metrics` | 4.5K | I/O operation metrics and counters |
| `rio` | 6.9K | Composable reader chain (encrypt → compress → hash → limit) |
| `object-io` | 2.4K | High-level object read/write using rio + ecstore |
| `concurrency` | 1.8K | Concurrency control wrappers over io-core |
**Storage Engine:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `ecstore` | 87K | ⚠️ Erasure-coded storage: disks, pools, buckets, replication, lifecycle |
| `filemeta` | 10K | File/object metadata types and versioning |
| `checksums` | 732 | Checksum computation |
| `lock` | 7.1K | Distributed lock manager |
| `heal` | 5.9K | Data healing / bitrot repair |
| `scanner` | 5.4K | Background data usage scanner |
| `object-capacity` | 2.5K | Capacity tracking and management |
**Security & Auth:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `crypto` | 1.6K | Encryption primitives |
| `credentials` | 713 | Credential types (access key / secret key) |
| `signer` | 1.4K | S3 v4 request signing |
| `iam` | 9.0K | Identity and access management |
| `policy` | 8.8K | Policy engine (S3 bucket/IAM policies) |
| `kms` | 8.1K | Key management service integration |
| `keystone` | 1.9K | OpenStack Keystone auth |
| `appauth` | 143 | Application-level auth tokens |
**Protocol & API:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `protos` | 5.7K | Protobuf/gRPC definitions for inter-node RPC |
| `protocols` | 18K | FTP/FTPS, WebDAV, Swift API support |
| `s3-common` | 738 | Shared S3 types |
| `s3select-api` | 1.9K | S3 Select interface |
| `s3select-query` | 3.6K | S3 Select query engine |
**Observability:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `metrics` | 8.4K | Prometheus metric collectors |
| `io-metrics` | 4.5K | I/O-specific metrics |
| `obs` | 5.6K | OpenTelemetry tracing and telemetry |
| `audit` | 2.4K | Audit logging |
**Events:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `notify` | 5.5K | Event notification system |
| `targets` | 3.2K | Notification targets (Kafka, AMQP, webhook, etc.) |
**Other:**
| Crate | Lines | Purpose |
|-------|-------|---------|
| `trusted-proxies` | 4.0K | Trusted proxy / IP forwarding |
| `zip` | 986 | ZIP archive support for bulk downloads |
| `workers` | 136 | Simple worker abstraction |
## Architecture Invariants
> These are rules that the codebase should follow. Some are currently violated
> (marked with ⚠️). Documenting them here makes the violations explicit and
> trackable.
1. **Layers flow downward.** Server → Admin/App → Storage → ecstore → rio/io-core.
No upward imports.
2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`,
`io-metrics`, `madmin`, `s3-common` should depend only on external crates.
- ⚠️ VIOLATED: `utils` depends on `config`, `common` depends on `filemeta` and `madmin`.
3. **Each type has exactly one definition.** Types shared across crates must be defined
in one crate and re-exported or imported by others.
- ⚠️ VIOLATED: `ReplicationStats` (4 copies), `LastMinuteLatency` (3 copies),
`BackpressureConfig` (3 copies), `DataUsageInfo` (2 copies).
4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
storage-level abstractions (objects, buckets, disks, pools).
5. **The `rustfs` binary crate is the only place that wires everything together.**
Individual crates should be testable in isolation.
6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
not bare `Error`).
- ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
`heal` use `anyhow` in library code.
## Known Structural Issues
> This section documents known problems in the current architecture.
> It exists so the team can track and address them deliberately.
### Critical
- **common/scanner code duplication (~3K lines).** `scanner` depends on `common`
but maintains its own copies of `DataUsageInfo`, `LastMinuteLatency`, and related
types instead of importing them.
- **ecstore is a monolith (87K lines, 163 files).** It contains disk management,
bucket management, erasure coding, replication, lifecycle, RPC, and configuration
— all in one crate. It should be decomposed along its existing subdirectories.
### High
- **Dependency inversions.** `utils → config` and `common → filemeta/madmin` break
the layering model. These need to be untangled.
- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
concurrency, and rustfs/storage. Should be defined once with builder/composition.
### Medium
- **Inconsistent error handling.** Three strategies (thiserror/snafu/anyhow) and
mixed naming (bare `Error` vs descriptive names).
- **Ambiguous common vs utils boundary.** Both described as "utilities and data
structures." Need clear ownership rules.
## Cross-Cutting Concerns
### Error Handling
The project convention is `thiserror` for typed errors with descriptive names.
See `AGENTS.md`: "Prefer thiserror for library-facing error types."
```rust
// GOOD
#[derive(Debug, thiserror::Error)]
pub enum StorageError {
#[error("disk not found: {0}")]
DiskNotFound(String),
}
// AVOID
pub enum Error { ... } // too generic
anyhow::Result<T> // in library code (OK in tests/CLI)
```
### Logging & Tracing
- Use `tracing` crate (`info!`, `warn!`, `error!`, `debug!`, `trace!`)
- Structured fields: `tracing::info!(bucket = %name, "created bucket")`
- Spans for request-scoped context
### Metrics
- Prometheus-style metrics via `rustfs-obs` runtime and schema
- I/O-specific counters via `rustfs-io-metrics`
- Registration happens at crate level, collection/reporting in `rustfs-obs`
### Testing
- Unit tests: `#[cfg(test)] mod tests` in the same file
- Integration tests: inside respective crates (not top-level `tests/`)
- E2E tests: `crates/e2e_test/` — tests against a running server
- Run all: `make test` or `cargo nextest run`
## Startup Sequence
The binary (`main.rs`) boots in this order:
1. Environment variable compatibility (`MINIO_*``RUSTFS_*`)
2. Tokio runtime construction
3. CLI argument parsing
4. License, observability, TLS, trusted proxies initialization
5. Config parsing, server address resolution
6. Credentials, endpoints, local disks, lock client initialization
7. Capacity management initialization
8. HTTP server start (S3 API + optional console)
9. ECStore initialization (erasure coding storage engine)
10. Global config, background replication, KMS
11. Optional: FTP/FTPS/WebDAV servers
12. Event notifier, audit system, deadlock detector
13. Bucket metadata, IAM, Keystone, OIDC
14. Scanner and heal manager
15. Metrics system, mark `FullReady`
16. Wait for shutdown signal → graceful shutdown
## Dependency Diagram (Simplified)
```
┌─────────┐
│ rustfs │ (binary + lib, 75K lines)
│ main │
└────┬────┘
┌───────────────┼───────────────┐
│ │ │
┌────▼────┐ ┌────▼────┐ ┌──────▼─────┐
│ server │ │ admin │ │ app │
│ (HTTP) │ │(console)│ │(use-cases) │
└────┬────┘ └────┬────┘ └──────┬─────┘
│ │ │
└───────────────┼───────────────┘
┌──────▼──────┐
│ storage │
│ (ecfs, SSE, │
│ RPC, ACL) │
└──────┬──────┘
┌──────────────────┼──────────────────┐
│ │ │
┌─────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ ecstore │ │ rio │ │ io-core │
│ (87K,core) │ │ (readers) │ │ (zero-copy) │
└─────┬──────┘ └─────────────┘ └─────────────┘
┌─────┬──┼──┬─────┬──────┐
│ │ │ │ │ │
common utils config policy filemeta ...
```
## How to Navigate
- **"Where does S3 PutObject go?"**
`server/` routes → `app/object_usecase` validates → `storage/ecfs` encodes →
`ecstore` distributes → `rio` encrypts/compresses → `io-core` writes
- **"Where are bucket policies enforced?"**
`app/bucket_usecase` calls into `crates/policy/`
- **"Where is replication configured?"**
`admin/handlers/replication.rs` and `admin/handlers/site_replication.rs` for API,
`ecstore/src/bucket/replication/` for engine
- **"Where do I add a new admin endpoint?"**
Add handler in `admin/handlers/`, register in `admin/router.rs`
- **"Where do I add a new metric?"**
Define descriptor/collector in `crates/obs/src/metrics/`, expose via `/minio/v2/metrics`
---
*Inspired by [matklad's ARCHITECTURE.md](https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html)
and [rust-analyzer's architecture.md](https://github.com/rust-analyzer/rust-analyzer/blob/master/docs/book/src/contributing/architecture.md).*
-160
View File
@@ -1,160 +0,0 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Fixed
- **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
### Added
- **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
- Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate
- Task-local storage for async-safe credential passing between middleware and auth handlers
- Automatic detection of Keystone credentials (access keys prefixed with `keystone:`)
- Role-based permission mapping (admin/reseller_admin roles grant owner permissions)
- Token caching for high-performance validation with configurable cache size and TTL
- Dual authentication support: Keystone and standard AWS Signature v4 work simultaneously
- Immediate 401 response for invalid tokens (no fallback to local auth)
- XML-formatted error responses compatible with S3 API
- Comprehensive integration documentation with manual testing guide
- **32 unit and integration tests** covering middleware, auth handlers, task-local storage, and role detection
- **SFTPv3 Protocol Support**: SSH-hosted SFTPv3 subsystem that translates each file operation into S3 calls against the local object store. Authentication uses IAM credentials (SSH username = access key, SSH password = secret key).
- Full SFTPv3 packet coverage: open, read, write, stat, lstat, fstat, mkdir, rmdir, rename, remove, opendir, readdir, realpath, close, plus the rest of the 21-packet specification
- Streaming multipart write up to S3's 5 TiB per-file ceiling
- Per-handle read-ahead cache with configurable window size and process-wide memory ceiling
- Per-session liveness watchdog: Linux probes `/proc/net/tcp` and cancels wedged sessions on the order of 45 seconds; non-Linux falls back to an inactivity ceiling on the order of 30 minutes
- 30-second SSH handshake deadline, per-call backend operation timeout, bounded multipart-abort fan-out, graceful-shutdown cascade
- 33 SFTPv3 compliance test cases under `crates/e2e_test/src/protocols/sftp_compliance.rs` spread across three entry points: `test_sftp_compliance_suite` (shared session), `test_sftp_compliance_readonly` (read-only mode), and `test_sftp_compliance_standalone` (one rustfs spawn per case)
- Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary
### Changed
- **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
- **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
- **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
- **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.
- **`Protocol` enum**: new `Protocol::Sftp` variant with corresponding `S3Action` mappings. Every match arm on `Protocol` updated to handle the new variant exhaustively.
### Technical Details
- Middleware is self-contained in `rustfs-keystone` crate following the trusted-proxies pattern for integration-specific middleware
- Uses `BoxBody` pattern for Hyper 1.x compatibility
- Task-local storage provides request-scoped credential passing without modifying HTTP request/response types
- Integration preserves existing S3 authentication flow while adding Keystone support
- Zero breaking changes to existing functionality
- No new top-level directories in main binary crate (middleware lives in integration crate)
- SSH/SFTP wire handling via the `russh` and `russh-sftp` crates. SFTPv3 framing is implemented by `russh-sftp`; the rustfs-side `SftpDriver` implements `russh_sftp::server::Handler` and dispatches to the storage backend
- Drop-time abort for in-flight multipart uploads honours IAM Deny on `AbortMultipartUpload`. `start_multipart_upload` caches the authorisation decision so the synchronous `Drop` path can honour Allow / Deny policies without re-querying IAM
- Per-handle read cache uses an `Arc<AtomicU64>` shared across every `SftpDriver` instance to enforce a process-wide memory ceiling. On ceiling breach the populate is skipped and the read serves correctly via a single-call backend fetch
- Per-session liveness watchdog runs as a tokio task per accepted connection. Reads `/proc/net/tcp` and `/proc/net/tcp6` to look up the (local, peer) tuple's TCP state and cancels via `tokio_util::sync::CancellationToken` when wedge conditions are confirmed across two consecutive ticks
- Path canonicalisation rejects paths containing `\0`, `\r`, or `\n` and resolves traversal via `path::clean()` before any backend dispatch
- Cipher / KEX / MAC / host-key algorithm allowlists are hardcoded with no environment override. Strict-KEX (CVE-2023-48795 / Terrapin) marker presence asserted by unit test
- Per-session handle cap (default 64, configurable 8 to 1024) with UUID-generated handle ids
- Crate-level `#![deny(unsafe_code)]` is in force across `crates/protocols`. Socket fd duplication for the watchdog uses the safe `AsFd::try_clone_to_owned` path (Linux/Unix); non-Unix falls back to the inactivity ceiling
- `cfg(unix)` gating around platform-specific imports (`std::os::fd::AsFd`, `std::os::unix::fs::PermissionsExt`); non-Unix targets fail SFTP at config-load with `SftpInitError::UnsupportedPlatform`
### Documentation
- Updated `crates/keystone/README.md` with complete integration architecture and workflow
- Added detailed manual testing guide with 10 test scenarios
- Updated main `README.md` to list Keystone authentication as available feature
- Added troubleshooting section for common integration issues
- Module-level rustdoc on `crates/protocols/src/sftp/mod.rs` describing the public API surface, configuration contract, and the architecture of the read cache and the wedge watchdog
### Configuration
New environment variables:
- `RUSTFS_KEYSTONE_ENABLE` - Enable/disable Keystone authentication (default: false)
- `RUSTFS_KEYSTONE_AUTH_URL` - Keystone API endpoint URL
- `RUSTFS_KEYSTONE_VERSION` - Keystone API version (v3)
- `RUSTFS_KEYSTONE_ADMIN_USER` - Admin username for privileged operations
- `RUSTFS_KEYSTONE_ADMIN_PASSWORD` - Admin password
- `RUSTFS_KEYSTONE_ADMIN_PROJECT` - Admin project name
- `RUSTFS_KEYSTONE_ADMIN_DOMAIN` - Admin domain name (default: Default)
- `RUSTFS_KEYSTONE_CACHE_SIZE` - Token cache size (default: 10000)
- `RUSTFS_KEYSTONE_CACHE_TTL` - Token cache TTL in seconds (default: 300)
- `RUSTFS_KEYSTONE_VERIFY_SSL` - Verify SSL certificates (default: true)
- `RUSTFS_SFTP_ENABLE` - Enable/disable SFTP (default: false)
- `RUSTFS_SFTP_ADDRESS` - Listen address (default: 0.0.0.0:2222)
- `RUSTFS_SFTP_HOST_KEY_DIR` - Directory containing host key files (must exist; each file must be 0o600 or 0o400)
- `RUSTFS_SFTP_IDLE_TIMEOUT` - Session idle timeout in seconds (default: 600)
- `RUSTFS_SFTP_PART_SIZE` - Multipart part size in bytes (default: 16 MiB)
- `RUSTFS_SFTP_READ_ONLY` - Reject write packets at the protocol layer (default: false)
- `RUSTFS_SFTP_BANNER` - Optional SSH banner text
- `RUSTFS_SFTP_HANDLES_PER_SESSION` - Per-session open-handle cap, 8 to 1024 (default: 64)
- `RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS` - Per-call backend deadline in seconds, 5 to 600 (default: 60)
- `RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES` - Per-handle read-cache window in bytes, 256 KiB to 64 MiB or 0 to disable (default: 4 MiB)
- `RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES` - Process-wide read-cache memory ceiling in bytes, 16 MiB minimum (default: 256 MiB)
### Files Added
- `crates/protocols/src/sftp/mod.rs` - SFTP module entry point, public API surface, crate-level rustdoc, regression-prevention test
- `crates/protocols/src/sftp/config.rs` - `SftpConfig` and `SftpInitError` types, env-var resolvers, host-key directory loader with permission enforcement
- `crates/protocols/src/sftp/constants.rs` - Named constants grouped by purpose: S3 error codes, HTTP error codes, POSIX mode bits, protocol identifiers, operational limits
- `crates/protocols/src/sftp/server.rs` - `SftpServer` SSH server, russh handler, password authentication against IAM, accept loop, per-session task spawn
- `crates/protocols/src/sftp/driver.rs` - `SftpDriver` per-session SFTPv3 handler dispatching each operation onto the `StorageBackend`
- `crates/protocols/src/sftp/state.rs` - `HandleState` variants for read, write-buffering, write-streaming, write-failed handles
- `crates/protocols/src/sftp/lifecycle.rs` - Per-session activity stamp, weak-ref registry, `/proc/net/tcp` probe for the wedge watchdog
- `crates/protocols/src/sftp/wedge_watchdog.rs` - Per-session liveness watchdog cancelling sessions silent at the SFTP layer while the kernel reports CLOSE_WAIT
- `crates/protocols/src/sftp/read_cache.rs` - Per-handle in-memory read-ahead cache with shared atomic accumulator for the process-wide memory ceiling
- `crates/protocols/src/sftp/attrs.rs` - SFTPv3 `FileAttributes` mapping for objects and directories, longname formatting, mtime clamping
- `crates/protocols/src/sftp/dir.rs` - OPENDIR / READDIR pagination, root-bucket listing, sub-directory listing under a prefix
- `crates/protocols/src/sftp/errors.rs` - `SftpError` thiserror enum and S3-error classification into SFTPv3 status codes
- `crates/protocols/src/sftp/paths.rs` - Path canonicalisation, traversal rejection, `\0` / `\r` / `\n` rejection, bucket+key decomposition
- `crates/protocols/src/sftp/read.rs` - READ packet handler, EOF semantics, `MAX_READ_LEN` bound, integration with the read cache
- `crates/protocols/src/sftp/write.rs` - WRITE packet handler, in-memory buffering up to part size, transition to streaming multipart, CLOSE finalisation
- `crates/protocols/src/sftp/test_support.rs` - Test fixtures and helper builders for SFTP unit tests
- `crates/protocols/src/common/dummy_storage.rs` - In-memory `StorageBackend` test backend covering every method, used by SFTP unit tests and the FTPS / Swift / WebDAV test suites
- `crates/e2e_test/src/protocols/sftp_core.rs` - End-to-end regressions for the handshake deadline, idle-timeout disconnect, and the wedge watchdog
- `crates/e2e_test/src/protocols/sftp_compliance.rs` - SFTPv3 compliance suite entry points (`test_sftp_compliance_suite`, `test_sftp_compliance_readonly`, `test_sftp_compliance_standalone`)
- `crates/e2e_test/src/protocols/sftp_compliance_tests.rs` - Per-case test bodies (CMPTST-01..33), shared fixture helpers, lifecycle counters
- `crates/e2e_test/src/protocols/sftp_helpers.rs` - SFTP-specific test helpers and fixture seeders
### Files Modified
- `crates/keystone/src/middleware.rs` - Created Keystone authentication middleware (self-contained in keystone crate)
- `crates/keystone/src/lib.rs` - Exported middleware module and KEYSTONE_CREDENTIALS
- `crates/keystone/Cargo.toml` - Added Tower/HTTP dependencies for middleware functionality
- `rustfs/src/server/http.rs` - Integrated KeystoneAuthLayer from rustfs-keystone crate
- `rustfs/src/auth.rs` - Enhanced IAMAuth and check_key_valid for Keystone support, imported KEYSTONE_CREDENTIALS from rustfs-keystone
- `crates/keystone/README.md` - Comprehensive integration documentation
- `README.md` - Added Keystone as available feature
- `Cargo.toml` - Added the `sftp` feature alongside the existing protocol features
- `Cargo.lock` - Updated to include the new `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` dependencies and their transitive crates
- `crates/protocols/Cargo.toml` - Declared `russh`, `russh-sftp`, `socket2`, `tokio-util`, `subtle`, `uuid` under the `sftp` feature flag
- `crates/protocols/src/lib.rs` - Added `pub mod sftp` behind `#[cfg(feature = "sftp")]` plus the crate-level `#![deny(unsafe_code)]` lint
- `crates/protocols/src/common/client/s3.rs` - Extended the `StorageBackend` trait with `create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`, and `upload_part_copy`
- `crates/protocols/src/common/session.rs` - Added the `Protocol::Sftp` variant and its `S3Action` mappings
- `crates/protocols/src/common/gateway.rs` - Handles the new `Protocol::Sftp` variant exhaustively
- `crates/protocols/src/common/mod.rs` - Exposed the new `dummy_storage` module
- `crates/protocols/src/constants.rs` - Added shared POSIX mode-bit constants used by SFTP and other protocols
- `crates/config/src/constants/protocols.rs` - `RUSTFS_SFTP_*` environment variable names and defaults
- `crates/utils/src/retry.rs` - Added the generic exponential-backoff retry helper used by the SFTP write path
- `crates/e2e_test/Cargo.toml` - Added the e2e test dependencies for SFTP (paramiko fixture, SSH keypair generation)
- `crates/e2e_test/src/protocols/mod.rs` - Registered the new `sftp_core`, `sftp_compliance`, `sftp_compliance_tests`, and `sftp_helpers` modules
- `crates/e2e_test/src/protocols/README.md` - Documented the SFTP test entry points and case index
- `crates/e2e_test/src/protocols/test_env.rs` - Added SFTP host-key directory provisioning to the shared protocol test environment
- `crates/e2e_test/src/protocols/test_runner.rs` - Wired the SFTP entry points into the runner
- `rustfs/Cargo.toml` - Added the `sftp` feature flag
- `rustfs/src/lib.rs` - One-line addition exporting the SFTP wiring
- `rustfs/src/init.rs` - Build and start the `SftpServer` when `RUSTFS_SFTP_ENABLE` is true
- `rustfs/src/main.rs` - Routed shutdown signals to the SFTP server alongside the other protocols
- `rustfs/src/protocols/client.rs` - Client-builder support for the new `Protocol::Sftp` variant
### Testing
- 16 unit tests in rustfs-keystone crate (config, auth, middleware, identity)
- 10 integration tests in rustfs-keystone crate (task-local storage, middleware layer, scope isolation)
- 6 auth unit tests in rustfs crate (role detection, task-local storage, Keystone credential handling)
- **Total: 32 tests** passing with zero compilation errors
- Manual testing guide provided for end-to-end validation
- All Keystone tests passing with `cargo test --all --exclude e2e_test`
- 33 SFTPv3 compliance test cases (CMPTST-01..33) split across three entry points: `test_sftp_compliance_suite` (shared session, cases 01-14), `test_sftp_compliance_readonly` (read-only mode, cases 15-23), `test_sftp_compliance_standalone` (one rustfs spawn per case, cases 24-33)
- Regression-prevention tests at four layers: compile-time module assertion in `crates/protocols/src/lib.rs`, module-presence unit test in `crates/protocols/src/sftp/mod.rs`, cross-module `Protocol` enum assertion, and end-to-end SSH banner test against the running binary
- Standalone end-to-end regressions for the SSH handshake deadline, the idle-timeout disconnect path, and the wedge watchdog (Linux fast-kill and the cross-platform fallback path)
- Inline unit tests in every SFTP source file covering pure helpers (path canonicalisation, attribute mapping, S3-error classification, env-var bound resolvers)
- Strict-KEX (CVE-2023-48795) marker presence assertion as a unit test in `crates/protocols/src/sftp/server.rs`
- All tests passing with `cargo test --all --features sftp` against a 64-bit Linux target
---
## Previous Releases
See [GitHub Releases](https://github.com/rustfs/rustfs/releases) for previous version history.
-60
View File
@@ -1,60 +0,0 @@
# RustFS Individual Contributor License Agreement
Thank you for your interest in contributing to RustFS. In order to clarify the intellectual property license granted with Contributions from any person or entity, RustFS, Inc. ("RustFS") must have a Contributor License Agreement ("CLA") on file that has been signed by each Contributor, indicating agreement to the license terms below. This license is for your protection as a Contributor as well as the protection of RustFS and its users; it does not change your rights to use your own Contributions for any other purpose.
You accept and agree to the following terms and conditions for Your present and future Contributions submitted to RustFS. Except for the license granted herein to RustFS and recipients of software distributed by RustFS, You reserve all right, title, and interest in and to Your Contributions.
**You understand and agree that You will not receive any royalty or other compensation for Your Contributions.**
## 1. Definitions
* **"You"** (or **"Your"**) shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with RustFS. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
* **"Contribution"** shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to RustFS for inclusion in, or documentation of, any of the products or projects owned or managed by RustFS (the "Work"). For the purposes of this definition, "submitted" means any form of electronic or written communication sent to RustFS or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, RustFS for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
## 2. Grant of Copyright License
Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works under any license, including proprietary or commercial licenses and open-source licenses, at RustFS's sole discretion.
## 3. Grant of Patent License
Subject to the terms and conditions of this Agreement, You hereby grant to RustFS and to recipients of software distributed by RustFS a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
## 4. Representations
You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to RustFS, or that your employer has executed a separate Corporate CLA with RustFS.
You represent that each of Your Contributions is Your original creation. You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions.
## 5. Support and Warranty
You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of **TITLE**, **NON-INFRINGEMENT**, **MERCHANTABILITY**, or **FITNESS FOR A PARTICULAR PURPOSE**.
## 6. Third-Party Work
Should You wish to submit work that is not Your original creation, You may submit it to RustFS separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".
## 7. Governing Law and Jurisdiction
This Agreement shall be governed by and construed in accordance with the laws of the State of Delaware, United States of America, without regard to its conflict of laws principles. The parties expressly agree that the United Nations Convention on Contracts for the International Sale of Goods will not apply. Any legal action or proceeding arising under or in connection with this Agreement shall be brought exclusively in the state or federal courts located in the State of Delaware, United States of America, and the parties hereby irrevocably consent to the personal jurisdiction and venue therein.
## 8. Severability
If any provision of this Agreement is found to be invalid or unenforceable, the remaining provisions will continue in full force and effect.
## 9. Entire Agreement and Assignment
This Agreement constitutes the entire agreement between the parties with respect to the subject matter hereof and supersedes all prior agreements, understandings, negotiations, and discussions, whether written or oral. RustFS may assign its rights and obligations under this Agreement to any third party. You may not assign Your rights or obligations under this Agreement without the prior written consent of RustFS.
---
**Please read the terms of this Agreement carefully. By submitting a Contribution to RustFS, You agree to be bound by the terms and conditions of this Agreement.**
| | |
|---|---|
| **Full Name** | __________________________________ |
| **GitHub Username** | __________________________________ |
| **Email Address** | __________________________________ |
| **Date** | __________________________________ |
*(Electronic signature or acknowledgement via GitHub commit/Pull Request constitutes valid acceptance of this Agreement).*
-128
View File
@@ -1,128 +0,0 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
hello@rustfs.com.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.

Some files were not shown because too many files have changed in this diff Show More