Compare commits

..

17 Commits

Author SHA1 Message Date
安正超 f095f56e20 fix(ci): revert docker alpha latest tag (#2209) 2026-03-18 21:05:52 +08:00
weisd b9b7d86ae4 feat: improve legacy metadata and admin compatibility (#2202) 2026-03-18 21:05:09 +08:00
马登山 84077adf17 fix(admin): avoid unbounded metrics sampling by default (#2203) 2026-03-18 14:43:17 +08:00
马登山 237c933f38 test(lifecycle): add prefix regression and zero-day checks (#2201) 2026-03-18 13:20:00 +08:00
安正超 c20b3c7f19 fix(ecstore): handle EODM rules without due date (#2198) 2026-03-18 12:54:41 +08:00
安正超 c9a2fd756c feat(rustfs): add optional license gating feature (#2197) 2026-03-18 08:45:05 +08:00
dependabot[bot] bddb0d0a05 build(deps): bump astral-tokio-tar from 0.5.6 to 0.6.0 (#2196)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-18 08:22:12 +08:00
houseme 9ce3c7742c fix(targets): pass credentials to MQTT broker availability check (#2192)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-17 23:53:28 +08:00
houseme b5d881f399 fix(notify): Fix XML Filter parsing and add comprehensive tests (#2191) 2026-03-17 23:08:07 +08:00
安正超 ce1f7cfdcb chore(skills): add repository-local workflow skills (#2190) 2026-03-17 22:13:46 +08:00
马登山 c66c6d97ec fix(lifecycle): respect Filter.Prefix and safe delete marker expiry (#2185)
Signed-off-by: likewu <likewu@126.com>
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: likewu <likewu@126.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-17 18:45:38 +08:00
dependabot[bot] be89b5fc6a build(deps): bump lz4_flex from 0.12.0 to 0.12.1 (#2181)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-17 10:10:41 +08:00
houseme 94cdb89e29 feat(obs): add init_obs_with_config API and signature guard test (#2175)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
2026-03-16 18:17:55 +08:00
heihutu 06dff96c09 chore(deps): update flake.lock (#2173) 2026-03-16 16:01:36 +08:00
安正超 c1d5106acc feat(ci): allow selecting build platforms in build workflow (#2171) 2026-03-15 22:01:44 +08:00
heihutu 0a2411f59c chore(deps): update flake.lock (#2169)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-03-15 16:10:12 +08:00
houseme 1ede71b881 chore: update nix-flake-update.yml to use FLAKE_UPDATE_TOKEN for user… (#2168) 2026-03-15 14:49:38 +08:00
173 changed files with 16871 additions and 2462 deletions
@@ -0,0 +1,78 @@
---
name: code-change-verification
description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
---
# Code Change Verification
Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
## Quick Start
1. Read the scope: commit, PR, patch, or file list.
2. Map each changed area by risk and user impact.
3. Inspect each risky change in context.
4. Report findings first, ordered by severity.
5. Close with residual risks and verification recommendations.
## Core Workflow
### 1) Scope and assumptions
- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
- If context is missing, state assumptions before deeper analysis.
- Focus only on requested scope; avoid reviewing unrelated files.
### 2) Risk map
- Prioritize in this order:
- Data correctness and user-visible behavior
- API/contract compatibility
- Security and authz/authn boundaries
- Concurrency and lifecycle correctness
- Performance and resource usage
- Give higher priority to stateful paths, migration logic, defaults, and error handling.
### 3) Evidence-based inspection
- Read each modified hunk with neighboring context.
- Trace call paths and call-site expectations.
- Check for:
- invariant breaks and missing guards
- unchecked assumptions and null/empty/error-path handling
- stale tests, fixtures, and configs
- hidden coupling to shared helpers/constants/features
- If a point is uncertain, mark it as an open question instead of guessing.
### 4) Findings-first output
- Order findings by severity:
- P0: critical failure, security breach, or data loss risk
- P1: high-impact regression
- P2: medium risk correctness gap
- P3: low risk/quality debt
- For each finding include:
- Severity
- `path:line` reference
- concise issue statement
- impact and likely failure mode
- specific fix or mitigation
- validation step to confirm
- If no issues exist, explicitly state `No findings` and why.
### 5) Close
- Report assumptions and unknowns.
- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
## Output Template
1. Findings
2. No findings (if applicable)
3. Assumptions / Unknowns
4. Recommended verification steps
## Finding Template
- `[P1] Missing timeout for downstream call`
- Location: `path/to/file.rs:123`
- Issue: ...
- Impact: ...
- Fix suggestion: ...
- Validation: ...
@@ -0,0 +1,4 @@
interface:
display_name: "Code Change Verification"
short_description: "Prioritize risks and verify code changes before merge."
default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."
@@ -0,0 +1,88 @@
---
name: pr-creation-checker
description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
---
# PR Creation Checker
Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
## Read sources of truth first
- Read `AGENTS.md`.
- Read `.github/pull_request_template.md`.
- Use `Makefile` and `.config/make/` for local quality commands.
- Use `.github/workflows/ci.yml` for CI expectations.
- Do not restate long command matrices or template sections from memory when the files exist.
## Workflow
1. Collect PR context
- Confirm base branch, current branch, change goal, and scope.
- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
- Confirm whether the branch includes only intended changes.
2. Inspect change scope
- Review the diff and summarize what changed.
- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
3. Verify readiness requirements
- Require `make pre-commit` before marking the PR ready.
- If `make` is unavailable, use the equivalent commands from `.config/make/`.
- Add scope-specific verification commands when the changed area needs more than the baseline.
- If required checks fail, stop and return `BLOCKED`.
4. Draft PR metadata
- Write the PR title in English using Conventional Commits and keep it within 72 characters.
- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
- Keep the PR body in English.
- Use the exact section headings from `.github/pull_request_template.md`.
- Fill non-applicable sections with `N/A`.
- Include verification commands in the PR description.
- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
5. Prepare reviewer context
- Summarize why the change exists.
- Summarize what was verified.
- Call out risks, rollout notes, config impact, and rollback notes when applicable.
- Mention assumptions or missing context instead of guessing.
6. Prepare CLI-safe output
- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
- Return a ready-to-save PR body plus a short title.
- If not ready, return blockers first and list the minimum steps needed to unblock.
## Output format
### Status
- `READY` or `BLOCKED`
### Title
- `<type>(<scope>): <summary>`
### PR Body
- Reproduce the repository template headings exactly.
- Fill every section.
- Omit local absolute paths unless explicitly required.
### Verification
- List each command run.
- State pass/fail.
### Risks
- List breaking changes, config changes, migration impact, or `N/A`.
## Blocker rules
- Return `BLOCKED` if `make pre-commit` has not passed.
- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
- Return `BLOCKED` if required template sections are missing.
- Return `BLOCKED` if the title/body is not in English.
- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
## Reference
- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.
@@ -0,0 +1,4 @@
interface:
display_name: "PR Creation Checker"
short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."
@@ -0,0 +1,14 @@
# PR Readiness Checklist
- Confirm the branch is based on current `main`.
- Confirm the diff matches the stated scope.
- Confirm no secrets, logs, temp files, or unrelated refactors are included.
- Confirm `make pre-commit` passed, or document why it could not run.
- Confirm extra verification commands are listed for risky changes.
- Confirm the PR title uses Conventional Commits and stays within 72 characters.
- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
- Confirm the PR body is in English.
- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
- Confirm non-applicable sections are filled with `N/A`.
- Confirm the PR body does not include local absolute paths unless explicitly required.
- Confirm multiline GitHub CLI commands use `--body-file`.
@@ -0,0 +1,66 @@
---
name: test-coverage-improver
description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
---
# Test Coverage Improver
Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
## Usage assumptions
- Focus scope is either changed lines/files, a module, or the whole repository.
- Coverage artifact must be generated or provided in a supported format.
- If required context is missing, call out assumptions explicitly before proposing work.
## Workflow
1. Define scope and baseline
- Confirm target language, framework, and branch.
- Confirm whether the scope is changed files only or full-repo.
2. Produce coverage snapshot
- Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
- JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
- Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
- Collect total, per-file, and changed-line coverage.
3. Rank highest-risk gaps
- Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
- Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
- Keep shortlist to 58 gaps.
- For each gap, capture: file, lines, uncovered branches, and estimated risk score.
4. Propose high-impact tests
- For each shortlisted gap, output:
- Intent and expected behavior.
- Normal, edge, and failure scenarios.
- Assertions and side effects to verify.
- Setup needs (fixtures, mocks, integration dependencies).
- Estimated effort (`S/M/L`).
5. Close with validation plan
- State which gaps remain after proposals.
- Provide concrete verification command and acceptance threshold.
- List assumptions or blockers (environment, fixtures, flaky dependencies).
## Output template
### Coverage Snapshot
- total / branch coverage
- changed-file coverage
- top missing regions by size
### Top Gaps (ranked)
- `path:line-range` | risk score | why critical
### Test Proposals
- `path:line-range`
- Test name
- scenarios
- assertions
- effort
### Validation Plan
- command
- pass criteria
- remaining risk
@@ -0,0 +1,4 @@
interface:
display_name: "Test Coverage Improver"
short_description: "Find top uncovered risk areas and propose high-impact tests."
default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."
@@ -0,0 +1,25 @@
# Coverage Gap Prioritization Guide
Use this rubric for each uncovered area.
Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
- Criticality:
- 5: authz/authn, data-loss, payment/consistency path
- 4: state mutation, cache invalidation, scheduling
- 3: error handling + fallbacks in user-visible flows
- 2: parsing/format conversion paths
- 1: logging-only or low-impact utilities
- CoverageDebt:
- 0: 05 uncovered lines
- 1: 620 uncovered lines
- 2: 2140 uncovered lines
- 3: 41+ uncovered lines
- Volatility:
- 1: stable legacy code with few recent edits
- 2: changed in last 2 releases
- 3: touched in last 30 days or currently in active PR
Sort by score descending, then by business impact.
File diff suppressed because it is too large Load Diff
+62 -49
View File
@@ -23,6 +23,7 @@
#
# Manual Parameters:
# - build_docker: Build and push Docker images (default: true)
# - platforms: Comma-separated platform IDs or 'all' (default: all)
name: Build and Release
@@ -53,6 +54,11 @@ on:
required: false
default: true
type: boolean
platforms:
description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
required: false
default: "all"
type: string
permissions:
contents: read
@@ -138,63 +144,70 @@ jobs:
echo " - Is prerelease: $is_prerelease"
# Build RustFS binaries
prepare-platform-matrix:
name: Prepare Platform Matrix
runs-on: ubicloud-standard-2
outputs:
matrix: ${{ steps.select.outputs.matrix }}
selected: ${{ steps.select.outputs.selected }}
steps:
- name: Select target platforms
id: select
shell: bash
run: |
set -euo pipefail
selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
fi
all='{"include":[
{"target_id":"linux-x86_64-musl","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
{"target_id":"linux-aarch64-musl","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
{"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
{"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
{"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
]}'
if [[ "${selected}" == "all" ]]; then
matrix="$(jq -c . <<<"${all}")"
else
unknown="$(jq -rn --arg selected "${selected}" --argjson all "${all}" '
($selected | split(",") | map(select(length > 0))) as $req
| ($all.include | map(.target_id)) as $known
| [$req[] | select(( $known | index(.) ) == null)]
')"
if [[ "$(jq 'length' <<<"${unknown}")" -gt 0 ]]; then
echo "Unknown platforms: $(jq -r 'join(\",\")' <<<"${unknown}")" >&2
echo "Allowed: $(jq -r '.include[].target_id' <<<"${all}" | paste -sd ',' -)" >&2
exit 1
fi
matrix="$(jq -c --arg selected "${selected}" '
($selected | split(",") | map(select(length > 0))) as $req
| .include |= map(select(.target_id as $id | ($req | index($id))))
' <<<"${all}")"
fi
echo "selected=${selected}" >> "$GITHUB_OUTPUT"
echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
echo "Selected platforms: ${selected}"
build-rustfs:
name: Build RustFS
needs: [ build-check ]
if: needs.build-check.outputs.should_build == 'true'
needs: [ build-check, prepare-platform-matrix ]
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
runs-on: ${{ matrix.os }}
timeout-minutes: 60
env:
RUSTFLAGS: ${{ matrix.rustflags }}
strategy:
fail-fast: false
matrix:
include:
# Linux builds
# Use x86-64-v2 (SSE4.2 baseline) instead of native to ensure distributed
# binaries run on older x86_64 CPUs (e.g. Intel Celeron/Atom, Synology NAS).
# See: https://github.com/rustfs/rustfs/issues/1838
- os: ubicloud-standard-2
target: x86_64-unknown-linux-musl
cross: false
platform: linux
rustflags: ''
- os: ubicloud-standard-2
target: aarch64-unknown-linux-musl
cross: true
platform: linux
rustflags: ''
- os: ubicloud-standard-2
target: x86_64-unknown-linux-gnu
cross: false
platform: linux
rustflags: ''
- os: ubicloud-standard-2
target: aarch64-unknown-linux-gnu
cross: true
platform: linux
rustflags: ''
# macOS builds
- os: macos-latest
target: aarch64-apple-darwin
cross: false
platform: macos
rustflags: ''
- os: macos-latest
target: x86_64-apple-darwin
cross: false
platform: macos
rustflags: ''
# Windows builds (temporarily disabled)
- os: windows-latest
target: x86_64-pc-windows-msvc
cross: false
platform: windows
rustflags: ''
#- os: windows-latest
# target: aarch64-pc-windows-msvc
# cross: true
# platform: windows
matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
steps:
- name: Checkout repository
uses: actions/checkout@v6
+20 -4
View File
@@ -163,7 +163,14 @@ jobs:
if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
echo "🧪 Building Docker image for prerelease: $version"
# TODO: Temporary change - currently allows alpha versions to also create latest tags
# After the version is stable, you need to remove the following line and restore the original logic (latest is created only for stable versions)
if [[ "$version" == *"alpha"* ]]; then
create_latest=true
echo "🧪 Building Docker image for prerelease: $version (temporarily allowing creation of latest tag)"
else
echo "🧪 Building Docker image for prerelease: $version"
fi
else
build_type="release"
create_latest=true
@@ -209,7 +216,14 @@ jobs:
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
echo "🧪 Building with prerelease version: $input_version"
# TODO: Temporary change - currently allows alpha versions to also create latest tags
# After the version is stable, you need to remove the if block below and restore the original logic.
if [[ "$input_version" == *"alpha"* ]]; then
create_latest=true
echo "🧪 Building with prerelease version: $input_version (temporarily allowing creation of latest tag)"
else
echo "🧪 Building with prerelease version: $input_version"
fi
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
@@ -436,8 +450,10 @@ jobs:
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been explicitly created for prerelease"
# TODO: Temporary change - alpha versions currently create the latest tag
# After the version is stable, you need to restore the following prompt information
if [[ "$VERSION" == *"alpha"* ]] && [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for alpha version (temporary measures)"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
+7 -3
View File
@@ -35,7 +35,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@v3
@@ -46,14 +46,18 @@ jobs:
id: update
uses: DeterminateSystems/update-flake-lock@main
with:
git-author-name: heihutu
git-author-email: heihutu@gmail.com
git-committer-name: heihutu
git-committer-email: heihutu@gmail.com
pr-title: "chore(deps): update flake.lock"
pr-labels: |
dependencies
nix
automated
commit-msg: "chore(deps): update flake.lock"
pr-reviewers: houseme, heihutu
token: ${{ secrets.GITHUB_TOKEN }}
pr-reviewers: houseme, overtrue, majinghe
token: ${{ secrets.FLAKE_UPDATE_TOKEN }}
- name: Log PR details
if: steps.update.outputs.pull-request-number
+3
View File
@@ -16,6 +16,7 @@ vendor
cli/rustfs-gui/embedded-rustfs/rustfs
*.log
deploy/certs/*
deploy/data/*
*jsonl
.env
.rustfs.sys
@@ -46,3 +47,5 @@ docs
result*
*.gz
rustfs-webdav.code-workspace
.aiexclude
Generated
+152 -43
View File
@@ -69,6 +69,17 @@ dependencies = [
"subtle",
]
[[package]]
name = "ahash"
version = "0.7.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "891477e0c6a8957309ee5c45a6368af3ae14bb510732d2684ffa19af310920f9"
dependencies = [
"getrandom 0.2.17",
"once_cell",
"version_check",
]
[[package]]
name = "ahash"
version = "0.8.12"
@@ -285,7 +296,7 @@ version = "57.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4c8955af33b25f3b175ee10af580577280b4bd01f7e823d94c7cdef7cf8c9aef"
dependencies = [
"ahash",
"ahash 0.8.12",
"arrow-buffer",
"arrow-data",
"arrow-schema",
@@ -442,7 +453,7 @@ version = "57.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68bf3e3efbd1278f770d67e5dc410257300b161b93baedb3aae836144edcaf4b"
dependencies = [
"ahash",
"ahash 0.8.12",
"arrow-array",
"arrow-buffer",
"arrow-data",
@@ -508,9 +519,9 @@ dependencies = [
[[package]]
name = "astral-tokio-tar"
version = "0.5.6"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec179a06c1769b1e42e1e2cbe74c7dcdb3d6383c838454d063eaac5bbb7ebbe5"
checksum = "3c23f3af104b40a3430ccb90ed5f7bd877a8dc5c26fc92fde51a22b40890dcf9"
dependencies = [
"filetime",
"futures-core",
@@ -731,7 +742,7 @@ dependencies = [
"http 0.2.12",
"http 1.4.0",
"http-body 1.0.1",
"lru",
"lru 0.16.3",
"percent-encoding",
"regex-lite",
"sha2 0.10.9",
@@ -2192,7 +2203,7 @@ dependencies = [
"hashbrown 0.14.5",
"lock_api",
"once_cell",
"parking_lot_core",
"parking_lot_core 0.9.12",
]
[[package]]
@@ -2244,7 +2255,7 @@ dependencies = [
"liblzma",
"log",
"object_store",
"parking_lot",
"parking_lot 0.12.5",
"parquet",
"rand 0.9.2",
"regex",
@@ -2277,7 +2288,7 @@ dependencies = [
"itertools 0.14.0",
"log",
"object_store",
"parking_lot",
"parking_lot 0.12.5",
"tokio",
]
@@ -2310,7 +2321,7 @@ version = "52.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ea2df29b9592a5d55b8238eaf67d2f21963d5a08cd1a8b7670134405206caabd"
dependencies = [
"ahash",
"ahash 0.8.12",
"arrow",
"arrow-ipc",
"chrono",
@@ -2468,7 +2479,7 @@ dependencies = [
"itertools 0.14.0",
"log",
"object_store",
"parking_lot",
"parking_lot 0.12.5",
"parquet",
"tokio",
]
@@ -2494,7 +2505,7 @@ dependencies = [
"futures",
"log",
"object_store",
"parking_lot",
"parking_lot 0.12.5",
"rand 0.9.2",
"tempfile",
"url",
@@ -2573,7 +2584,7 @@ version = "52.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "325a00081898945d48d6194d9ca26120e523c993be3bb7c084061a5a2a72e787"
dependencies = [
"ahash",
"ahash 0.8.12",
"arrow",
"datafusion-common",
"datafusion-doc",
@@ -2594,7 +2605,7 @@ version = "52.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "809bbcb1e0dbec5d0ce30d493d135aea7564f1ba4550395f7f94321223df2dae"
dependencies = [
"ahash",
"ahash 0.8.12",
"arrow",
"datafusion-common",
"datafusion-expr-common",
@@ -2636,7 +2647,7 @@ dependencies = [
"datafusion-common",
"datafusion-expr",
"datafusion-physical-plan",
"parking_lot",
"parking_lot 0.12.5",
"paste",
]
@@ -2705,7 +2716,7 @@ version = "52.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d3a86264bb9163e7360b6622e789bc7fcbb43672e78a8493f0bc369a41a57c6"
dependencies = [
"ahash",
"ahash 0.8.12",
"arrow",
"datafusion-common",
"datafusion-expr",
@@ -2716,7 +2727,7 @@ dependencies = [
"hashbrown 0.16.1",
"indexmap 2.13.0",
"itertools 0.14.0",
"parking_lot",
"parking_lot 0.12.5",
"paste",
"petgraph",
"recursive",
@@ -2744,7 +2755,7 @@ version = "52.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ae769ea5d688b4e74e9be5cad6f9d9f295b540825355868a3ab942380dd97ce"
dependencies = [
"ahash",
"ahash 0.8.12",
"arrow",
"chrono",
"datafusion-common",
@@ -2752,7 +2763,7 @@ dependencies = [
"hashbrown 0.16.1",
"indexmap 2.13.0",
"itertools 0.14.0",
"parking_lot",
"parking_lot 0.12.5",
]
[[package]]
@@ -2780,7 +2791,7 @@ version = "52.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "79949cbb109c2a45c527bfe0d956b9f2916807c05d4d2e66f3fd0af827ac2b61"
dependencies = [
"ahash",
"ahash 0.8.12",
"arrow",
"arrow-ord",
"arrow-schema",
@@ -2800,7 +2811,7 @@ dependencies = [
"indexmap 2.13.0",
"itertools 0.14.0",
"log",
"parking_lot",
"parking_lot 0.12.5",
"pin-project-lite",
"tokio",
]
@@ -2833,7 +2844,7 @@ dependencies = [
"datafusion-execution",
"datafusion-expr",
"datafusion-physical-plan",
"parking_lot",
"parking_lot 0.12.5",
]
[[package]]
@@ -2873,9 +2884,9 @@ dependencies = [
"http-body-util",
"libc",
"log",
"lru",
"lru 0.16.3",
"mime_guess",
"parking_lot",
"parking_lot 0.12.5",
"percent-encoding",
"pin-project-lite",
"reflink-copy",
@@ -4032,6 +4043,9 @@ name = "hashbrown"
version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888"
dependencies = [
"ahash 0.7.8",
]
[[package]]
name = "hashbrown"
@@ -4501,7 +4515,7 @@ version = "0.11.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "232929e1d75fe899576a3d5c7416ad0d88dbfbb3c3d6aa00873a7408a50ddb88"
dependencies = [
"ahash",
"ahash 0.8.12",
"indexmap 2.13.0",
"is-terminal",
"itoa",
@@ -4519,7 +4533,7 @@ version = "0.12.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "90807d610575744524d9bdc69f3885d96f0e6c3354565b0828354a7ff2a262b8"
dependencies = [
"ahash",
"ahash 0.8.12",
"clap",
"crossbeam-channel",
"crossbeam-utils",
@@ -4553,6 +4567,15 @@ dependencies = [
"hybrid-array",
]
[[package]]
name = "instant"
version = "0.1.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222"
dependencies = [
"cfg-if",
]
[[package]]
name = "integer-encoding"
version = "3.0.4"
@@ -5040,6 +5063,15 @@ version = "0.4.29"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897"
[[package]]
name = "lru"
version = "0.7.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e999beba7b6e8345721bd280141ed958096a2e4abdf74f67ff4ce49b4b54e47a"
dependencies = [
"hashbrown 0.12.3",
]
[[package]]
name = "lru"
version = "0.16.3"
@@ -5076,9 +5108,9 @@ dependencies = [
[[package]]
name = "lz4_flex"
version = "0.12.0"
version = "0.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab6473172471198271ff72e9379150e9dfd70d8e533e0752a27e515b48dd375e"
checksum = "98c23545df7ecf1b16c303910a69b079e8e251d60f7dd2cc9b4177f2afaf1746"
dependencies = [
"twox-hash",
]
@@ -5202,7 +5234,7 @@ version = "0.24.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d5312e9ba3771cfa961b585728215e3d972c950a3eed9252aa093d6301277e8"
dependencies = [
"ahash",
"ahash 0.8.12",
"portable-atomic",
]
@@ -5271,7 +5303,7 @@ dependencies = [
"equivalent",
"event-listener",
"futures-util",
"parking_lot",
"parking_lot 0.12.5",
"portable-atomic",
"smallvec",
"tagptr",
@@ -5306,7 +5338,7 @@ dependencies = [
"libc",
"log",
"neli-proc-macros",
"parking_lot",
"parking_lot 0.12.5",
]
[[package]]
@@ -5629,7 +5661,7 @@ dependencies = [
"http 1.4.0",
"humantime",
"itertools 0.14.0",
"parking_lot",
"parking_lot 0.12.5",
"percent-encoding",
"thiserror 2.0.18",
"tokio",
@@ -5882,6 +5914,17 @@ version = "2.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba"
[[package]]
name = "parking_lot"
version = "0.11.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d17b78036a60663b797adeaee46f5c9dfebb86948d1255007a1d6be0271ff99"
dependencies = [
"instant",
"lock_api",
"parking_lot_core 0.8.6",
]
[[package]]
name = "parking_lot"
version = "0.12.5"
@@ -5889,7 +5932,21 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a"
dependencies = [
"lock_api",
"parking_lot_core",
"parking_lot_core 0.9.12",
]
[[package]]
name = "parking_lot_core"
version = "0.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60a2cfe6f0ad2bfc16aefa463b497d5c7a5ecd44a23efa72aa342d90177356dc"
dependencies = [
"cfg-if",
"instant",
"libc",
"redox_syscall 0.2.16",
"smallvec",
"winapi",
]
[[package]]
@@ -5911,7 +5968,7 @@ version = "57.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ee96b29972a257b855ff2341b37e61af5f12d6af1158b6dcdb5b31ea07bb3cb"
dependencies = [
"ahash",
"ahash 0.8.12",
"arrow-array",
"arrow-buffer",
"arrow-cast",
@@ -6403,7 +6460,7 @@ dependencies = [
"fnv",
"lazy_static",
"memchr",
"parking_lot",
"parking_lot 0.12.5",
"thiserror 2.0.18",
]
@@ -6596,6 +6653,7 @@ version = "0.39.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "958f21e8e7ceb5a1aa7fa87fab28e7c75976e0bfe7e23ff069e0a260f894067d"
dependencies = [
"encoding_rs",
"memchr",
"serde",
"tokio",
@@ -6762,7 +6820,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "36ea961700fd7260e7fa3701c8287d901b2172c51f9c1421fa0f21d7f7e184b7"
dependencies = [
"clocksource",
"parking_lot",
"parking_lot 0.12.5",
"thiserror 1.0.69",
]
@@ -6826,6 +6884,15 @@ dependencies = [
"syn 2.0.117",
]
[[package]]
name = "redox_syscall"
version = "0.2.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb5a58c1855b4b6819d59012155603f0b22ad30cad752600aadfcb695265519a"
dependencies = [
"bitflags 1.3.2",
]
[[package]]
name = "redox_syscall"
version = "0.5.18"
@@ -6855,6 +6922,21 @@ dependencies = [
"thiserror 2.0.18",
]
[[package]]
name = "reed-solomon-erasure"
version = "6.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7263373d500d4d4f505d43a2a662d475a894aa94503a1ee28e9188b5f3960d4f"
dependencies = [
"cc",
"libc",
"libm",
"lru 0.7.8",
"parking_lot 0.11.2",
"smallvec",
"spin 0.9.8",
]
[[package]]
name = "reed-solomon-simd"
version = "3.1.0"
@@ -7291,6 +7373,7 @@ dependencies = [
"rustfs-common",
"rustfs-config",
"rustfs-credentials",
"rustfs-crypto",
"rustfs-ecstore",
"rustfs-filemeta",
"rustfs-heal",
@@ -7325,6 +7408,7 @@ dependencies = [
"starshard",
"subtle",
"sysinfo",
"temp-env",
"tempfile",
"thiserror 2.0.18",
"tikv-jemalloc-ctl",
@@ -7487,12 +7571,13 @@ dependencies = [
"md-5 0.11.0-rc.5",
"moka",
"num_cpus",
"parking_lot",
"parking_lot 0.12.5",
"path-absolutize",
"pin-project-lite",
"quick-xml 0.39.2",
"rand 0.10.0",
"ratelimit",
"reed-solomon-erasure",
"reed-solomon-simd",
"regex",
"reqwest 0.13.2",
@@ -7515,8 +7600,10 @@ dependencies = [
"rustls",
"s3s",
"serde",
"serde_bytes",
"serde_json",
"serde_urlencoded",
"serial_test",
"sha1 0.11.0-rc.5",
"sha2 0.11.0-rc.5",
"shadow-rs",
@@ -7631,8 +7718,10 @@ dependencies = [
"reqwest 0.13.2",
"rustfs-credentials",
"rustfs-policy",
"rustfs-utils",
"serde",
"serde_json",
"temp-env",
"thiserror 2.0.18",
"time",
"tokio",
@@ -7654,9 +7743,11 @@ dependencies = [
"moka",
"rand 0.10.0",
"reqwest 0.13.2",
"rustfs-utils",
"serde",
"serde_json",
"sha2 0.11.0-rc.5",
"temp-env",
"tempfile",
"thiserror 2.0.18",
"tokio",
@@ -7674,7 +7765,7 @@ dependencies = [
"async-trait",
"crossbeam-queue",
"futures",
"parking_lot",
"parking_lot 0.12.5",
"rustfs-utils",
"serde",
"serde_json",
@@ -7764,10 +7855,14 @@ dependencies = [
name = "rustfs-obs"
version = "0.0.5"
dependencies = [
"crossbeam-channel",
"crossbeam-deque",
"crossbeam-utils",
"flate2",
"glob",
"jiff",
"metrics",
"num_cpus",
"nvml-wrapper",
"opentelemetry",
"opentelemetry-appender-tracing",
@@ -7779,6 +7874,7 @@ dependencies = [
"rustfs-config",
"rustfs-utils",
"serde",
"serde_json",
"sysinfo",
"temp-env",
"tempfile",
@@ -7789,6 +7885,7 @@ dependencies = [
"tracing-error",
"tracing-opentelemetry",
"tracing-subscriber",
"zstd",
]
[[package]]
@@ -7933,7 +8030,7 @@ dependencies = [
"futures-core",
"http 1.4.0",
"object_store",
"parking_lot",
"parking_lot 0.12.5",
"pin-project-lite",
"rustfs-common",
"rustfs-ecstore",
@@ -7956,7 +8053,7 @@ dependencies = [
"datafusion",
"derive_builder 0.20.2",
"futures",
"parking_lot",
"parking_lot 0.12.5",
"rustfs-s3select-api",
"s3s",
"snafu 0.9.0",
@@ -8058,6 +8155,7 @@ name = "rustfs-utils"
version = "0.0.5"
dependencies = [
"base64-simd",
"blake2 0.11.0-rc.5",
"blake3",
"brotli",
"bytes",
@@ -8089,6 +8187,7 @@ dependencies = [
"siphasher",
"snap",
"sysinfo",
"temp-env",
"tempfile",
"thiserror 2.0.18",
"tokio",
@@ -8298,7 +8397,7 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "s3s"
version = "0.14.0-dev"
source = "git+https://github.com/s3s-project/s3s?rev=c2dc7b16535659904d4efff52c558fc039be1ef3#c2dc7b16535659904d4efff52c558fc039be1ef3"
source = "git+https://github.com/rustfs/s3s?rev=d9556e3c0036bd3f2b330966009cbaa5aebf19a3#d9556e3c0036bd3f2b330966009cbaa5aebf19a3"
dependencies = [
"arc-swap",
"arrayvec",
@@ -8510,6 +8609,16 @@ dependencies = [
"serde",
]
[[package]]
name = "serde_bytes"
version = "0.11.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8"
dependencies = [
"serde",
"serde_core",
]
[[package]]
name = "serde_core"
version = "1.0.228"
@@ -8637,7 +8746,7 @@ dependencies = [
"futures-util",
"log",
"once_cell",
"parking_lot",
"parking_lot 0.12.5",
"scc",
"serial_test_derive",
]
@@ -9233,7 +9342,7 @@ version = "0.3.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "96374855068f47402c3121c6eed88d29cb1de8f3ab27090e273e420bdabcf050"
dependencies = [
"parking_lot",
"parking_lot 0.12.5",
]
[[package]]
@@ -9460,7 +9569,7 @@ dependencies = [
"bytes",
"libc",
"mio",
"parking_lot",
"parking_lot 0.12.5",
"pin-project-lite",
"signal-hook-registry",
"socket2",
+9 -3
View File
@@ -148,6 +148,7 @@ rmcp = { version = "1.2.0" }
rmp = { version = "0.8.15" }
rmp-serde = { version = "1.3.1" }
serde = { version = "1.0.228", features = ["derive"] }
serde_bytes = "0.11"
serde_json = { version = "1.0.149", features = ["raw_value"] }
serde_urlencoded = "0.7.1"
schemars = "1.2.1"
@@ -155,6 +156,7 @@ schemars = "1.2.1"
# Cryptography and Security
aes-gcm = { version = "0.11.0-rc.3", features = ["rand_core"] }
argon2 = { version = "0.6.0-rc.7" }
blake2 = "0.11.0-rc.5"
blake3 = { version = "1.8.3", features = ["rayon", "mmap"] }
chacha20poly1305 = { version = "0.11.0-rc.3" }
crc-fast = "1.9.0"
@@ -179,7 +181,7 @@ time = { version = "0.3.47", features = ["std", "parsing", "formatting", "macros
# Utilities and Tools
anyhow = "1.0.102"
arc-swap = "1.8.2"
astral-tokio-tar = "0.5.6"
astral-tokio-tar = "0.6.0"
atoi = "2.0.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.8.15" }
@@ -197,6 +199,9 @@ const-str = { version = "1.1.0", features = ["std", "proc"] }
convert_case = "0.11.0"
criterion = { version = "0.8", features = ["html_reports"] }
crossbeam-queue = "0.3.12"
crossbeam-channel = "0.5.15"
crossbeam-deque = "0.8.6"
crossbeam-utils = "0.8.21"
datafusion = "52.3.0"
derive_builder = "0.20.2"
enumset = "1.1.10"
@@ -233,13 +238,14 @@ pretty_assertions = "1.4.1"
rand = { version = "0.10.0", features = ["serde"] }
ratelimit = "0.10.0"
rayon = "1.11.0"
reed-solomon-simd = { version = "3.1.0" }
reed-solomon-erasure = { version = "6.0", default-features = false, features = ["std", "simd-accel"] }
reed-solomon-simd = "3.1.0"
regex = { version = "1.12.3" }
rumqttc = { version = "0.25.1" }
rustix = { version = "1.1.4", features = ["fs"] }
rust-embed = { version = "8.11.0" }
rustc-hash = { version = "2.1.1" }
s3s = { git = "https://github.com/s3s-project/s3s", rev = "c2dc7b16535659904d4efff52c558fc039be1ef3", features = ["minio"] }
s3s = { git = "https://github.com/rustfs/s3s", rev = "d9556e3c0036bd3f2b330966009cbaa5aebf19a3", features = ["minio"] }
serial_test = "3.4.0"
shadow-rs = { version = "1.7.1", default-features = false }
siphasher = "1.0.2"
+1 -6
View File
@@ -86,12 +86,7 @@ RUN addgroup -g 10001 -S rustfs && \
chown -R rustfs:rustfs /data /logs && \
chmod 0750 /data /logs
ENV RUSTFS_ADDRESS=":9000" \
RUSTFS_CONSOLE_ADDRESS=":9001" \
RUSTFS_ACCESS_KEY="rustfsadmin" \
RUSTFS_SECRET_KEY="rustfsadmin" \
RUSTFS_CONSOLE_ENABLE="true" \
RUSTFS_CORS_ALLOWED_ORIGINS="*" \
ENV RUSTFS_CORS_ALLOWED_ORIGINS="*" \
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS="*" \
RUSTFS_VOLUMES="/data" \
RUSTFS_OBS_LOGGER_LEVEL=warn \
+65
View File
@@ -0,0 +1,65 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM rust:1.91-trixie
RUN set -eux; \
export DEBIAN_FRONTEND=noninteractive; \
apt-get update; \
apt-get install -y --no-install-recommends \
build-essential \
ca-certificates \
curl \
git \
pkg-config \
libssl-dev \
lld \
protobuf-compiler \
flatbuffers-compiler; \
rm -rf /var/lib/apt/lists/*
WORKDIR /usr/src/rustfs
COPY . .
RUN ./scripts/static.sh
RUN cargo run --bin gproto
RUN cargo build --release --locked --bin rustfs
RUN set -eux; \
groupadd -g 10001 rustfs; \
useradd -u 10001 -g rustfs -M -s /usr/sbin/nologin rustfs
WORKDIR /app
RUN set -eux; \
mkdir -p /data /logs; \
chown -R rustfs:rustfs /data /logs /app; \
chmod 0750 /data /logs
COPY entrypoint.sh /entrypoint.sh
RUN install -m 0755 /usr/src/rustfs/target/release/rustfs /usr/bin/rustfs && chmod +x /entrypoint.sh
ENV RUSTFS_VOLUMES="/data" \
RUST_LOG="warn" \
RUSTFS_OBS_LOG_DIRECTORY="/logs" \
RUSTFS_USERNAME="rustfs" \
RUSTFS_GROUPNAME="rustfs" \
RUSTFS_UID="10001" \
RUSTFS_GID="10001"
EXPOSE 9000 9001
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/usr/bin/rustfs"]
+2 -10
View File
@@ -157,11 +157,7 @@ WORKDIR /app
ENV CARGO_INCREMENTAL=1
# Ensure we have the same default env vars available
ENV RUSTFS_ADDRESS=":9000" \
RUSTFS_ACCESS_KEY="rustfsadmin" \
RUSTFS_SECRET_KEY="rustfsadmin" \
RUSTFS_CONSOLE_ENABLE="true" \
RUSTFS_VOLUMES="/data" \
ENV RUSTFS_VOLUMES="/data" \
RUST_LOG="warn" \
RUSTFS_OBS_LOG_DIRECTORY="/logs" \
RUSTFS_USERNAME="rustfs" \
@@ -222,11 +218,7 @@ COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /entrypoint.sh
# Default environment (override in docker run/compose as needed)
ENV RUSTFS_ADDRESS=":9000" \
RUSTFS_ACCESS_KEY="rustfsadmin" \
RUSTFS_SECRET_KEY="rustfsadmin" \
RUSTFS_CONSOLE_ENABLE="true" \
RUSTFS_VOLUMES="/data" \
ENV RUSTFS_VOLUMES="/data" \
RUST_LOG="warn" \
RUSTFS_USERNAME="rustfs" \
RUSTFS_GROUPNAME="rustfs" \
+27
View File
@@ -48,6 +48,12 @@ pub const ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_TOTAL_SIZ
pub const ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES";
pub const ENV_OBS_LOG_COMPRESS_OLD_FILES: &str = "RUSTFS_OBS_LOG_COMPRESS_OLD_FILES";
pub const ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL: &str = "RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL";
pub const ENV_OBS_LOG_COMPRESSION_ALGORITHM: &str = "RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM";
pub const ENV_OBS_LOG_PARALLEL_COMPRESS: &str = "RUSTFS_OBS_LOG_PARALLEL_COMPRESS";
pub const ENV_OBS_LOG_PARALLEL_WORKERS: &str = "RUSTFS_OBS_LOG_PARALLEL_WORKERS";
pub const ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL: &str = "RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL";
pub const ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP: &str = "RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP";
pub const ENV_OBS_LOG_ZSTD_WORKERS: &str = "RUSTFS_OBS_LOG_ZSTD_WORKERS";
pub const ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: &str = "RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS";
pub const ENV_OBS_LOG_EXCLUDE_PATTERNS: &str = "RUSTFS_OBS_LOG_EXCLUDE_PATTERNS";
pub const ENV_OBS_LOG_DELETE_EMPTY_FILES: &str = "RUSTFS_OBS_LOG_DELETE_EMPTY_FILES";
@@ -61,13 +67,24 @@ pub const DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES: u64 = 2 * 1024 * 1024 * 1024; //
pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No single file limit
pub const DEFAULT_OBS_LOG_COMPRESS_OLD_FILES: bool = true;
pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL: u32 = 6;
pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP: &str = "gzip";
pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD: &str = "zstd";
pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM: &str = DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD;
pub const DEFAULT_OBS_LOG_PARALLEL_COMPRESS: bool = true;
pub const DEFAULT_OBS_LOG_PARALLEL_WORKERS: usize = 6;
pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL: i32 = 8;
pub const DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP: bool = true;
pub const DEFAULT_OBS_LOG_ZSTD_WORKERS: usize = 1;
pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION: &str = "gz";
pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_ALL_EXTENSION: &str = concat!(".", DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION);
pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_EXTENSION: &str = "zst";
pub const DEFAULT_OBS_LOG_ZSTD_COMPRESSION_ALL_EXTENSION: &str = concat!(".", DEFAULT_OBS_LOG_ZSTD_COMPRESSION_EXTENSION);
pub const DEFAULT_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS: u64 = 30; // Retain compressed files for 30 days
pub const DEFAULT_OBS_LOG_DELETE_EMPTY_FILES: bool = true;
pub const DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS: u64 = 3600; // 1 hour
pub const DEFAULT_OBS_LOG_CLEANUP_INTERVAL_SECONDS: u64 = 1800; // 0.5 hours
pub const DEFAULT_OBS_LOG_DRY_RUN: bool = false;
pub const DEFAULT_OBS_LOG_MATCH_MODE_PREFIX: &str = "prefix";
pub const DEFAULT_OBS_LOG_MATCH_MODE: &str = "suffix";
/// Default values for observability configuration
@@ -113,6 +130,12 @@ mod tests {
assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");
assert_eq!(ENV_OBS_LOG_COMPRESS_OLD_FILES, "RUSTFS_OBS_LOG_COMPRESS_OLD_FILES");
assert_eq!(ENV_OBS_LOG_GZIP_COMPRESSION_LEVEL, "RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL");
assert_eq!(ENV_OBS_LOG_COMPRESSION_ALGORITHM, "RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM");
assert_eq!(ENV_OBS_LOG_PARALLEL_COMPRESS, "RUSTFS_OBS_LOG_PARALLEL_COMPRESS");
assert_eq!(ENV_OBS_LOG_PARALLEL_WORKERS, "RUSTFS_OBS_LOG_PARALLEL_WORKERS");
assert_eq!(ENV_OBS_LOG_ZSTD_COMPRESSION_LEVEL, "RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL");
assert_eq!(ENV_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, "RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP");
assert_eq!(ENV_OBS_LOG_ZSTD_WORKERS, "RUSTFS_OBS_LOG_ZSTD_WORKERS");
assert_eq!(
ENV_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS,
"RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS"
@@ -131,6 +154,10 @@ mod tests {
assert_eq!(DEFAULT_OBS_ENVIRONMENT_DEVELOPMENT, "development");
assert_eq!(DEFAULT_OBS_ENVIRONMENT_TEST, "test");
assert_eq!(DEFAULT_OBS_ENVIRONMENT_STAGING, "staging");
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP, "gzip");
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD, "zstd");
assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE_PREFIX, "prefix");
assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE, "suffix");
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, "zstd");
}
}
+1 -1
View File
@@ -15,7 +15,7 @@ base64-simd = { workspace = true }
rand = { workspace = true }
serde = { workspace = true }
serde_json.workspace = true
time = { workspace = true, features = ["serde-human-readable"] }
time = { workspace = true, features = ["serde", "parsing", "formatting", "macros"] }
[lints]
workspace = true
+37
View File
@@ -273,15 +273,25 @@ impl<'a> fmt::Display for Masked<'a> {
///
#[derive(Serialize, Deserialize, Clone, Default)]
pub struct Credentials {
#[serde(rename = "accessKey", alias = "access_key", default)]
pub access_key: String,
#[serde(rename = "secretKey", alias = "secret_key", default)]
pub secret_key: String,
#[serde(rename = "sessionToken", alias = "session_token", default)]
pub session_token: String,
#[serde(default, with = "crate::serde_datetime::option")]
pub expiration: Option<OffsetDateTime>,
#[serde(default)]
pub status: String,
#[serde(rename = "parentUser", alias = "parent_user", default)]
pub parent_user: String,
#[serde(default)]
pub groups: Option<Vec<String>>,
#[serde(default)]
pub claims: Option<HashMap<String, Value>>,
#[serde(default)]
pub name: Option<String>,
#[serde(default)]
pub description: Option<String>,
}
@@ -491,4 +501,31 @@ mod tests {
assert_eq!(format!("{:?}", Masked(Some("中文"))), "中***|2");
assert_eq!(format!("{:?}", Masked(Some("中文测试"))), "中***试|4");
}
#[test]
fn test_credentials_expiration_serialize_as_rfc3339() {
use time::OffsetDateTime;
let c = Credentials {
access_key: "ak".to_string(),
secret_key: "sk12345678".to_string(),
expiration: Some(OffsetDateTime::now_utc()),
..Default::default()
};
let json = serde_json::to_string(&c).expect("serialize");
assert!(
json.contains('T') && (json.contains('Z') || json.contains("+00:00")),
"Credentials expiration should be RFC3339; got: {}",
json
);
}
#[test]
fn test_credentials_deserialize_minio_style_rfc3339_expiration() {
let minio_style = r#"{"accessKey":"ak","secretKey":"sk12345678","expiration":"2025-03-07T12:00:00Z"}"#;
let c: Credentials = serde_json::from_str(minio_style).expect("deserialize");
assert_eq!(c.access_key, "ak");
assert!(c.expiration.is_some());
}
}
+1
View File
@@ -14,6 +14,7 @@
mod constants;
mod credentials;
mod serde_datetime;
pub use constants::*;
pub use credentials::*;
+68
View File
@@ -0,0 +1,68 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Serde helpers for expiration timestamp: serialize as RFC3339 (MinIO-compatible),
//! deserialize from RFC3339 or legacy RustFS human-readable format.
use time::OffsetDateTime;
use time::format_description;
use time::format_description::well_known::Rfc3339;
static LEGACY_FORMAT: std::sync::OnceLock<time::format_description::OwnedFormatItem> = std::sync::OnceLock::new();
fn legacy_format() -> &'static time::format_description::OwnedFormatItem {
LEGACY_FORMAT.get_or_init(|| {
format_description::parse_owned::<2>(
"[year]-[month]-[day] [hour]:[minute]:[second].[subsecond] [offset_hour sign:mandatory]:[offset_minute]:[offset_second]",
)
.expect("legacy format description is valid")
});
LEGACY_FORMAT.get().expect("initialized above")
}
fn parse_rfc3339_or_legacy(s: &str) -> Result<OffsetDateTime, time::Error> {
OffsetDateTime::parse(s, &Rfc3339).or_else(|_| OffsetDateTime::parse(s, legacy_format()).map_err(Into::into))
}
/// Option<OffsetDateTime>: serialize as RFC3339; deserialize from RFC3339 or legacy.
pub mod option {
use serde::{Deserialize, Deserializer, Serializer};
use time::OffsetDateTime;
use super::{Rfc3339, parse_rfc3339_or_legacy};
pub fn serialize<S>(opt: &Option<OffsetDateTime>, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
match opt {
Some(dt) => {
let s = dt.format(&Rfc3339).map_err(serde::ser::Error::custom)?;
serializer.serialize_some(&s)
}
None => serializer.serialize_none(),
}
}
pub fn deserialize<'de, D>(deserializer: D) -> Result<Option<OffsetDateTime>, D::Error>
where
D: Deserializer<'de>,
{
let opt: Option<&str> = Option::deserialize(deserializer)?;
match opt {
None => Ok(None),
Some(s) => parse_rfc3339_or_legacy(s).map(Some).map_err(serde::de::Error::custom),
}
}
}
+3
View File
@@ -21,5 +21,8 @@ pub(crate) mod id;
pub(crate) mod decrypt;
pub(crate) mod encrypt;
#[cfg(any(test, feature = "crypto"))]
pub(crate) mod stream_io;
#[cfg(test)]
mod tests;
+219
View File
@@ -0,0 +1,219 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! sio-go compatible stream encryption for IAM config.
//! Header: salt(32) + alg_id(1) + nonce_prefix(8) = 41 bytes.
//! Body: DARE-style fragmented AEAD (bufSize=16384, per-fragment nonce).
#![allow(deprecated)] // AeadInPlace deprecated in favor of AeadInOut; keep for aead 0.6 compatibility
use crate::encdec::id::ID;
use crate::error::Error;
use aes_gcm::{
Aes256Gcm,
aead::{AeadCore, AeadInPlace, KeyInit as _, array::Array},
};
use chacha20poly1305::ChaCha20Poly1305;
const STREAM_IO_HEADER_LEN: usize = 41;
const SIO_BUF_SIZE: usize = 16384;
const SIO_NONCE_PREFIX_LEN: usize = 8;
const AES_GCM_OVERHEAD: usize = 16;
const CHACHA_OVERHEAD: usize = 16;
/// Decrypt data in stream_io (sio-go) format.
pub fn decrypt_stream_io(password: &[u8], data: &[u8]) -> Result<Vec<u8>, Error> {
if data.len() < STREAM_IO_HEADER_LEN {
return Err(Error::ErrUnexpectedHeader);
}
let salt = &data[0..32];
let id = ID::try_from(data[32])?;
let nonce_prefix = &data[33..41];
let body = &data[STREAM_IO_HEADER_LEN..];
let key = id.get_key(password, salt)?;
match id {
ID::Argon2idChaCHa20Poly1305 => decrypt_stream(
ChaCha20Poly1305::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
nonce_prefix,
body,
CHACHA_OVERHEAD,
),
_ => decrypt_stream(
Aes256Gcm::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
nonce_prefix,
body,
AES_GCM_OVERHEAD,
),
}
}
fn decrypt_stream<A>(aead: A, nonce_prefix: &[u8], body: &[u8], overhead: usize) -> Result<Vec<u8>, Error>
where
A: AeadInPlace,
{
let ciphertext_len = SIO_BUF_SIZE + overhead;
let ad = build_associated_data(&aead, nonce_prefix)?;
let mut plain = Vec::with_capacity(body.len());
let mut seq_num: u32 = 1;
let mut pos = 0;
while pos < body.len() {
let remaining = body.len() - pos;
let frag_len = remaining.min(ciphertext_len);
let is_last = (pos + frag_len) == body.len();
if frag_len < overhead {
return Err(Error::ErrDecryptFailed(aes_gcm::aead::Error));
}
let mut nonce = [0u8; 12];
nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
nonce[8..12].copy_from_slice(&seq_num.to_le_bytes());
let mut ad_mut = ad.clone();
ad_mut[0] = if is_last { 0x80 } else { 0x00 };
let fragment = &body[pos..pos + frag_len];
let tag_len = overhead;
let (ct, tag) = fragment.split_at(frag_len - tag_len);
let mut buffer = ct.to_vec();
let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
.map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
let tag_arr =
Array::<u8, <A as AeadCore>::TagSize>::try_from(tag).map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
aead.decrypt_in_place_detached(&nonce_arr, &ad_mut, &mut buffer, &tag_arr)
.map_err(|_| Error::ErrDecryptFailed(aes_gcm::aead::Error))?;
plain.extend_from_slice(&buffer);
pos += frag_len;
seq_num += 1;
if is_last {
break;
}
}
Ok(plain)
}
fn build_associated_data<A>(aead: &A, nonce_prefix: &[u8]) -> Result<Vec<u8>, Error>
where
A: AeadInPlace,
{
let mut nonce = [0u8; 12];
nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
nonce[8..12].copy_from_slice(&0u32.to_le_bytes());
let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
.map_err(|_| Error::ErrEncryptFailed(aes_gcm::aead::Error))?;
let mut empty: [u8; 0] = [];
let tag = aead
.encrypt_in_place_detached(&nonce_arr, &[] as &[u8], &mut empty)
.map_err(Error::ErrEncryptFailed)?;
let mut ad = vec![0u8; 1 + tag.len()];
ad[0] = 0x00;
ad[1..].copy_from_slice(tag.as_slice());
Ok(ad)
}
/// Encrypt data in stream_io (sio-go) format.
pub fn encrypt_stream_io(password: &[u8], data: &[u8]) -> Result<Vec<u8>, Error> {
let salt: [u8; 32] = rand::random();
#[cfg(feature = "fips")]
let id = ID::Pbkdf2AESGCM;
#[cfg(not(feature = "fips"))]
let id = if crate::encdec::encrypt::native_aes() {
ID::Argon2idAESGCM
} else {
ID::Argon2idChaCHa20Poly1305
};
let key = id.get_key(password, &salt)?;
let nonce_prefix: [u8; SIO_NONCE_PREFIX_LEN] = rand::random();
let mut out = Vec::with_capacity(STREAM_IO_HEADER_LEN + data.len() + 32);
out.extend_from_slice(&salt);
out.push(id as u8);
out.extend_from_slice(&nonce_prefix);
match id {
ID::Argon2idChaCHa20Poly1305 => encrypt_stream(
ChaCha20Poly1305::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
&nonce_prefix,
data,
&mut out,
CHACHA_OVERHEAD,
)?,
_ => encrypt_stream(
Aes256Gcm::new_from_slice(&key).map_err(|e| Error::ErrInvalidInput(e.to_string()))?,
&nonce_prefix,
data,
&mut out,
AES_GCM_OVERHEAD,
)?,
}
Ok(out)
}
fn encrypt_stream<A>(
aead: A,
nonce_prefix: &[u8; SIO_NONCE_PREFIX_LEN],
data: &[u8],
out: &mut Vec<u8>,
_overhead: usize,
) -> Result<(), Error>
where
A: AeadInPlace,
{
let ad = build_associated_data(&aead, nonce_prefix)?;
let mut seq_num: u32 = 1;
let mut pos = 0;
while pos < data.len() {
let remaining = data.len() - pos;
let is_last = remaining <= SIO_BUF_SIZE;
let chunk_len = if is_last { remaining } else { SIO_BUF_SIZE };
let chunk = &data[pos..pos + chunk_len];
let mut nonce = [0u8; 12];
nonce[0..SIO_NONCE_PREFIX_LEN].copy_from_slice(nonce_prefix);
nonce[8..12].copy_from_slice(&seq_num.to_le_bytes());
let mut ad_mut = ad.clone();
ad_mut[0] = if is_last { 0x80 } else { 0x00 };
let mut buffer = chunk.to_vec();
let nonce_arr = Array::<u8, <A as AeadCore>::NonceSize>::try_from(&nonce[..])
.map_err(|_| Error::ErrEncryptFailed(aes_gcm::aead::Error))?;
let tag = aead
.encrypt_in_place_detached(&nonce_arr, &ad_mut, &mut buffer)
.map_err(Error::ErrEncryptFailed)?;
out.extend_from_slice(&buffer);
out.extend_from_slice(tag.as_slice());
pos += chunk_len;
seq_num += 1;
}
Ok(())
}
+65 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{decrypt_data, encrypt_data};
use crate::{decrypt_data, decrypt_stream_io, encrypt_data, encrypt_stream_io};
const PASSWORD: &[u8] = "test_password".as_bytes();
const LONG_PASSWORD: &[u8] = "very_long_password_with_many_characters_for_testing_purposes_123456789".as_bytes();
@@ -317,3 +317,67 @@ fn test_concurrent_encryption_safety() -> Result<(), crate::Error> {
Ok(())
}
#[test]
fn test_stream_io_roundtrip() -> Result<(), crate::Error> {
let password = b"access:secret";
let data = br#"{"Version":1,"policy":"readonly"}"#;
let encrypted = encrypt_stream_io(password, data)?;
let decrypted = decrypt_stream_io(password, &encrypted)?;
assert_eq!(data, decrypted.as_slice());
Ok(())
}
#[test]
fn test_stream_io_large_data_roundtrip() -> Result<(), crate::Error> {
let password = b"access:secret";
let data = vec![0xAB; 32 * 1024]; // > SIO_BUF_SIZE to test fragmentation
let encrypted = encrypt_stream_io(password, &data)?;
let decrypted = decrypt_stream_io(password, &encrypted)?;
assert_eq!(data, decrypted);
Ok(())
}
#[test]
fn test_stream_io_wrong_password_fails() {
let password = b"access:secret";
let data = br#"{"Version":1}"#;
let encrypted = encrypt_stream_io(password, data).expect("encrypt should succeed");
let result = decrypt_stream_io(b"wrong:password", &encrypted);
assert!(result.is_err(), "decrypt with wrong password should fail");
}
#[test]
fn test_stream_io_empty_data() -> Result<(), crate::Error> {
let password = b"access:secret";
let data: &[u8] = &[];
let encrypted = encrypt_stream_io(password, data)?;
let decrypted = decrypt_stream_io(password, &encrypted)?;
assert!(decrypted.is_empty());
Ok(())
}
#[test]
fn test_stream_io_header_format() -> Result<(), crate::Error> {
let password = b"access:secret";
let data = b"test";
let encrypted = encrypt_stream_io(password, data)?;
// stream_io header: salt(32) + alg_id(1) + nonce_prefix(8) = 41 bytes
const STREAM_IO_HEADER_LEN: usize = 41;
assert!(encrypted.len() >= STREAM_IO_HEADER_LEN, "encrypted should have at least 41-byte header");
assert!(
encrypted[32] == 0x00 || encrypted[32] == 0x01 || encrypted[32] == 0x02,
"alg_id should be 0x00, 0x01, or 0x02"
);
Ok(())
}
#[test]
fn test_stream_io_truncated_data_fails() {
let password = b"access:secret";
let data = b"test";
let encrypted = encrypt_stream_io(password, data).expect("encrypt should succeed");
let truncated = &encrypted[..40]; // less than 41-byte header
let result = decrypt_stream_io(password, truncated);
assert!(result.is_err(), "truncated data should fail decrypt");
}
+3
View File
@@ -19,6 +19,9 @@ mod jwt;
pub use encdec::decrypt::decrypt_data;
pub use encdec::encrypt::encrypt_data;
#[cfg(feature = "crypto")]
pub use encdec::stream_io::{decrypt_stream_io, encrypt_stream_io};
pub use error::Error;
pub use jwt::decode::decode as jwt_decode;
pub use jwt::encode::encode as jwt_encode;
+82 -21
View File
@@ -19,7 +19,6 @@ use aws_sdk_s3::config::{Credentials, Region};
use bytes::Bytes;
use serial_test::serial;
use std::error::Error;
use tokio::time::sleep;
const ENDPOINT: &str = "http://localhost:9000";
const ACCESS_KEY: &str = "rustfsadmin";
@@ -62,6 +61,7 @@ async fn setup_test_bucket(client: &Client) -> Result<(), Box<dyn Error>> {
#[ignore = "requires running RustFS server at localhost:9000"]
async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error::Error>> {
use aws_sdk_s3::types::{BucketLifecycleConfiguration, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter};
use chrono::{Duration as ChronoDuration, Utc};
use tokio::time::Duration;
let client = create_aws_s3_client().await?;
@@ -70,6 +70,7 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
// Upload test object first
let test_content = "Test object for lifecycle expiration";
let lifecycle_object_key = "lifecycle-test-object.txt";
let untouched_object_key = "keep-object.txt";
client
.put_object()
.bucket(BUCKET)
@@ -77,13 +78,29 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
.body(Bytes::from(test_content.as_bytes()).into())
.send()
.await?;
client
.put_object()
.bucket(BUCKET)
.key(untouched_object_key)
.body(Bytes::from("should-stay".as_bytes()).into())
.send()
.await?;
// Verify object exists initially
let resp = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await?;
assert!(resp.content_length().unwrap_or(0) > 0);
let untouched_resp = client.get_object().bucket(BUCKET).key(untouched_object_key).send().await?;
assert!(untouched_resp.content_length().unwrap_or(0) > 0);
// Configure lifecycle rule: expire after current time + 3 seconds
let expiration = LifecycleExpiration::builder().days(0).build();
// Use a past midnight UTC date to trigger immediate lifecycle expiry without requiring days=0.
let yesterday_midnight_utc = Utc::now()
.date_naive()
.and_hms_opt(0, 0, 0)
.expect("midnight should always be valid")
- ChronoDuration::days(1);
let expiration = LifecycleExpiration::builder()
.date(aws_sdk_s3::primitives::DateTime::from_secs(yesterday_midnight_utc.and_utc().timestamp()))
.build();
let filter = LifecycleRuleFilter::builder().prefix(lifecycle_object_key).build();
let rule = LifecycleRule::builder()
.id("expire-test-object")
@@ -105,29 +122,73 @@ async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error:
let rules = resp.rules();
assert!(rules.iter().any(|r| r.id().unwrap_or("") == "expire-test-object"));
// Wait for lifecycle processing (scanner runs every 1 second)
sleep(Duration::from_secs(3)).await;
// After lifecycle processing, the object should be deleted by the lifecycle rule
let get_result = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await;
match get_result {
Ok(_) => {
panic!("Expected object to be deleted by lifecycle rule, but it still exists");
}
Err(e) => {
if let Some(service_error) = e.as_service_error() {
if service_error.is_no_such_key() {
println!("Lifecycle configuration test completed - object was successfully deleted by lifecycle rule");
} else {
panic!("Expected NoSuchKey error, but got: {e:?}");
// Poll for deletion instead of using a fixed sleep to keep the test deterministic.
// Default scanner cycle interval is 60s with jitter, so allow enough time for one full cycle.
let deadline = tokio::time::Instant::now() + Duration::from_secs(150);
loop {
let get_result = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await;
match get_result {
Ok(_) => {
if tokio::time::Instant::now() >= deadline {
panic!("Expected object to be deleted by lifecycle rule within 150s, but it still exists");
}
tokio::time::sleep(Duration::from_secs(1)).await;
}
Err(e) => {
if let Some(service_error) = e.as_service_error() {
if service_error.is_no_such_key() {
println!("Lifecycle configuration test completed - object was successfully deleted by lifecycle rule");
break;
}
panic!("Expected NoSuchKey error, but got: {e:?}");
} else {
panic!("Expected service error, but got: {e:?}");
}
} else {
panic!("Expected service error, but got: {e:?}");
}
}
}
println!("Lifecycle configuration test completed.");
// Non-matching prefix object should remain available.
let untouched_after = client.get_object().bucket(BUCKET).key(untouched_object_key).send().await?;
assert!(untouched_after.content_length().unwrap_or(0) > 0);
Ok(())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
#[serial]
#[ignore = "requires running RustFS server at localhost:9000"]
async fn test_bucket_lifecycle_rejects_zero_days() -> Result<(), Box<dyn std::error::Error>> {
use aws_sdk_s3::types::{BucketLifecycleConfiguration, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter};
let client = create_aws_s3_client().await?;
setup_test_bucket(&client).await?;
let expiration = LifecycleExpiration::builder().days(0).build();
let filter = LifecycleRuleFilter::builder().prefix("zero-days/").build();
let rule = LifecycleRule::builder()
.id("expire-zero-days")
.filter(filter)
.expiration(expiration)
.status(aws_sdk_s3::types::ExpirationStatus::Enabled)
.build()?;
let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
let err = client
.put_bucket_lifecycle_configuration()
.bucket(BUCKET)
.lifecycle_configuration(lifecycle)
.send()
.await
.expect_err("zero-day lifecycle expiration should be rejected");
let err_msg = format!("{err:?}");
assert!(
err_msg.contains("InvalidArgument") && err_msg.contains("greater than 0"),
"unexpected error: {err_msg}"
);
Ok(())
}
+3
View File
@@ -66,6 +66,7 @@ http-body = { workspace = true }
http-body-util.workspace = true
url.workspace = true
uuid = { workspace = true, features = ["v4", "fast-rng", "serde"] }
reed-solomon-erasure = { workspace = true }
reed-solomon-simd = { workspace = true }
lazy_static.workspace = true
rustfs-lock.workspace = true
@@ -73,6 +74,7 @@ regex = { workspace = true }
path-absolutize = { workspace = true }
rmp.workspace = true
rmp-serde.workspace = true
serde_bytes.workspace = true
tokio-util = { workspace = true, features = ["io", "compat"] }
base64 = { workspace = true }
hmac = { workspace = true }
@@ -120,6 +122,7 @@ tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
criterion = { workspace = true, features = ["html_reports"] }
temp-env = { workspace = true }
tracing-subscriber = { workspace = true }
serial_test = { workspace = true }
[build-dependencies]
shadow-rs = { workspace = true, features = ["build", "metadata"] }
+51 -68
View File
@@ -118,41 +118,38 @@ fn bench_encode_performance(c: &mut Criterion) {
});
group.finish();
// Test direct SIMD implementation for large shards (>= 512 bytes)
// Test direct reed-solomon-erasure implementation for large shards (>= 512 bytes)
let shard_size = calc_shard_size(config.data_size, config.data_shards);
if shard_size >= 512 {
let mut simd_group = c.benchmark_group("encode_simd_direct");
simd_group.throughput(Throughput::Bytes(config.data_size as u64));
simd_group.sample_size(10);
simd_group.measurement_time(Duration::from_secs(5));
if shard_size >= 512 && config.parity_shards > 0 {
use reed_solomon_erasure::galois_8::ReedSolomon;
simd_group.bench_with_input(BenchmarkId::new("simd_direct", &config.name), &(&data, &config), |b, (data, config)| {
b.iter(|| {
// Direct SIMD implementation
let per_shard_size = calc_shard_size(data.len(), config.data_shards);
match reed_solomon_simd::ReedSolomonEncoder::new(config.data_shards, config.parity_shards, per_shard_size) {
Ok(mut encoder) => {
// Create properly sized buffer and fill with data
let mut buffer = vec![0u8; per_shard_size * config.data_shards];
let mut rse_group = c.benchmark_group("encode_rse_direct");
rse_group.throughput(Throughput::Bytes(config.data_size as u64));
rse_group.sample_size(10);
rse_group.measurement_time(Duration::from_secs(5));
if let Ok(rs) = ReedSolomon::new(config.data_shards, config.parity_shards) {
let total_shards = config.data_shards + config.parity_shards;
let per_shard_size = calc_shard_size(config.data_size, config.data_shards);
let need_total = per_shard_size * total_shards;
rse_group.bench_with_input(
BenchmarkId::new("rse_direct", &config.name),
&(&data, need_total, per_shard_size),
|b, (data, need_total, per_shard_size)| {
b.iter(|| {
let mut buffer = vec![0u8; *need_total];
let copy_len = data.len().min(buffer.len());
buffer[..copy_len].copy_from_slice(&data[..copy_len]);
// Add data shards with correct shard size
for chunk in buffer.chunks_exact(per_shard_size) {
encoder.add_original_shard(black_box(chunk)).unwrap();
}
let result = encoder.encode().unwrap();
black_box(result);
}
Err(_) => {
// SIMD doesn't support this configuration, skip
black_box(());
}
}
});
});
simd_group.finish();
let mut slices: Vec<&mut [u8]> = buffer.chunks_exact_mut(*per_shard_size).collect();
rs.encode(&mut slices).unwrap();
black_box(buffer);
});
},
);
}
rse_group.finish();
}
}
}
@@ -203,47 +200,33 @@ fn bench_decode_performance(c: &mut Criterion) {
);
group.finish();
// Test direct SIMD decoding for large shards
// Test direct reed-solomon-erasure decoding for large shards
let shard_size = calc_shard_size(config.data_size, config.data_shards);
if shard_size >= 512 {
let mut simd_group = c.benchmark_group("decode_simd_direct");
simd_group.throughput(Throughput::Bytes(config.data_size as u64));
simd_group.sample_size(10);
simd_group.measurement_time(Duration::from_secs(5));
if shard_size >= 512 && config.parity_shards > 0 {
use reed_solomon_erasure::galois_8::ReedSolomon;
simd_group.bench_with_input(
BenchmarkId::new("simd_direct", &config.name),
&(&encoded_shards, &config),
|b, (shards, config)| {
b.iter(|| {
let per_shard_size = calc_shard_size(config.data_size, config.data_shards);
match reed_solomon_simd::ReedSolomonDecoder::new(config.data_shards, config.parity_shards, per_shard_size)
{
Ok(mut decoder) => {
// Add available shards (except lost ones)
for (i, shard) in shards.iter().enumerate() {
if i != config.data_shards - 1 && i != config.data_shards {
if i < config.data_shards {
decoder.add_original_shard(i, black_box(shard)).unwrap();
} else {
let recovery_idx = i - config.data_shards;
decoder.add_recovery_shard(recovery_idx, black_box(shard)).unwrap();
}
}
}
if let Ok(rs) = ReedSolomon::new(config.data_shards, config.parity_shards) {
let mut rse_group = c.benchmark_group("decode_rse_direct");
rse_group.throughput(Throughput::Bytes(config.data_size as u64));
rse_group.sample_size(10);
rse_group.measurement_time(Duration::from_secs(5));
let result = decoder.decode().unwrap();
black_box(result);
}
Err(_) => {
// SIMD doesn't support this configuration, skip
black_box(());
}
}
});
},
);
simd_group.finish();
rse_group.bench_with_input(
BenchmarkId::new("rse_direct", &config.name),
&(&encoded_shards, &config),
|b, (shards, config)| {
b.iter(|| {
let mut shards_opt: Vec<Option<Vec<u8>>> = shards.iter().map(|s| Some(s.to_vec())).collect();
shards_opt[config.data_shards - 1] = None;
shards_opt[config.data_shards] = None;
rs.reconstruct_data(&mut shards_opt).unwrap();
black_box(shards_opt);
});
},
);
rse_group.finish();
}
}
}
}
+4 -4
View File
@@ -119,7 +119,7 @@ mod tests {
async fn test_create_bitrot_reader_with_inline_data() {
let test_data = b"hello world test data";
let shard_size = 16;
let checksum_algo = HashAlgorithm::HighwayHash256;
let checksum_algo = HashAlgorithm::HighwayHash256S;
let result =
create_bitrot_reader(Some(test_data), None, "test-bucket", "test-path", 0, 0, shard_size, checksum_algo, false).await;
@@ -131,7 +131,7 @@ mod tests {
#[tokio::test]
async fn test_create_bitrot_reader_without_data_or_disk() {
let shard_size = 16;
let checksum_algo = HashAlgorithm::HighwayHash256;
let checksum_algo = HashAlgorithm::HighwayHash256S;
let result =
create_bitrot_reader(None, None, "test-bucket", "test-path", 0, 1024, shard_size, checksum_algo, false).await;
@@ -151,7 +151,7 @@ mod tests {
"test-path",
1024, // length
1024, // shard_size
HashAlgorithm::HighwayHash256,
HashAlgorithm::HighwayHash256S,
)
.await;
@@ -183,7 +183,7 @@ mod tests {
"test-path",
1024, // length
1024, // shard_size
HashAlgorithm::HighwayHash256,
HashAlgorithm::HighwayHash256S,
)
.await;
+22 -31
View File
@@ -43,11 +43,13 @@ use rustfs_config::{DEFAULT_TRUST_LEAF_CERT_AS_CA, ENV_TRUST_LEAF_CERT_AS_CA, RU
use rustfs_filemeta::{ReplicationStatusType, ReplicationType};
use rustfs_utils::http::{
AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_LOCK_BYPASS_GOVERNANCE, AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE,
AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_STORAGE_CLASS, AMZ_WEBSITE_REDIRECT_LOCATION, RUSTFS_BUCKET_REPLICATION_CHECK,
RUSTFS_BUCKET_REPLICATION_DELETE_MARKER, RUSTFS_BUCKET_REPLICATION_REQUEST, RUSTFS_BUCKET_SOURCE_ETAG,
RUSTFS_BUCKET_SOURCE_MTIME, RUSTFS_BUCKET_SOURCE_VERSION_ID, RUSTFS_FORCE_DELETE, is_amz_header, is_minio_header,
AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_STORAGE_CLASS, AMZ_WEBSITE_REDIRECT_LOCATION, is_amz_header, is_minio_header,
is_rustfs_header, is_standard_header, is_storageclass_header,
};
use rustfs_utils::http::{
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK,
SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, insert_header,
};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::error::Error;
@@ -68,8 +70,6 @@ use uuid::Uuid;
const DEFAULT_HEALTH_CHECK_DURATION: Duration = Duration::from_secs(5);
const DEFAULT_HEALTH_CHECK_RELOAD_DURATION: Duration = Duration::from_secs(30 * 60);
const REPLICATION_REQUEST_TRUE: HeaderValue = HeaderValue::from_static("true");
pub static GLOBAL_BUCKET_TARGET_SYS: OnceLock<BucketTargetSys> = OnceLock::new();
#[derive(Debug, Clone)]
@@ -1081,23 +1081,21 @@ impl PutObjectOptions {
}
if !self.internal.source_version_id.is_empty() {
header.insert(
RUSTFS_BUCKET_SOURCE_VERSION_ID,
HeaderValue::from_str(&self.internal.source_version_id).expect("err"),
);
insert_header(&mut header, SUFFIX_SOURCE_VERSION_ID, &self.internal.source_version_id);
}
if self.internal.source_etag.is_empty() {
header.insert(RUSTFS_BUCKET_SOURCE_ETAG, HeaderValue::from_str(&self.internal.source_etag).expect("err"));
insert_header(&mut header, SUFFIX_SOURCE_ETAG, &self.internal.source_etag);
}
if self.internal.source_mtime.unix_timestamp() != 0 {
header.insert(
RUSTFS_BUCKET_SOURCE_MTIME,
HeaderValue::from_str(&self.internal.source_mtime.format(&Rfc3339).unwrap_or_default()).expect("err"),
insert_header(
&mut header,
SUFFIX_SOURCE_MTIME,
self.internal.source_mtime.format(&Rfc3339).unwrap_or_default(),
);
}
if self.internal.replication_request {
header.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, REPLICATION_REQUEST_TRUE);
insert_header(&mut header, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
}
header
@@ -1266,10 +1264,8 @@ impl TargetClient {
let builder = self.client.put_object();
let version_id = opts.internal.source_version_id.clone();
if !version_id.is_empty()
&& let Ok(header_value) = HeaderValue::from_str(&version_id)
{
headers.insert(RUSTFS_BUCKET_SOURCE_VERSION_ID, header_value);
if !version_id.is_empty() {
insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
}
match builder
@@ -1303,13 +1299,11 @@ impl TargetClient {
) -> Result<String, S3ClientError> {
let mut headers = HeaderMap::new();
let version_id = opts.internal.source_version_id.clone();
if !version_id.is_empty()
&& let Ok(header_value) = HeaderValue::from_str(&version_id)
{
headers.insert(RUSTFS_BUCKET_SOURCE_VERSION_ID, header_value);
if !version_id.is_empty() {
insert_header(&mut headers, SUFFIX_SOURCE_VERSION_ID, &version_id);
}
if opts.internal.replication_request {
headers.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, REPLICATION_REQUEST_TRUE);
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
}
match self
@@ -1418,21 +1412,18 @@ impl TargetClient {
) -> Result<(), S3ClientError> {
let mut headers = HeaderMap::new();
if opts.force_delete {
headers.insert(RUSTFS_FORCE_DELETE, "true".parse().unwrap());
insert_header(&mut headers, SUFFIX_FORCE_DELETE, "true");
}
if opts.governance_bypass {
headers.insert(AMZ_OBJECT_LOCK_BYPASS_GOVERNANCE, "true".parse().unwrap());
}
if opts.replication_delete_marker {
headers.insert(RUSTFS_BUCKET_REPLICATION_DELETE_MARKER, "true".parse().unwrap());
insert_header(&mut headers, SUFFIX_SOURCE_DELETEMARKER, "true");
}
if let Some(t) = opts.replication_mtime {
headers.insert(
RUSTFS_BUCKET_SOURCE_MTIME,
t.format(&Rfc3339).unwrap_or_default().as_str().parse().unwrap(),
);
insert_header(&mut headers, SUFFIX_SOURCE_MTIME, t.format(&Rfc3339).unwrap_or_default());
}
if !opts.replication_status.is_empty() {
@@ -1440,10 +1431,10 @@ impl TargetClient {
}
if opts.replication_request {
headers.insert(RUSTFS_BUCKET_REPLICATION_REQUEST, "true".parse().unwrap());
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_REQUEST, "true");
}
if opts.replication_validity_check {
headers.insert(RUSTFS_BUCKET_REPLICATION_CHECK, "true".parse().unwrap());
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_CHECK, "true");
}
match self
@@ -45,8 +45,7 @@ use rustfs_common::heal_channel::rep_has_active_rules;
use rustfs_common::metrics::{IlmAction, Metrics};
use rustfs_filemeta::{NULL_VERSION_ID, RestoreStatusOps, is_restored_object_on_disk};
use rustfs_s3_common::EventName;
use rustfs_utils::path::encode_dir_object;
use rustfs_utils::string::strings_has_prefix_fold;
use rustfs_utils::{get_env_i64, get_env_usize, path::encode_dir_object, string::strings_has_prefix_fold};
use s3s::Body;
use s3s::dto::{
BucketLifecycleConfiguration, DefaultRetention, ReplicationConfiguration, RestoreRequest, RestoreRequestType, RestoreStatus,
@@ -97,8 +96,14 @@ impl LifecycleSys {
}
pub async fn get(&self, bucket: &str) -> Option<BucketLifecycleConfiguration> {
let lc = get_lifecycle_config(bucket).await.expect("get_lifecycle_config err!").0;
Some(lc)
match get_lifecycle_config(bucket).await {
Ok((lc, _)) => Some(lc),
Err(err) if err == Error::ConfigNotFound => None,
Err(err) => {
warn!(bucket, error = ?err, "failed to load lifecycle config");
None
}
}
}
pub fn trace(_oi: &ObjectInfo) -> TraceFn {
@@ -471,10 +476,7 @@ impl TransitionState {
}
pub async fn init(api: Arc<ECStore>) {
let max_workers = env::var("RUSTFS_MAX_TRANSITION_WORKERS")
.ok()
.and_then(|s| s.parse::<i64>().ok())
.unwrap_or_else(|| std::cmp::min(num_cpus::get() as i64, 16));
let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
let mut n = max_workers;
let tw = 8; //globalILMConfig.getTransitionWorkers();
if tw > 0 {
@@ -569,17 +571,11 @@ impl TransitionState {
pub async fn update_workers_inner(api: Arc<ECStore>, n: i64) {
let mut n = n;
if n == 0 {
let max_workers = env::var("RUSTFS_MAX_TRANSITION_WORKERS")
.ok()
.and_then(|s| s.parse::<i64>().ok())
.unwrap_or_else(|| std::cmp::min(num_cpus::get() as i64, 16));
let max_workers = get_env_i64("RUSTFS_MAX_TRANSITION_WORKERS", std::cmp::min(num_cpus::get() as i64, 16));
n = max_workers;
}
// Allow environment override of maximum workers
let absolute_max = env::var("RUSTFS_ABSOLUTE_MAX_WORKERS")
.ok()
.and_then(|s| s.parse::<i64>().ok())
.unwrap_or(32);
let absolute_max = get_env_i64("RUSTFS_ABSOLUTE_MAX_WORKERS", 32);
n = std::cmp::min(n, absolute_max);
let mut num_workers = GLOBAL_TransitionState.num_workers.load(Ordering::SeqCst);
@@ -603,10 +599,7 @@ impl TransitionState {
}
pub async fn init_background_expiry(api: Arc<ECStore>) {
let mut workers = env::var("RUSTFS_MAX_EXPIRY_WORKERS")
.ok()
.and_then(|s| s.parse::<usize>().ok())
.unwrap_or_else(|| std::cmp::min(num_cpus::get(), 16));
let mut workers = get_env_usize("RUSTFS_MAX_EXPIRY_WORKERS", std::cmp::min(num_cpus::get(), 16));
//globalILMConfig.getExpirationWorkers()
if let Ok(env_expiration_workers) = env::var("_RUSTFS_ILM_EXPIRATION_WORKERS") {
if let Ok(num_expirations) = env_expiration_workers.parse::<usize>() {
@@ -615,10 +608,7 @@ pub async fn init_background_expiry(api: Arc<ECStore>) {
}
if workers == 0 {
workers = env::var("RUSTFS_DEFAULT_EXPIRY_WORKERS")
.ok()
.and_then(|s| s.parse::<usize>().ok())
.unwrap_or(8);
workers = get_env_usize("RUSTFS_DEFAULT_EXPIRY_WORKERS", 8);
}
//let expiry_state = GLOBAL_ExpiryStSate.write().await;
+664 -45
View File
@@ -20,8 +20,8 @@
use rustfs_filemeta::{ReplicationStatusType, VersionPurgeStatusType};
use s3s::dto::{
BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, NoncurrentVersionTransition,
ObjectLockConfiguration, ObjectLockEnabled, RestoreRequest, Transition, TransitionStorageClass,
BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleAndOperator,
NoncurrentVersionTransition, ObjectLockConfiguration, ObjectLockEnabled, RestoreRequest, Transition, TransitionStorageClass,
};
use std::cmp::Ordering;
use std::collections::HashMap;
@@ -49,6 +49,8 @@ const ERR_LIFECYCLE_INVALID_EXPIRATION_DAYS: &str = "Lifecycle expiration days m
const ERR_LIFECYCLE_INVALID_EXPIRATION_DATE_NOT_MIDNIGHT: &str = "Expiration.Date must be at midnight UTC";
const ERR_LIFECYCLE_INVALID_RULE_ID_TOO_LONG: &str = "Rule ID must be at most 255 characters";
const ERR_LIFECYCLE_INVALID_RULE_STATUS: &str = "Rule status must be either Enabled or Disabled";
const ERR_LIFECYCLE_DEL_MARKER_WITH_TAGS: &str = "Rule with DelMarkerExpiration cannot have tags based filtering";
const ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION: &str = "Rule must have at least one of Expiration, Transition, NoncurrentVersionExpiration, NoncurrentVersionTransition, or DelMarkerExpiration";
pub use rustfs_common::metrics::IlmAction;
@@ -117,23 +119,66 @@ impl RuleValidate for LifecycleRule {
}*/
fn validate(&self) -> Result<(), std::io::Error> {
/*self.validate_id()?;
self.validate_status()?;
self.validate_expiration()?;
self.validate_noncurrent_expiration()?;
self.validate_prefix_and_filter()?;
self.validate_transition()?;
self.validate_noncurrent_transition()?;
if (!self.Filter.Tag.IsEmpty() || len(self.Filter.And.Tags) != 0) && !self.delmarker_expiration.Empty() {
return errInvalidRuleDelMarkerExpiration
// Rule with DelMarkerExpiration cannot have tags based filtering
let has_tag_filter = self
.filter
.as_ref()
.map_or(false, |f| f.tag.is_some() || f.and.as_ref().and_then(|a| a.tags.as_ref()).is_some());
if has_tag_filter && self.del_marker_expiration.is_some() {
return Err(std::io::Error::other(ERR_LIFECYCLE_DEL_MARKER_WITH_TAGS));
}
// Rule must have at least one action
let has_expiration = self.expiration.is_some();
let has_transition = self.transitions.as_ref().map_or(false, |t| !t.is_empty());
let has_noncurrent_expiration = self
.noncurrent_version_expiration
.as_ref()
.and_then(|e| e.noncurrent_days)
.map_or(false, |d| d != 0);
let has_noncurrent_transition = self
.noncurrent_version_transitions
.as_ref()
.and_then(|t| t.first())
.and_then(|t| t.storage_class.as_ref())
.is_some();
let has_abort_incomplete_multipart_upload = self.abort_incomplete_multipart_upload.is_some();
let has_del_marker_expiration = self
.del_marker_expiration
.as_ref()
.and_then(|d| d.days)
.map_or(false, |d| d > 0);
if !has_expiration
&& !has_transition
&& !has_noncurrent_expiration
&& !has_noncurrent_transition
&& !has_abort_incomplete_multipart_upload
&& !has_del_marker_expiration
{
return Err(std::io::Error::other(ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION));
}
if !self.expiration.set && !self.transition.set && !self.noncurrent_version_expiration.set && !self.noncurrent_version_transitions.unwrap()[0].set && self.delmarker_expiration.Empty() {
return errXMLNotWellFormed
}*/
Ok(())
}
}
fn lifecycle_rule_prefix(rule: &LifecycleRule) -> Option<&str> {
// Prefer a non-empty legacy prefix; treat an empty legacy prefix as if it were not set
if let Some(p) = rule.prefix.as_deref() {
if !p.is_empty() {
return Some(p);
}
}
let Some(filter) = rule.filter.as_ref() else {
return None;
};
if let Some(p) = filter.prefix.as_deref() {
return Some(p);
}
filter.and.as_ref().and_then(|and| and.prefix.as_deref())
}
#[async_trait::async_trait]
pub trait Lifecycle {
async fn has_transition(&self) -> bool;
@@ -177,8 +222,11 @@ impl Lifecycle for BucketLifecycleConfiguration {
continue;
}
let rule_prefix = &rule.prefix.clone().unwrap_or_default();
if prefix.len() > 0 && rule_prefix.len() > 0 && !prefix.starts_with(rule_prefix) && !rule_prefix.starts_with(&prefix)
let rule_prefix = lifecycle_rule_prefix(rule).unwrap_or("");
if !prefix.is_empty()
&& !rule_prefix.is_empty()
&& !prefix.starts_with(rule_prefix)
&& !rule_prefix.starts_with(prefix)
{
continue;
}
@@ -297,8 +345,8 @@ impl Lifecycle for BucketLifecycleConfiguration {
if rule.status.as_str() == ExpirationStatus::DISABLED {
continue;
}
if let Some(prefix) = rule.prefix.clone() {
if !obj.name.starts_with(prefix.as_str()) {
if let Some(rule_prefix) = lifecycle_rule_prefix(rule) {
if !obj.name.starts_with(rule_prefix) {
continue;
}
}
@@ -414,43 +462,32 @@ impl Lifecycle for BucketLifecycleConfiguration {
if let Some(ref lc_rules) = self.filter_rules(obj).await {
for rule in lc_rules.iter() {
if obj.expired_object_deletemarker() {
if obj.is_latest && obj.expired_object_deletemarker() {
if let Some(expiration) = rule.expiration.as_ref() {
if let Some(expired_object_delete_marker) = expiration.expired_object_delete_marker {
events.push(Event {
action: IlmAction::DeleteVersionAction,
rule_id: rule.id.clone().unwrap_or_default(),
due: Some(now),
noncurrent_days: 0,
newer_noncurrent_versions: 0,
storage_class: "".into(),
});
break;
}
if let Some(days) = expiration.days {
let expected_expiry = expected_expiry_time(mod_time, days /*, date*/);
if now.unix_timestamp() >= expected_expiry.unix_timestamp() {
if expiration.expired_object_delete_marker.is_some_and(|v| v) {
// Preserve explicit date/days scheduling when configured.
// If only ExpiredObjectDeleteMarker=true is set, delete immediately.
let due = expiration.next_due(obj).unwrap_or(now);
if now.unix_timestamp() >= due.unix_timestamp() {
events.push(Event {
action: IlmAction::DeleteVersionAction,
rule_id: rule.id.clone().unwrap_or_default(),
due: Some(expected_expiry),
due: Some(due),
noncurrent_days: 0,
newer_noncurrent_versions: 0,
storage_class: "".into(),
});
// Stop after scheduling an expired delete-marker event.
break;
}
}
}
}
if obj.is_latest {
if let Some(ref expiration) = rule.expiration {
if let Some(expired_object_delete_marker) = expiration.expired_object_delete_marker {
if obj.delete_marker && expired_object_delete_marker {
let due = expiration.next_due(obj);
if let Some(due) = due {
// DelMarkerExpiration: expire delete marker after N days from mod_time
if obj.delete_marker {
if let Some(ref dme) = rule.del_marker_expiration {
if let Some(days) = dme.days {
if days > 0 {
let due = expected_expiry_time(mod_time, days);
if now.unix_timestamp() >= due.unix_timestamp() {
events.push(Event {
action: IlmAction::DelMarkerDeleteAllVersionsAction,
@@ -461,8 +498,8 @@ impl Lifecycle for BucketLifecycleConfiguration {
storage_class: "".into(),
});
}
continue;
}
continue;
}
}
}
@@ -694,8 +731,16 @@ impl LifecycleCalculate for LifecycleExpiration {
if !obj.is_latest || !obj.delete_marker {
return None;
}
// Check date first (date-based expiration takes priority over days).
// A zero unix timestamp means "not set" (default value) and is skipped.
if let Some(ref date) = self.date {
let expiry_date = OffsetDateTime::from(date.clone());
if expiry_date.unix_timestamp() != 0 {
return Some(expiry_date);
}
}
match self.days {
Some(days) => Some(expected_expiry_time(obj.mod_time.unwrap(), days)),
Some(days) => obj.mod_time.map(|mod_time| expected_expiry_time(mod_time, days)),
None => None,
}
}
@@ -860,10 +905,15 @@ impl Default for TransitionOptions {
#[cfg(test)]
mod tests {
use super::*;
use s3s::dto::LifecycleRuleFilter;
use serial_test::serial;
use std::sync::Arc;
#[tokio::test]
#[serial]
async fn validate_rejects_non_positive_expiration_days() {
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
@@ -871,6 +921,7 @@ mod tests {
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: None,
noncurrent_version_expiration: None,
@@ -889,8 +940,10 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn validate_accepts_positive_expiration_days() {
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
@@ -898,6 +951,7 @@ mod tests {
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: None,
noncurrent_version_expiration: None,
@@ -913,8 +967,36 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn validate_accepts_abort_incomplete_multipart_upload_only_rule() {
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: None,
abort_incomplete_multipart_upload: Some(s3s::dto::AbortIncompleteMultipartUpload {
days_after_initiation: Some(2),
}),
del_marker_expiration: None,
filter: None,
id: Some("abort-only".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: Some("test/".to_string()),
transitions: None,
}],
};
lc.validate(&ObjectLockConfiguration::default())
.await
.expect("expected validation to pass");
}
#[tokio::test]
#[serial]
async fn validate_rejects_non_midnight_expiration_date() {
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
@@ -922,6 +1004,7 @@ mod tests {
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: None,
noncurrent_version_expiration: None,
@@ -937,9 +1020,11 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn predict_expiration_selects_closest_expiry_for_put_object() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![
LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
@@ -948,6 +1033,7 @@ mod tests {
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("rule-days".to_string()),
noncurrent_version_expiration: None,
@@ -962,6 +1048,7 @@ mod tests {
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("rule-date".to_string()),
noncurrent_version_expiration: None,
@@ -988,8 +1075,10 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn validate_accepts_multiple_rules_without_ids() {
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![
LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
@@ -998,6 +1087,7 @@ mod tests {
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: None,
noncurrent_version_expiration: None,
@@ -1012,6 +1102,7 @@ mod tests {
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: None,
noncurrent_version_expiration: None,
@@ -1028,8 +1119,10 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn validate_rejects_rule_id_too_long() {
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
@@ -1037,6 +1130,7 @@ mod tests {
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("a".repeat(256)),
noncurrent_version_expiration: None,
@@ -1052,8 +1146,276 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn validate_rejects_duplicate_rule_ids() {
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![
LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(1),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("dup-rule".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
},
LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(2),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
filter: None,
id: Some("dup-rule".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
del_marker_expiration: None,
},
],
};
let err = lc.validate(&ObjectLockConfiguration::default()).await.unwrap_err();
assert_eq!(err.to_string(), ERR_LIFECYCLE_DUPLICATE_ID);
}
#[tokio::test]
#[serial]
async fn eval_inner_expires_latest_object_after_days_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(1),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("expire-days".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
}],
};
let opts = ObjectOpts {
name: "obj".to_string(),
mod_time: Some(base_time),
is_latest: true,
..Default::default()
};
let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
assert_eq!(event.action, IlmAction::DeleteAction);
assert_eq!(event.rule_id, "expire-days");
assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
}
#[tokio::test]
#[serial]
async fn eval_inner_keeps_latest_object_before_days_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(2),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("expire-days".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
}],
};
let opts = ObjectOpts {
name: "obj".to_string(),
mod_time: Some(base_time),
is_latest: true,
..Default::default()
};
let event = lc.eval_inner(&opts, base_time + Duration::hours(12), 0).await;
assert_eq!(event.action, IlmAction::NoneAction);
}
#[tokio::test]
#[serial]
async fn eval_inner_transitions_latest_object_after_days_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("transition-days".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: Some(vec![Transition {
days: Some(1),
date: None,
storage_class: Some(TransitionStorageClass::from_static("COLDTIER44")),
}]),
}],
};
let opts = ObjectOpts {
name: "obj".to_string(),
mod_time: Some(base_time),
is_latest: true,
transition_status: "".to_string(),
..Default::default()
};
let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
assert_eq!(event.action, IlmAction::TransitionAction);
assert_eq!(event.rule_id, "transition-days");
assert_eq!(event.storage_class, "COLDTIER44");
}
#[tokio::test]
#[serial]
async fn eval_inner_expires_noncurrent_version_after_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("noncurrent-expire".to_string()),
noncurrent_version_expiration: Some(s3s::dto::NoncurrentVersionExpiration {
noncurrent_days: Some(1),
newer_noncurrent_versions: None,
}),
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
}],
};
let opts = ObjectOpts {
name: "obj".to_string(),
mod_time: Some(base_time),
successor_mod_time: Some(base_time),
is_latest: false,
version_id: Some(Uuid::new_v4()),
..Default::default()
};
let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
assert_eq!(event.action, IlmAction::DeleteVersionAction);
assert_eq!(event.rule_id, "noncurrent-expire");
assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
}
#[tokio::test]
#[serial]
async fn eval_inner_transitions_noncurrent_version_after_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("noncurrent-transition".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: Some(vec![NoncurrentVersionTransition {
noncurrent_days: Some(1),
newer_noncurrent_versions: None,
storage_class: Some(TransitionStorageClass::from_static("COLDTIER44")),
}]),
prefix: None,
transitions: None,
}],
};
let opts = ObjectOpts {
name: "obj".to_string(),
mod_time: Some(base_time),
successor_mod_time: Some(base_time),
is_latest: false,
transition_status: "".to_string(),
version_id: Some(Uuid::new_v4()),
..Default::default()
};
let event = lc.eval_inner(&opts, base_time + Duration::days(2), 0).await;
assert_eq!(event.action, IlmAction::TransitionVersionAction);
assert_eq!(event.rule_id, "noncurrent-transition");
assert_eq!(event.storage_class, "COLDTIER44");
}
#[tokio::test]
#[serial]
async fn noncurrent_versions_expiration_limit_returns_configured_limits() {
let lc = Arc::new(BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("noncurrent-limit".to_string()),
noncurrent_version_expiration: Some(s3s::dto::NoncurrentVersionExpiration {
noncurrent_days: Some(7),
newer_noncurrent_versions: Some(3),
}),
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
}],
});
let opts = ObjectOpts {
name: "obj".to_string(),
mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
is_latest: false,
version_id: Some(Uuid::new_v4()),
..Default::default()
};
let event = lc.noncurrent_versions_expiration_limit(&opts).await;
assert_eq!(event.action, IlmAction::DeleteVersionAction);
assert_eq!(event.rule_id, "noncurrent-limit");
assert_eq!(event.noncurrent_days, 7);
assert_eq!(event.newer_noncurrent_versions, 3);
}
#[tokio::test]
#[serial]
async fn validate_rejects_invalid_status_case_sensitive() {
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static("enabled"),
expiration: Some(LifecycleExpiration {
@@ -1061,6 +1423,7 @@ mod tests {
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: None,
noncurrent_version_expiration: None,
@@ -1074,4 +1437,260 @@ mod tests {
assert_eq!(err.to_string(), ERR_LIFECYCLE_INVALID_RULE_STATUS);
}
#[tokio::test]
#[serial]
async fn filter_rules_respects_filter_prefix() {
let mut filter = LifecycleRuleFilter::default();
filter.prefix = Some("prefix".to_string());
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(30),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
filter: Some(filter),
id: Some("rule".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
del_marker_expiration: None,
}],
};
let match_obj = ObjectOpts {
name: "prefix/file".to_string(),
mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
is_latest: true,
..Default::default()
};
let matched = lc.filter_rules(&match_obj).await.unwrap();
assert_eq!(matched.len(), 1);
let non_match_obj = ObjectOpts {
name: "other/file".to_string(),
mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
is_latest: true,
..Default::default()
};
let not_matched = lc.filter_rules(&non_match_obj).await.unwrap();
assert_eq!(not_matched.len(), 0);
}
#[tokio::test]
#[serial]
async fn filter_rules_respects_filter_and_prefix() {
let mut filter = LifecycleRuleFilter::default();
let mut and = LifecycleRuleAndOperator::default();
and.prefix = Some("prefix".to_string());
filter.and = Some(and);
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(30),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
filter: Some(filter),
id: Some("rule-and-prefix".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
del_marker_expiration: None,
}],
};
let match_obj = ObjectOpts {
name: "prefix/file".to_string(),
mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
is_latest: true,
..Default::default()
};
let matched = lc.filter_rules(&match_obj).await.unwrap();
assert_eq!(matched.len(), 1);
let non_match_obj = ObjectOpts {
name: "other/file".to_string(),
mod_time: Some(OffsetDateTime::from_unix_timestamp(1_000_000).unwrap()),
is_latest: true,
..Default::default()
};
let not_matched = lc.filter_rules(&non_match_obj).await.unwrap();
assert_eq!(not_matched.len(), 0);
}
#[tokio::test]
#[serial]
async fn expired_object_delete_marker_requires_single_version() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(1),
expired_object_delete_marker: Some(true),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
filter: None,
id: Some("rule-expired-del-marker".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
del_marker_expiration: None,
}],
};
let opts = ObjectOpts {
name: "obj".to_string(),
mod_time: Some(base_time),
is_latest: true,
delete_marker: true,
num_versions: 2,
version_id: Some(Uuid::new_v4()),
..Default::default()
};
let now = base_time + Duration::days(2);
let event = lc.eval_inner(&opts, now, 0).await;
assert_eq!(event.action, IlmAction::NoneAction);
}
#[tokio::test]
#[serial]
async fn expired_object_delete_marker_deletes_only_delete_marker_after_due() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(1),
expired_object_delete_marker: Some(true),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
filter: None,
id: Some("rule-expired-del-marker".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
del_marker_expiration: None,
}],
};
let opts = ObjectOpts {
name: "obj".to_string(),
mod_time: Some(base_time),
is_latest: true,
delete_marker: true,
num_versions: 1,
version_id: Some(Uuid::new_v4()),
..Default::default()
};
let now = base_time + Duration::days(2);
let event = lc.eval_inner(&opts, now, 0).await;
assert_eq!(event.action, IlmAction::DeleteVersionAction);
assert_eq!(event.due, Some(expected_expiry_time(base_time, 1)));
}
#[tokio::test]
async fn expired_object_delete_marker_without_date_or_days_deletes_immediately() {
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
expired_object_delete_marker: Some(true),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
filter: None,
id: Some("rule-expired-del-marker-immediate".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
del_marker_expiration: None,
}],
};
let opts = ObjectOpts {
name: "obj".to_string(),
mod_time: Some(base_time),
is_latest: true,
delete_marker: true,
num_versions: 1,
version_id: Some(Uuid::new_v4()),
..Default::default()
};
let now = base_time + Duration::days(2);
let event = lc.eval_inner(&opts, now, 0).await;
assert_eq!(event.action, IlmAction::DeleteVersionAction);
assert_eq!(event.due, Some(now));
}
#[tokio::test]
async fn expired_object_delete_marker_date_based_not_yet_due() {
// A date-based rule that has not yet reached its expiry date must not
// trigger immediate deletion (unwrap_or(now) must not override the date).
let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
let future_date = base_time + Duration::days(10);
let lc = BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
date: Some(future_date.into()),
expired_object_delete_marker: Some(true),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
filter: None,
id: Some("rule-date-del-marker".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
del_marker_expiration: None,
}],
};
let opts = ObjectOpts {
name: "obj".to_string(),
mod_time: Some(base_time),
is_latest: true,
delete_marker: true,
num_versions: 1,
version_id: Some(Uuid::new_v4()),
..Default::default()
};
// now is before the configured date — must not schedule deletion
let now_before = base_time + Duration::days(5);
let event_before = lc.eval_inner(&opts, now_before, 0).await;
assert_eq!(event_before.action, IlmAction::NoneAction);
// now is after the configured date — must schedule deletion
let now_after = base_time + Duration::days(11);
let event_after = lc.eval_inner(&opts, now_after, 0).await;
assert_eq!(event_after.action, IlmAction::DeleteVersionAction);
assert_eq!(event_after.due, Some(future_date));
}
}
+218 -56
View File
@@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
use super::object_lock::ObjectLockApi;
use super::versioning::VersioningApi;
use super::{quota::BucketQuota, target::BucketTargets};
@@ -22,7 +23,6 @@ use crate::error::{Error, Result};
use crate::new_object_layer_fn;
use crate::store::ECStore;
use byteorder::{BigEndian, ByteOrder, LittleEndian};
use rmp_serde::Serializer as rmpSerializer;
use rustfs_policy::policy::BucketPolicy;
use s3s::dto::{
BucketLifecycleConfiguration, CORSConfiguration, NotificationConfiguration, ObjectLockConfiguration,
@@ -30,12 +30,41 @@ use s3s::dto::{
VersioningConfiguration,
};
use serde::Serializer;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::io::{Read, Write};
use std::sync::Arc;
use time::OffsetDateTime;
use tracing::error;
fn read_msgp_str<R: Read>(rd: &mut R) -> Result<String> {
let len = rmp::decode::read_str_len(rd)? as usize;
let mut buf = vec![0u8; len];
rd.read_exact(&mut buf)?;
Ok(String::from_utf8(buf)?)
}
fn read_msgp_time_value<R: Read>(rd: &mut R) -> Result<OffsetDateTime> {
let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
match marker {
rmp::Marker::Null => Ok(OffsetDateTime::UNIX_EPOCH),
rmp::Marker::Ext8 => read_msgp_ext8_time(rd),
_ => Err(Error::other(format!("expected time ext or nil, got marker: {marker:?}"))),
}
}
fn read_msgp_bin<R: Read>(rd: &mut R) -> Result<Vec<u8>> {
let len = rmp::decode::read_bin_len(rd)? as usize;
let mut buf = vec![0u8; len];
rd.read_exact(&mut buf)?;
Ok(buf)
}
fn write_bin_field<W: Write>(wr: &mut W, key: &str, val: &[u8]) -> Result<()> {
rmp::encode::write_str(wr, key)?;
rmp::encode::write_bin(wr, val)?;
Ok(())
}
pub const BUCKET_METADATA_FILE: &str = ".metadata.bin";
pub const BUCKET_METADATA_FORMAT: u16 = 1;
pub const BUCKET_METADATA_VERSION: u16 = 1;
@@ -54,8 +83,7 @@ pub const BUCKET_CORS_CONFIG: &str = "cors.xml";
pub const BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG: &str = "public-access-block.xml";
pub const BUCKET_ACL_CONFIG: &str = "bucket-acl.json";
#[derive(Debug, Deserialize, Serialize, Clone)]
#[serde(rename_all = "PascalCase", default)]
#[derive(Debug, Clone)]
pub struct BucketMetadata {
pub name: String,
pub created: OffsetDateTime,
@@ -90,36 +118,21 @@ pub struct BucketMetadata {
pub public_access_block_config_updated_at: OffsetDateTime,
pub bucket_acl_config_updated_at: OffsetDateTime,
#[serde(skip)]
pub new_field_updated_at: OffsetDateTime,
#[serde(skip)]
pub policy_config: Option<BucketPolicy>,
#[serde(skip)]
pub notification_config: Option<NotificationConfiguration>,
#[serde(skip)]
pub lifecycle_config: Option<BucketLifecycleConfiguration>,
#[serde(skip)]
pub object_lock_config: Option<ObjectLockConfiguration>,
#[serde(skip)]
pub versioning_config: Option<VersioningConfiguration>,
#[serde(skip)]
pub sse_config: Option<ServerSideEncryptionConfiguration>,
#[serde(skip)]
pub tagging_config: Option<Tagging>,
#[serde(skip)]
pub quota_config: Option<BucketQuota>,
#[serde(skip)]
pub replication_config: Option<ReplicationConfiguration>,
#[serde(skip)]
pub bucket_target_config: Option<BucketTargets>,
#[serde(skip)]
pub bucket_target_config_meta: Option<HashMap<String, String>>,
#[serde(skip)]
pub cors_config: Option<CORSConfiguration>,
#[serde(skip)]
pub public_access_block_config: Option<PublicAccessBlockConfiguration>,
#[serde(skip)]
pub bucket_acl_config: Option<String>,
}
@@ -198,17 +211,137 @@ impl BucketMetadata {
self.lock_enabled || (self.versioning_config.as_ref().is_some_and(|v| v.enabled()))
}
/// Decode from msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
pub fn decode_from<R: Read>(&mut self, rd: &mut R) -> Result<()> {
let mut fields = rmp::decode::read_map_len(rd)?;
*self = Self::default();
while fields > 0 {
fields -= 1;
let key_len = rmp::decode::read_str_len(rd)?;
let mut key_buf = vec![0u8; key_len as usize];
rd.read_exact(&mut key_buf)?;
let key = String::from_utf8(key_buf)?;
match key.as_str() {
"Name" => self.name = read_msgp_str(rd)?,
"Created" => self.created = read_msgp_time_value(rd)?,
"LockEnabled" => self.lock_enabled = rmp::decode::read_bool(rd)?,
"PolicyConfigJSON" => self.policy_config_json = read_msgp_bin(rd)?,
"NotificationConfigXML" => self.notification_config_xml = read_msgp_bin(rd)?,
"LifecycleConfigXML" => self.lifecycle_config_xml = read_msgp_bin(rd)?,
"ObjectLockConfigXML" => self.object_lock_config_xml = read_msgp_bin(rd)?,
"VersioningConfigXML" => self.versioning_config_xml = read_msgp_bin(rd)?,
"EncryptionConfigXML" => self.encryption_config_xml = read_msgp_bin(rd)?,
"TaggingConfigXML" => self.tagging_config_xml = read_msgp_bin(rd)?,
"QuotaConfigJSON" => self.quota_config_json = read_msgp_bin(rd)?,
"ReplicationConfigXML" => self.replication_config_xml = read_msgp_bin(rd)?,
"BucketTargetsConfigJSON" => self.bucket_targets_config_json = read_msgp_bin(rd)?,
"BucketTargetsConfigMetaJSON" => self.bucket_targets_config_meta_json = read_msgp_bin(rd)?,
"PolicyConfigUpdatedAt" => self.policy_config_updated_at = read_msgp_time_value(rd)?,
"ObjectLockConfigUpdatedAt" => self.object_lock_config_updated_at = read_msgp_time_value(rd)?,
"EncryptionConfigUpdatedAt" => self.encryption_config_updated_at = read_msgp_time_value(rd)?,
"TaggingConfigUpdatedAt" => self.tagging_config_updated_at = read_msgp_time_value(rd)?,
"QuotaConfigUpdatedAt" => self.quota_config_updated_at = read_msgp_time_value(rd)?,
"ReplicationConfigUpdatedAt" => self.replication_config_updated_at = read_msgp_time_value(rd)?,
"VersioningConfigUpdatedAt" => self.versioning_config_updated_at = read_msgp_time_value(rd)?,
"LifecycleConfigUpdatedAt" => self.lifecycle_config_updated_at = read_msgp_time_value(rd)?,
"NotificationConfigUpdatedAt" => self.notification_config_updated_at = read_msgp_time_value(rd)?,
"BucketTargetsConfigUpdatedAt" => self.bucket_targets_config_updated_at = read_msgp_time_value(rd)?,
"BucketTargetsConfigMetaUpdatedAt" => self.bucket_targets_config_meta_updated_at = read_msgp_time_value(rd)?,
"CorsConfigXML" => self.cors_config_xml = read_msgp_bin(rd)?,
"PublicAccessBlockConfigXML" => self.public_access_block_config_xml = read_msgp_bin(rd)?,
"BucketAclConfigJSON" => self.bucket_acl_config_json = read_msgp_bin(rd)?,
"CorsConfigUpdatedAt" => self.cors_config_updated_at = read_msgp_time_value(rd)?,
"PublicAccessBlockConfigUpdatedAt" => self.public_access_block_config_updated_at = read_msgp_time_value(rd)?,
"BucketAclConfigUpdatedAt" => self.bucket_acl_config_updated_at = read_msgp_time_value(rd)?,
other => {
tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
skip_msgp_value(rd)?;
}
}
}
Ok(())
}
/// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
// Map size: MinIO fields (25) + RustFS extensions (6)
let map_len: u32 = 31;
rmp::encode::write_map_len(wr, map_len)?;
// MinIO field order (same as Go struct)
rmp::encode::write_str(wr, "Name")?;
rmp::encode::write_str(wr, &self.name)?;
rmp::encode::write_str(wr, "Created")?;
write_msgp_time(wr, self.created)?;
rmp::encode::write_str(wr, "LockEnabled")?;
rmp::encode::write_bool(wr, self.lock_enabled)?;
write_bin_field(wr, "PolicyConfigJSON", &self.policy_config_json)?;
write_bin_field(wr, "NotificationConfigXML", &self.notification_config_xml)?;
write_bin_field(wr, "LifecycleConfigXML", &self.lifecycle_config_xml)?;
write_bin_field(wr, "ObjectLockConfigXML", &self.object_lock_config_xml)?;
write_bin_field(wr, "VersioningConfigXML", &self.versioning_config_xml)?;
write_bin_field(wr, "EncryptionConfigXML", &self.encryption_config_xml)?;
write_bin_field(wr, "TaggingConfigXML", &self.tagging_config_xml)?;
write_bin_field(wr, "QuotaConfigJSON", &self.quota_config_json)?;
write_bin_field(wr, "ReplicationConfigXML", &self.replication_config_xml)?;
write_bin_field(wr, "BucketTargetsConfigJSON", &self.bucket_targets_config_json)?;
write_bin_field(wr, "BucketTargetsConfigMetaJSON", &self.bucket_targets_config_meta_json)?;
rmp::encode::write_str(wr, "PolicyConfigUpdatedAt")?;
write_msgp_time(wr, self.policy_config_updated_at)?;
rmp::encode::write_str(wr, "ObjectLockConfigUpdatedAt")?;
write_msgp_time(wr, self.object_lock_config_updated_at)?;
rmp::encode::write_str(wr, "EncryptionConfigUpdatedAt")?;
write_msgp_time(wr, self.encryption_config_updated_at)?;
rmp::encode::write_str(wr, "TaggingConfigUpdatedAt")?;
write_msgp_time(wr, self.tagging_config_updated_at)?;
rmp::encode::write_str(wr, "QuotaConfigUpdatedAt")?;
write_msgp_time(wr, self.quota_config_updated_at)?;
rmp::encode::write_str(wr, "ReplicationConfigUpdatedAt")?;
write_msgp_time(wr, self.replication_config_updated_at)?;
rmp::encode::write_str(wr, "VersioningConfigUpdatedAt")?;
write_msgp_time(wr, self.versioning_config_updated_at)?;
rmp::encode::write_str(wr, "LifecycleConfigUpdatedAt")?;
write_msgp_time(wr, self.lifecycle_config_updated_at)?;
rmp::encode::write_str(wr, "NotificationConfigUpdatedAt")?;
write_msgp_time(wr, self.notification_config_updated_at)?;
rmp::encode::write_str(wr, "BucketTargetsConfigUpdatedAt")?;
write_msgp_time(wr, self.bucket_targets_config_updated_at)?;
rmp::encode::write_str(wr, "BucketTargetsConfigMetaUpdatedAt")?;
write_msgp_time(wr, self.bucket_targets_config_meta_updated_at)?;
// RustFS extensions
write_bin_field(wr, "CorsConfigXML", &self.cors_config_xml)?;
write_bin_field(wr, "PublicAccessBlockConfigXML", &self.public_access_block_config_xml)?;
write_bin_field(wr, "BucketAclConfigJSON", &self.bucket_acl_config_json)?;
rmp::encode::write_str(wr, "CorsConfigUpdatedAt")?;
write_msgp_time(wr, self.cors_config_updated_at)?;
rmp::encode::write_str(wr, "PublicAccessBlockConfigUpdatedAt")?;
write_msgp_time(wr, self.public_access_block_config_updated_at)?;
rmp::encode::write_str(wr, "BucketAclConfigUpdatedAt")?;
write_msgp_time(wr, self.bucket_acl_config_updated_at)?;
Ok(())
}
pub fn marshal_msg(&self) -> Result<Vec<u8>> {
let mut buf = Vec::new();
self.serialize(&mut rmpSerializer::new(&mut buf).with_struct_map())?;
self.encode_to(&mut buf)?;
Ok(buf)
}
pub fn unmarshal(buf: &[u8]) -> Result<Self> {
let t: BucketMetadata = rmp_serde::from_slice(buf)?;
Ok(t)
let mut bm = Self::default();
let mut cur = std::io::Cursor::new(buf);
bm.decode_from(&mut cur)?;
Ok(bm)
}
pub fn check_header(buf: &[u8]) -> Result<()> {
@@ -382,50 +515,80 @@ impl BucketMetadata {
}
fn parse_all_configs(&mut self, _api: Arc<ECStore>) -> Result<()> {
self.parse_policy_config()?;
if !self.notification_config_xml.is_empty() {
self.notification_config = Some(deserialize::<NotificationConfiguration>(&self.notification_config_xml)?);
if let Err(e) = self.parse_policy_config() {
tracing::warn!(bucket = %self.name, config = "policy", error = %e, "parse_all_configs: failed to parse");
}
if !self.lifecycle_config_xml.is_empty() {
self.lifecycle_config = Some(deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml)?);
if !self.notification_config_xml.is_empty()
&& let Err(e) = deserialize::<NotificationConfiguration>(&self.notification_config_xml)
.map(|c| self.notification_config = Some(c))
{
tracing::warn!(bucket = %self.name, config = "notification", error = %e, "parse_all_configs: failed to parse");
}
if !self.object_lock_config_xml.is_empty() {
self.object_lock_config = Some(deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml)?);
if !self.lifecycle_config_xml.is_empty()
&& let Err(e) =
deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
{
tracing::warn!(bucket = %self.name, config = "lifecycle", error = %e, "parse_all_configs: failed to parse");
}
if !self.versioning_config_xml.is_empty() {
self.versioning_config = Some(deserialize::<VersioningConfiguration>(&self.versioning_config_xml)?);
if !self.object_lock_config_xml.is_empty()
&& let Err(e) =
deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml).map(|c| self.object_lock_config = Some(c))
{
tracing::warn!(bucket = %self.name, config = "object_lock", error = %e, "parse_all_configs: failed to parse");
}
if !self.encryption_config_xml.is_empty() {
self.sse_config = Some(deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml)?);
if !self.versioning_config_xml.is_empty()
&& let Err(e) =
deserialize::<VersioningConfiguration>(&self.versioning_config_xml).map(|c| self.versioning_config = Some(c))
{
tracing::warn!(bucket = %self.name, config = "versioning", error = %e, "parse_all_configs: failed to parse");
}
if !self.tagging_config_xml.is_empty() {
self.tagging_config = Some(deserialize::<Tagging>(&self.tagging_config_xml)?);
if !self.encryption_config_xml.is_empty()
&& let Err(e) =
deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml).map(|c| self.sse_config = Some(c))
{
tracing::warn!(bucket = %self.name, config = "encryption", error = %e, "parse_all_configs: failed to parse");
}
if !self.quota_config_json.is_empty() {
self.quota_config = Some(serde_json::from_slice(&self.quota_config_json)?);
if !self.tagging_config_xml.is_empty()
&& let Err(e) = deserialize::<Tagging>(&self.tagging_config_xml).map(|c| self.tagging_config = Some(c))
{
tracing::warn!(bucket = %self.name, config = "tagging", error = %e, "parse_all_configs: failed to parse");
}
if !self.replication_config_xml.is_empty() {
self.replication_config = Some(deserialize::<ReplicationConfiguration>(&self.replication_config_xml)?);
if !self.quota_config_json.is_empty()
&& let Err(e) = serde_json::from_slice(&self.quota_config_json).map(|c| self.quota_config = Some(c))
{
tracing::warn!(bucket = %self.name, config = "quota", error = %e, "parse_all_configs: failed to parse");
}
if !self.replication_config_xml.is_empty()
&& let Err(e) =
deserialize::<ReplicationConfiguration>(&self.replication_config_xml).map(|c| self.replication_config = Some(c))
{
tracing::warn!(bucket = %self.name, config = "replication", error = %e, "parse_all_configs: failed to parse");
}
//let temp = self.bucket_targets_config_json.clone();
if !self.bucket_targets_config_json.is_empty() {
let bucket_targets: BucketTargets = serde_json::from_slice(&self.bucket_targets_config_json)?;
self.bucket_target_config = Some(bucket_targets);
if let Err(e) = serde_json::from_slice::<BucketTargets>(&self.bucket_targets_config_json)
.map(|t| self.bucket_target_config = Some(t))
{
tracing::warn!(bucket = %self.name, config = "bucket_targets", error = %e, "parse_all_configs: failed to parse");
self.bucket_target_config = Some(BucketTargets::default());
}
} else {
self.bucket_target_config = Some(BucketTargets::default())
self.bucket_target_config = Some(BucketTargets::default());
}
if !self.cors_config_xml.is_empty() {
self.cors_config = Some(deserialize::<CORSConfiguration>(&self.cors_config_xml)?);
if !self.cors_config_xml.is_empty()
&& let Err(e) = deserialize::<CORSConfiguration>(&self.cors_config_xml).map(|c| self.cors_config = Some(c))
{
tracing::warn!(bucket = %self.name, config = "cors", error = %e, "parse_all_configs: failed to parse");
}
if !self.public_access_block_config_xml.is_empty() {
self.public_access_block_config =
Some(deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)?);
if !self.public_access_block_config_xml.is_empty()
&& let Err(e) = deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)
.map(|c| self.public_access_block_config = Some(c))
{
tracing::warn!(bucket = %self.name, config = "public_access_block", error = %e, "parse_all_configs: failed to parse");
}
if !self.bucket_acl_config_json.is_empty() {
let acl = String::from_utf8(self.bucket_acl_config_json.clone())
.map_err(|e| Error::other(format!("invalid UTF-8 in bucket ACL: {}", e)))?;
self.bucket_acl_config = Some(acl);
if !self.bucket_acl_config_json.is_empty()
&& let Err(e) = String::from_utf8(self.bucket_acl_config_json.clone()).map(|acl| self.bucket_acl_config = Some(acl))
{
tracing::warn!(bucket = %self.name, config = "bucket_acl", error = %e, "parse_all_configs: failed to parse");
}
Ok(())
@@ -478,7 +641,6 @@ async fn read_bucket_metadata(api: Arc<ECStore>, bucket: &str) -> Result<BucketM
Ok(bm)
}
fn _write_time<S>(t: &OffsetDateTime, s: S) -> std::result::Result<S::Ok, S::Error>
where
S: Serializer,
+8 -6
View File
@@ -360,12 +360,14 @@ impl BucketMetadataSys {
};
if !meta.lifecycle_config_xml.is_empty() {
let cfg = deserialize::<BucketLifecycleConfiguration>(&meta.lifecycle_config_xml)?;
// TODO: FIXME:
// for _v in cfg.rules.iter() {
// break;
// }
if let Some(_v) = cfg.rules.first() {}
if let Ok(cfg) = deserialize::<BucketLifecycleConfiguration>(&meta.lifecycle_config_xml) {
if let Some(_v) = cfg.rules.first() {}
} else {
tracing::warn!(
bucket = %bucket,
"delete: failed to parse lifecycle config XML"
);
}
}
// TODO: other lifecycle handle
+261
View File
@@ -0,0 +1,261 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::metadata::BucketMetadata;
use time::OffsetDateTime;
/// Full BucketMetadata hex (all fields populated).
const TEST_BUCKET_METADATA_HEX: &str = "de0019a44e616d65b27275737466732d636f6d7061742d74657374a743726561746564c70c050000000065920080075bcd15ab4c6f636b456e61626c6564c3b0506f6c696379436f6e6669674a534f4ec4907b2256657273696f6e223a22323031322d31302d3137222c2253746174656d656e74223a5b7b22456666656374223a22416c6c6f77222c225072696e636970616c223a222a222c22416374696f6e223a2273333a4765744f626a656374222c225265736f75726365223a2261726e3a6177733a73333a3a3a7275737466732d636f6d7061742d746573742f2a227d5d7db54e6f74696669636174696f6e436f6e666967584d4cc4963c4e6f74696669636174696f6e436f6e66696775726174696f6e3e3c436c6f75645761746368436f6e66696775726174696f6e3e3c49643e6e313c2f49643e3c4576656e743e73333a4f626a656374437265617465643a2a3c2f4576656e743e3c2f436c6f75645761746368436f6e66696775726174696f6e3e3c2f4e6f74696669636174696f6e436f6e66696775726174696f6e3eb24c6966656379636c65436f6e666967584d4cc48c3c4c6966656379636c65436f6e66696775726174696f6e3e3c52756c653e3c49443e72756c65313c2f49443e3c5374617475733e456e61626c65643c2f5374617475733e3c45787069726174696f6e3e3c446179733e33303c2f446179733e3c2f45787069726174696f6e3e3c2f52756c653e3c2f4c6966656379636c65436f6e66696775726174696f6e3eb34f626a6563744c6f636b436f6e666967584d4cc4b83c4f626a6563744c6f636b436f6e66696775726174696f6e3e3c4f626a6563744c6f636b456e61626c65643e456e61626c65643c2f4f626a6563744c6f636b456e61626c65643e3c52756c653e3c44656661756c74526574656e74696f6e3e3c4d6f64653e474f5645524e414e43453c2f4d6f64653e3c446179733e373c2f446179733e3c2f44656661756c74526574656e74696f6e3e3c2f52756c653e3c2f4f626a6563744c6f636b436f6e66696775726174696f6e3eb356657273696f6e696e67436f6e666967584d4cc44b3c56657273696f6e696e67436f6e66696775726174696f6e3e3c5374617475733e456e61626c65643c2f5374617475733e3c2f56657273696f6e696e67436f6e66696775726174696f6e3eb3456e6372797074696f6e436f6e666967584d4cc4c03c53657276657253696465456e6372797074696f6e436f6e66696775726174696f6e3e3c52756c653e3c4170706c7953657276657253696465456e6372797074696f6e427944656661756c743e3c535345416c676f726974686d3e4145533235363c2f535345416c676f726974686d3e3c2f4170706c7953657276657253696465456e6372797074696f6e427944656661756c743e3c2f52756c653e3c2f53657276657253696465456e6372797074696f6e436f6e66696775726174696f6e3eb054616767696e67436f6e666967584d4cc4503c54616767696e673e3c5461675365743e3c5461673e3c4b65793e456e763c2f4b65793e3c56616c75653e546573743c2f56616c75653e3c2f5461673e3c2f5461675365743e3c2f54616767696e673eaf51756f7461436f6e6669674a534f4ec4707b2271756f7461223a313037333734313832342c2271756f74615f74797065223a2248617264222c22637265617465645f6174223a22323032342d30312d30315430303a30303a30305a222c22757064617465645f6174223a22323032342d30312d30315430303a30303a30305a227db45265706c69636174696f6e436f6e666967584d4cc4e73c5265706c69636174696f6e436f6e66696775726174696f6e3e3c526f6c653e61726e3a6177733a69616d3a3a3132333435363738393031323a726f6c652f7265706c3c2f526f6c653e3c52756c653e3c49443e72313c2f49443e3c5374617475733e456e61626c65643c2f5374617475733e3c5072656669783e646f632f3c2f5072656669783e3c44657374696e6174696f6e3e3c4275636b65743e61726e3a6177733a73333a3a3a646573743c2f4275636b65743e3c2f44657374696e6174696f6e3e3c2f52756c653e3c2f5265706c69636174696f6e436f6e66696775726174696f6e3eb74275636b657454617267657473436f6e6669674a534f4ec4535b7b22656e64706f696e74223a22687474703a2f2f7461726765742e6578616d706c652e636f6d222c227461726765744275636b6574223a227462222c22726567696f6e223a2275732d656173742d31227d5dbb4275636b657454617267657473436f6e6669674d6574614a534f4ec42d7b227265706c69636174696f6e4964223a227265706c2d31222c2273796e634d6f6465223a226173796e63227db5506f6c696379436f6e666967557064617465644174c70c050000000065a5022000000000b94f626a6563744c6f636b436f6e666967557064617465644174c70c050000000065a5022000000000b9456e6372797074696f6e436f6e666967557064617465644174c70c050000000065a5022000000000b654616767696e67436f6e666967557064617465644174c70c050000000065a5022000000000b451756f7461436f6e666967557064617465644174c70c050000000065a5022000000000ba5265706c69636174696f6e436f6e666967557064617465644174c70c050000000065a5022000000000b956657273696f6e696e67436f6e666967557064617465644174c70c050000000065a5022000000000b84c6966656379636c65436f6e666967557064617465644174c70c050000000065a5022000000000bb4e6f74696669636174696f6e436f6e666967557064617465644174c70c050000000065a5022000000000bc4275636b657454617267657473436f6e666967557064617465644174c70c050000000065a5022000000000d9204275636b657454617267657473436f6e6669674d657461557064617465644174c70c050000000065a5022000000000";
#[tokio::test]
async fn marshal_msg() {
let bm = BucketMetadata::new("dada");
let buf = bm.marshal_msg().unwrap();
let new = BucketMetadata::unmarshal(&buf).unwrap();
assert_eq!(bm.name, new.name);
}
/// Verifies that serialized time uses msgp ext type 5.
#[tokio::test]
async fn marshal_msg_uses_time_format() {
let mut bm = BucketMetadata::new("test-bucket");
bm.created = OffsetDateTime::from_unix_timestamp(1704067200).unwrap(); // 2024-01-01 00:00:00 UTC
let buf = bm.marshal_msg().unwrap();
// msgp uses ext8 (0xc7), len 12, type 5 for time
assert!(
buf.windows(3).any(|w| w == [0xc7, 0x0c, 0x05]),
"serialized data should contain msgp time ext (0xc7 0x0c 0x05)"
);
}
#[tokio::test]
async fn unmarshal_test_bucket_metadata() {
use faster_hex::hex_decode;
let mut bytes = vec![0u8; TEST_BUCKET_METADATA_HEX.len() / 2];
hex_decode(TEST_BUCKET_METADATA_HEX.as_bytes(), &mut bytes).expect("valid hex");
let bm = BucketMetadata::unmarshal(&bytes).expect("RustFS must unmarshal MinIO format");
assert_eq!(bm.name, "rustfs-compat-test");
assert_eq!(bm.created.unix_timestamp(), 1704067200);
assert_eq!(bm.created.nanosecond(), 123456789);
assert!(bm.lock_enabled);
assert!(!bm.policy_config_json.is_empty());
assert!(bm.policy_config_json.starts_with(b"{\"Version\""));
assert!(!bm.notification_config_xml.is_empty());
assert!(bm.notification_config_xml.starts_with(b"<Notification"));
assert!(!bm.lifecycle_config_xml.is_empty());
assert!(bm.lifecycle_config_xml.starts_with(b"<Lifecycle"));
assert!(!bm.object_lock_config_xml.is_empty());
assert!(bm.object_lock_config_xml.starts_with(b"<ObjectLock"));
assert!(!bm.versioning_config_xml.is_empty());
assert!(bm.versioning_config_xml.starts_with(b"<Versioning"));
assert!(!bm.encryption_config_xml.is_empty());
assert!(bm.encryption_config_xml.starts_with(b"<ServerSide"));
assert!(!bm.tagging_config_xml.is_empty());
assert!(bm.tagging_config_xml.starts_with(b"<Tagging"));
assert!(!bm.quota_config_json.is_empty());
assert!(bm.quota_config_json.starts_with(b"{\"quota\""));
assert!(!bm.replication_config_xml.is_empty());
assert!(bm.replication_config_xml.starts_with(b"<Replication"));
assert!(!bm.bucket_targets_config_json.is_empty());
assert!(bm.bucket_targets_config_json.starts_with(b"[{"));
assert!(!bm.bucket_targets_config_meta_json.is_empty());
assert!(bm.bucket_targets_config_meta_json.starts_with(b"{\"replication"));
let updated_sec = 1705312800; // 2024-01-15 12:00:00 UTC
assert_eq!(bm.policy_config_updated_at.unix_timestamp(), updated_sec);
assert_eq!(bm.object_lock_config_updated_at.unix_timestamp(), updated_sec);
assert_eq!(bm.encryption_config_updated_at.unix_timestamp(), updated_sec);
assert_eq!(bm.tagging_config_updated_at.unix_timestamp(), updated_sec);
assert_eq!(bm.quota_config_updated_at.unix_timestamp(), updated_sec);
assert_eq!(bm.replication_config_updated_at.unix_timestamp(), updated_sec);
assert_eq!(bm.versioning_config_updated_at.unix_timestamp(), updated_sec);
assert_eq!(bm.lifecycle_config_updated_at.unix_timestamp(), updated_sec);
assert_eq!(bm.notification_config_updated_at.unix_timestamp(), updated_sec);
assert_eq!(bm.bucket_targets_config_updated_at.unix_timestamp(), updated_sec);
assert_eq!(bm.bucket_targets_config_meta_updated_at.unix_timestamp(), updated_sec);
assert!(bm.cors_config_xml.is_empty());
assert!(bm.public_access_block_config_xml.is_empty());
assert!(bm.bucket_acl_config_json.is_empty());
}
#[tokio::test]
async fn marshal_msg_complete_example() {
// Create a complete BucketMetadata with various configurations
let mut bm = BucketMetadata::new("test-bucket");
// Set creation time to current time
bm.created = OffsetDateTime::now_utc();
bm.lock_enabled = true;
// Add policy configuration
let policy_json = r#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":"arn:aws:s3:::test-bucket/*"}]}"#;
bm.policy_config_json = policy_json.as_bytes().to_vec();
bm.policy_config_updated_at = OffsetDateTime::now_utc();
// Add lifecycle configuration
let lifecycle_xml = r#"<LifecycleConfiguration><Rule><ID>rule1</ID><Status>Enabled</Status><Expiration><Days>30</Days></Expiration></Rule></LifecycleConfiguration>"#;
bm.lifecycle_config_xml = lifecycle_xml.as_bytes().to_vec();
bm.lifecycle_config_updated_at = OffsetDateTime::now_utc();
// Add versioning configuration
let versioning_xml = r#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#;
bm.versioning_config_xml = versioning_xml.as_bytes().to_vec();
bm.versioning_config_updated_at = OffsetDateTime::now_utc();
// Add encryption configuration
let encryption_xml = r#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#;
bm.encryption_config_xml = encryption_xml.as_bytes().to_vec();
bm.encryption_config_updated_at = OffsetDateTime::now_utc();
// Add tagging configuration
let tagging_xml = r#"<Tagging><TagSet><Tag><Key>Environment</Key><Value>Test</Value></Tag><Tag><Key>Owner</Key><Value>RustFS</Value></Tag></TagSet></Tagging>"#;
bm.tagging_config_xml = tagging_xml.as_bytes().to_vec();
bm.tagging_config_updated_at = OffsetDateTime::now_utc();
// Add quota configuration
let quota_json =
r#"{"quota":1073741824,"quota_type":"Hard","created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#; // 1GB quota
bm.quota_config_json = quota_json.as_bytes().to_vec();
bm.quota_config_updated_at = OffsetDateTime::now_utc();
// Add object lock configuration
let object_lock_xml = r#"<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled><Rule><DefaultRetention><Mode>GOVERNANCE</Mode><Days>7</Days></DefaultRetention></Rule></ObjectLockConfiguration>"#;
bm.object_lock_config_xml = object_lock_xml.as_bytes().to_vec();
bm.object_lock_config_updated_at = OffsetDateTime::now_utc();
// Add notification configuration
let notification_xml = r#"<NotificationConfiguration><CloudWatchConfiguration><Id>notification1</Id><Event>s3:ObjectCreated:*</Event><CloudWatchConfiguration><LogGroupName>test-log-group</LogGroupName></CloudWatchConfiguration></CloudWatchConfiguration></NotificationConfiguration>"#;
bm.notification_config_xml = notification_xml.as_bytes().to_vec();
bm.notification_config_updated_at = OffsetDateTime::now_utc();
// Add replication configuration
let replication_xml = r#"<ReplicationConfiguration><Role>arn:aws:iam::123456789012:role/replication-role</Role><Rule><ID>rule1</ID><Status>Enabled</Status><Prefix>documents/</Prefix><Destination><Bucket>arn:aws:s3:::destination-bucket</Bucket></Destination></Rule></ReplicationConfiguration>"#;
bm.replication_config_xml = replication_xml.as_bytes().to_vec();
bm.replication_config_updated_at = OffsetDateTime::now_utc();
// Add bucket targets configuration
let bucket_targets_json = r#"[{"endpoint":"http://target1.example.com","credentials":{"accessKey":"key1","secretKey":"secret1"},"targetBucket":"target-bucket-1","region":"us-east-1"},{"endpoint":"http://target2.example.com","credentials":{"accessKey":"key2","secretKey":"secret2"},"targetBucket":"target-bucket-2","region":"us-west-2"}]"#;
bm.bucket_targets_config_json = bucket_targets_json.as_bytes().to_vec();
bm.bucket_targets_config_updated_at = OffsetDateTime::now_utc();
// Add bucket targets meta configuration
let bucket_targets_meta_json = r#"{"replicationId":"repl-123","syncMode":"async","bandwidth":"100MB"}"#;
bm.bucket_targets_config_meta_json = bucket_targets_meta_json.as_bytes().to_vec();
bm.bucket_targets_config_meta_updated_at = OffsetDateTime::now_utc();
// Add public access block configuration
let public_access_block_xml = r#"<PublicAccessBlockConfiguration><BlockPublicAcls>true</BlockPublicAcls><IgnorePublicAcls>true</IgnorePublicAcls><BlockPublicPolicy>true</BlockPublicPolicy><RestrictPublicBuckets>false</RestrictPublicBuckets></PublicAccessBlockConfiguration>"#;
bm.public_access_block_config_xml = public_access_block_xml.as_bytes().to_vec();
bm.public_access_block_config_updated_at = OffsetDateTime::now_utc();
let bucket_acl = r#"{"owner":{"id":"rustfsadmin","display_name":"RustFS Tester"},"grants":[{"grantee":{"grantee_type":"CanonicalUser","id":"rustfsadmin","display_name":"RustFS Tester","uri":null,"email_address":null},"permission":"FULL_CONTROL"}]}"#;
bm.bucket_acl_config_json = bucket_acl.as_bytes().to_vec();
bm.bucket_acl_config_updated_at = OffsetDateTime::now_utc();
// Test serialization
let buf = bm.marshal_msg().unwrap();
assert!(!buf.is_empty(), "Serialized buffer should not be empty");
// Test deserialization
let deserialized_bm = BucketMetadata::unmarshal(&buf).unwrap();
// Verify all fields are correctly serialized and deserialized
assert_eq!(bm.name, deserialized_bm.name);
assert_eq!(bm.created.unix_timestamp(), deserialized_bm.created.unix_timestamp());
assert_eq!(bm.lock_enabled, deserialized_bm.lock_enabled);
// Verify configuration data
assert_eq!(bm.policy_config_json, deserialized_bm.policy_config_json);
assert_eq!(bm.lifecycle_config_xml, deserialized_bm.lifecycle_config_xml);
assert_eq!(bm.versioning_config_xml, deserialized_bm.versioning_config_xml);
assert_eq!(bm.encryption_config_xml, deserialized_bm.encryption_config_xml);
assert_eq!(bm.tagging_config_xml, deserialized_bm.tagging_config_xml);
assert_eq!(bm.quota_config_json, deserialized_bm.quota_config_json);
assert_eq!(bm.public_access_block_config_xml, deserialized_bm.public_access_block_config_xml);
assert_eq!(bm.bucket_acl_config_json, deserialized_bm.bucket_acl_config_json);
assert_eq!(bm.object_lock_config_xml, deserialized_bm.object_lock_config_xml);
assert_eq!(bm.notification_config_xml, deserialized_bm.notification_config_xml);
assert_eq!(bm.replication_config_xml, deserialized_bm.replication_config_xml);
assert_eq!(bm.bucket_targets_config_json, deserialized_bm.bucket_targets_config_json);
assert_eq!(bm.bucket_targets_config_meta_json, deserialized_bm.bucket_targets_config_meta_json);
// Verify timestamps (comparing unix timestamps to avoid precision issues)
assert_eq!(
bm.policy_config_updated_at.unix_timestamp(),
deserialized_bm.policy_config_updated_at.unix_timestamp()
);
assert_eq!(
bm.lifecycle_config_updated_at.unix_timestamp(),
deserialized_bm.lifecycle_config_updated_at.unix_timestamp()
);
assert_eq!(
bm.versioning_config_updated_at.unix_timestamp(),
deserialized_bm.versioning_config_updated_at.unix_timestamp()
);
assert_eq!(
bm.encryption_config_updated_at.unix_timestamp(),
deserialized_bm.encryption_config_updated_at.unix_timestamp()
);
assert_eq!(
bm.tagging_config_updated_at.unix_timestamp(),
deserialized_bm.tagging_config_updated_at.unix_timestamp()
);
assert_eq!(
bm.quota_config_updated_at.unix_timestamp(),
deserialized_bm.quota_config_updated_at.unix_timestamp()
);
assert_eq!(
bm.object_lock_config_updated_at.unix_timestamp(),
deserialized_bm.object_lock_config_updated_at.unix_timestamp()
);
assert_eq!(
bm.notification_config_updated_at.unix_timestamp(),
deserialized_bm.notification_config_updated_at.unix_timestamp()
);
assert_eq!(
bm.replication_config_updated_at.unix_timestamp(),
deserialized_bm.replication_config_updated_at.unix_timestamp()
);
assert_eq!(
bm.bucket_targets_config_updated_at.unix_timestamp(),
deserialized_bm.bucket_targets_config_updated_at.unix_timestamp()
);
assert_eq!(
bm.bucket_targets_config_meta_updated_at.unix_timestamp(),
deserialized_bm.bucket_targets_config_meta_updated_at.unix_timestamp()
);
// Test that the serialized data contains expected content
let buf_str = String::from_utf8_lossy(&buf);
assert!(buf_str.contains("test-bucket"), "Serialized data should contain bucket name");
// Verify the buffer size is reasonable (should be larger due to all the config data)
assert!(buf.len() > 1000, "Buffer should be substantial in size due to all configurations");
println!("✅ Complete BucketMetadata serialization test passed");
println!(" - Bucket name: {}", deserialized_bm.name);
println!(" - Lock enabled: {}", deserialized_bm.lock_enabled);
println!(" - Policy config size: {} bytes", deserialized_bm.policy_config_json.len());
println!(" - Lifecycle config size: {} bytes", deserialized_bm.lifecycle_config_xml.len());
println!(" - Serialized buffer size: {} bytes", buf.len());
}
+420
View File
@@ -0,0 +1,420 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Migration of bucket metadata and IAM config from legacy format to RustFS format.
use crate::bucket::metadata::BUCKET_METADATA_FILE;
use crate::bucket::replication::{decode_resync_file, encode_resync_file};
use crate::disk::{BUCKET_META_PREFIX, MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
use crate::store_api::{BucketOptions, ObjectOptions, PutObjReader, StorageAPI};
use http::HeaderMap;
use rustfs_policy::auth::UserIdentity;
use rustfs_policy::policy::PolicyDoc;
use rustfs_utils::path::SLASH_SEPARATOR;
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use time::OffsetDateTime;
use tracing::{debug, info, warn};
/// IAM config prefix under meta bucket (e.g. config/iam/).
const IAM_CONFIG_PREFIX: &str = "config/iam";
const IAM_FORMAT_FILE_PATH: &str = "config/iam/format.json";
const IAM_USERS_PREFIX: &str = "config/iam/users/";
const IAM_SERVICE_ACCOUNTS_PREFIX: &str = "config/iam/service-accounts/";
const IAM_STS_PREFIX: &str = "config/iam/sts/";
const IAM_GROUPS_PREFIX: &str = "config/iam/groups/";
const IAM_POLICIES_PREFIX: &str = "config/iam/policies/";
const IAM_POLICY_DB_PREFIX: &str = "config/iam/policydb/";
const REPLICATION_META_DIR: &str = ".replication";
const RESYNC_META_FILE: &str = "resync.bin";
#[derive(Debug, Serialize, Deserialize)]
struct CompatIamFormat {
#[serde(default)]
version: i64,
}
#[derive(Debug, Serialize, Deserialize)]
struct CompatGroupInfo {
#[serde(default)]
version: i64,
#[serde(default = "default_group_status")]
status: String,
#[serde(default)]
members: Vec<String>,
#[serde(
rename = "updatedAt",
alias = "update_at",
default,
with = "rustfs_policy::serde_datetime::option"
)]
update_at: Option<OffsetDateTime>,
}
#[derive(Debug, Serialize, Deserialize)]
struct CompatMappedPolicy {
#[serde(default)]
version: i64,
#[serde(rename = "policy", alias = "policies", default)]
policy: String,
#[serde(
rename = "updatedAt",
alias = "update_at",
default,
with = "rustfs_policy::serde_datetime::option"
)]
update_at: Option<OffsetDateTime>,
}
fn default_group_status() -> String {
"enabled".to_string()
}
fn normalize_iam_config_blob(path: &str, data: &[u8]) -> std::result::Result<Option<Vec<u8>>, String> {
if path == IAM_FORMAT_FILE_PATH {
let mut format: CompatIamFormat =
serde_json::from_slice(data).map_err(|err| format!("parse IAM format failed: {err}"))?;
if format.version <= 0 {
format.version = 1;
}
return serde_json::to_vec(&format)
.map(Some)
.map_err(|err| format!("serialize IAM format failed: {err}"));
}
if is_identity_path(path) {
let mut identity: UserIdentity =
serde_json::from_slice(data).map_err(|err| format!("parse IAM identity failed: {err}"))?;
if identity.update_at.is_none() {
identity.update_at = Some(OffsetDateTime::now_utc());
}
return serde_json::to_vec(&identity)
.map(Some)
.map_err(|err| format!("serialize IAM identity failed: {err}"));
}
if is_group_path(path) {
let mut group: CompatGroupInfo = serde_json::from_slice(data).map_err(|err| format!("parse IAM group failed: {err}"))?;
if group.update_at.is_none() {
group.update_at = Some(OffsetDateTime::now_utc());
}
return serde_json::to_vec(&group)
.map(Some)
.map_err(|err| format!("serialize IAM group failed: {err}"));
}
if is_policy_doc_path(path) {
let mut doc = PolicyDoc::try_from(data.to_vec()).map_err(|err| format!("parse IAM policy doc failed: {err}"))?;
if doc.create_date.is_none() {
doc.create_date = doc.update_date;
}
if doc.update_date.is_none() {
doc.update_date = doc.create_date;
}
return serde_json::to_vec(&doc)
.map(Some)
.map_err(|err| format!("serialize IAM policy doc failed: {err}"));
}
if is_policy_mapping_path(path) {
let mut mapped: CompatMappedPolicy =
serde_json::from_slice(data).map_err(|err| format!("parse IAM policy mapping failed: {err}"))?;
if mapped.update_at.is_none() {
mapped.update_at = Some(OffsetDateTime::now_utc());
}
return serde_json::to_vec(&mapped)
.map(Some)
.map_err(|err| format!("serialize IAM policy mapping failed: {err}"));
}
Ok(None)
}
fn is_identity_path(path: &str) -> bool {
(path.starts_with(IAM_USERS_PREFIX) || path.starts_with(IAM_SERVICE_ACCOUNTS_PREFIX) || path.starts_with(IAM_STS_PREFIX))
&& path.ends_with("/identity.json")
}
fn is_group_path(path: &str) -> bool {
path.starts_with(IAM_GROUPS_PREFIX) && path.ends_with("/members.json")
}
fn is_policy_doc_path(path: &str) -> bool {
path.starts_with(IAM_POLICIES_PREFIX) && path.ends_with("/policy.json")
}
fn is_policy_mapping_path(path: &str) -> bool {
path.starts_with(IAM_POLICY_DB_PREFIX) && path.ends_with(".json")
}
fn is_resync_meta_path(path: &str) -> bool {
path.ends_with(&format!("{REPLICATION_META_DIR}/{RESYNC_META_FILE}"))
}
fn normalize_bucket_meta_blob(path: &str, data: &[u8]) -> std::result::Result<Option<Vec<u8>>, String> {
if !is_resync_meta_path(path) {
return Ok(None);
}
let status = decode_resync_file(data).map_err(|err| format!("decode resync meta failed: {err}"))?;
encode_resync_file(&status)
.map(Some)
.map_err(|err| format!("encode resync meta failed: {err}"))
}
/// Migrates bucket metadata from legacy format to RustFS.
/// Uses list_bucket (from disk volumes) to get bucket names, since list_objects_v2 on the legacy
/// meta bucket may not work (legacy format differs from object layer expectations).
/// Skips buckets that already exist in RustFS (idempotent).
pub async fn try_migrate_bucket_metadata<S: StorageAPI>(store: Arc<S>) {
let buckets_list = match store
.list_bucket(&BucketOptions {
no_metadata: true,
..Default::default()
})
.await
{
Ok(b) => b,
Err(e) => {
warn!("list buckets failed (skip migration): {e}");
return;
}
};
let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
if buckets.is_empty() {
debug!("No migrating bucket metadata found");
return;
}
debug!("Found {} migrating bucket metadata, migrating...", buckets.len());
let opts = ObjectOptions {
max_parity: true,
no_lock: true,
..Default::default()
};
let h = HeaderMap::new();
for bucket in buckets {
let meta_path = format!("{BUCKET_META_PREFIX}{SLASH_SEPARATOR}{bucket}{SLASH_SEPARATOR}{BUCKET_METADATA_FILE}");
migrate_one_if_missing(store.clone(), &opts, &h, &meta_path, &format!("bucket metadata: {bucket}")).await;
let resync_path = format!(
"{BUCKET_META_PREFIX}{SLASH_SEPARATOR}{bucket}{SLASH_SEPARATOR}{REPLICATION_META_DIR}{SLASH_SEPARATOR}{RESYNC_META_FILE}"
);
migrate_one_if_missing(store.clone(), &opts, &h, &resync_path, &format!("bucket replication resync: {bucket}")).await;
}
}
async fn migrate_one_if_missing<S: StorageAPI>(
store: Arc<S>,
opts: &ObjectOptions,
headers: &HeaderMap,
path: &str,
label: &str,
) {
if store
.get_object_info(RUSTFS_META_BUCKET, path, &ObjectOptions::default())
.await
.is_ok()
{
debug!("{label} already exists in RustFS, skip");
return;
}
let mut rd = match store
.get_object_reader(MIGRATING_META_BUCKET, path, None, headers.clone(), opts)
.await
{
Ok(r) => r,
Err(e) => {
debug!("read migrating {label}: {e}");
return;
}
};
let data = match rd.read_all().await {
Ok(d) if !d.is_empty() => d,
Ok(_) => return,
Err(e) => {
debug!("read migrating {label} body: {e}");
return;
}
};
let data = match normalize_bucket_meta_blob(path, &data) {
Ok(Some(normalized)) => normalized,
Ok(None) => data,
Err(e) => {
warn!("skip {label} migration due to incompatible format: {e}");
return;
}
};
if let Err(e) = store
.put_object(RUSTFS_META_BUCKET, path, &mut PutObjReader::from_vec(data), opts)
.await
{
warn!("write {label}: {e}");
} else {
info!("Migrated {label}");
}
}
/// Migrates IAM config from legacy meta bucket `config/iam/` to RustFS meta bucket.
/// Lists all objects under the IAM prefix in the source, copies each to the target if not present.
/// Skips objects that already exist in RustFS (idempotent).
/// If list_objects_v2 on the legacy bucket fails (e.g. format differs), migration is skipped.
pub async fn try_migrate_iam_config<S: StorageAPI>(store: Arc<S>) {
let opts = ObjectOptions {
max_parity: true,
no_lock: true,
..Default::default()
};
let h = HeaderMap::new();
let prefix = format!("{IAM_CONFIG_PREFIX}/");
let mut continuation: Option<String> = None;
let mut total_migrated = 0usize;
loop {
let list_result = match store
.clone()
.list_objects_v2(MIGRATING_META_BUCKET, &prefix, continuation, None, 500, false, None, false)
.await
{
Ok(r) => r,
Err(e) => {
debug!("list IAM config from legacy bucket failed (skip migration): {e}");
return;
}
};
for obj in list_result.objects {
let path = &obj.name;
if path.is_empty() || path.ends_with('/') {
continue;
}
if store
.get_object_info(RUSTFS_META_BUCKET, path, &ObjectOptions::default())
.await
.is_ok()
{
debug!("IAM config already exists in RustFS, skip: {path}");
continue;
}
let mut rd = match store
.get_object_reader(MIGRATING_META_BUCKET, path, None, h.clone(), &opts)
.await
{
Ok(r) => r,
Err(e) => {
debug!("read migrating IAM config {path}: {e}");
continue;
}
};
let data = match rd.read_all().await {
Ok(d) if !d.is_empty() => d,
Ok(_) => continue,
Err(e) => {
debug!("read migrating IAM config {path} body: {e}");
continue;
}
};
let data = match normalize_iam_config_blob(path, &data) {
Ok(Some(normalized)) => normalized,
Ok(None) => {
debug!("skip unsupported IAM config path during migration: {path}");
continue;
}
Err(e) => {
warn!("skip IAM config migration due to incompatible format, path: {path}, err: {e}");
continue;
}
};
if let Err(e) = store
.put_object(RUSTFS_META_BUCKET, path, &mut PutObjReader::from_vec(data), &opts)
.await
{
warn!("write IAM config {path}: {e}");
} else {
info!("Migrated IAM config: {path}");
total_migrated += 1;
}
}
continuation = list_result.next_continuation_token.or(list_result.continuation_token);
if !list_result.is_truncated || continuation.is_none() {
break;
}
}
if total_migrated > 0 {
info!("IAM migration complete: {} object(s) migrated", total_migrated);
}
}
#[cfg(test)]
mod tests {
use super::{normalize_bucket_meta_blob, normalize_iam_config_blob};
use crate::bucket::replication::{
BucketReplicationResyncStatus, ResyncStatusType, TargetReplicationResyncStatus, decode_resync_file, encode_resync_file,
};
use std::collections::HashMap;
#[test]
fn test_normalize_policy_mapping_legacy_timestamp_and_fields() {
let path = "config/iam/policydb/users/alice.json";
let input = r#"{"version":1,"policies":"readwrite","update_at":"2026-03-09 02:22:44.998954 +00:00:00"}"#;
let output = normalize_iam_config_blob(path, input.as_bytes())
.expect("normalize should succeed")
.expect("path should be supported");
let v: serde_json::Value = serde_json::from_slice(&output).expect("output should be valid JSON");
assert_eq!(v.get("policy").and_then(|x| x.as_str()), Some("readwrite"));
assert!(v.get("policies").is_none(), "legacy field should be normalized");
let updated_at = v
.get("updatedAt")
.and_then(|x| x.as_str())
.expect("updatedAt should exist as string");
assert!(updated_at.contains('T'), "updatedAt should be RFC3339-like");
}
#[test]
fn test_normalize_bucket_meta_blob_resync_reencode() {
let path = ".buckets/test/.replication/resync.bin";
let mut status = BucketReplicationResyncStatus::new();
status.id = 123;
status.targets_map = HashMap::from([(
"arn:replication::1:dest".to_string(),
TargetReplicationResyncStatus {
resync_id: "reset-1".to_string(),
resync_status: ResyncStatusType::ResyncStarted,
replicated_count: 1,
..Default::default()
},
)]);
let input = encode_resync_file(&status).expect("encode should succeed");
let output = normalize_bucket_meta_blob(path, &input)
.expect("normalize should succeed")
.expect("resync path should be normalized");
let decoded = decode_resync_file(&output).expect("decode should succeed");
assert_eq!(decoded.id, 123);
assert_eq!(decoded.targets_map["arn:replication::1:dest"].resync_id, "reset-1");
}
}
+4
View File
@@ -18,6 +18,10 @@ pub mod error;
pub mod lifecycle;
pub mod metadata;
pub mod metadata_sys;
#[cfg(test)]
mod metadata_test;
pub mod migration;
mod msgp_decode;
pub mod object_lock;
pub mod policy_sys;
pub mod quota;
+189
View File
@@ -0,0 +1,189 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! MessagePack decode helpers for bucket metadata, aligned with msgp format.
use crate::error::{Error, Result};
use byteorder::{BigEndian, ByteOrder};
use rmp::Marker;
use std::io::{Read, Write};
use time::OffsetDateTime;
/// Skip a single MessagePack value. Used for unknown map keys.
pub(crate) fn skip_msgp_value<R: Read>(rd: &mut R) -> Result<()> {
let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
let skip_len: usize = match marker {
Marker::Null | Marker::False | Marker::True => 0,
Marker::FixPos(_) | Marker::FixNeg(_) => 0,
Marker::U8 => 1,
Marker::U16 => 2,
Marker::U32 => 4,
Marker::U64 => 8,
Marker::I8 => 1,
Marker::I16 => 2,
Marker::I32 => 4,
Marker::I64 => 8,
Marker::F32 => 4,
Marker::F64 => 8,
Marker::FixStr(n) => n as usize,
Marker::Str8 => {
let mut b = [0u8; 1];
rd.read_exact(&mut b).map_err(Error::other)?;
b[0] as usize
}
Marker::Str16 => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::other)?;
u16::from_be_bytes(b) as usize
}
Marker::Str32 => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::other)?;
u32::from_be_bytes(b) as usize
}
Marker::Bin8 => {
let mut b = [0u8; 1];
rd.read_exact(&mut b).map_err(Error::other)?;
b[0] as usize
}
Marker::Bin16 => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::other)?;
u16::from_be_bytes(b) as usize
}
Marker::Bin32 => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::other)?;
u32::from_be_bytes(b) as usize
}
Marker::FixArray(n) => {
for _ in 0..n {
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::Array16 => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::other)?;
let n = u16::from_be_bytes(b);
for _ in 0..n {
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::Array32 => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::other)?;
let n = u32::from_be_bytes(b);
for _ in 0..n {
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::FixMap(n) => {
for _ in 0..n {
skip_msgp_value(rd)?;
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::Map16 => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::other)?;
let n = u16::from_be_bytes(b);
for _ in 0..n {
skip_msgp_value(rd)?;
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::Map32 => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::other)?;
let n = u32::from_be_bytes(b);
for _ in 0..n {
skip_msgp_value(rd)?;
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::FixExt1 => 1,
Marker::FixExt2 => 2,
Marker::FixExt4 => 4,
Marker::FixExt8 => 8,
Marker::FixExt16 => 16,
Marker::Ext8 => {
let mut b = [0u8; 1];
rd.read_exact(&mut b).map_err(Error::other)?;
let len = b[0] as usize;
1 + len // type byte + data
}
Marker::Ext16 => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::other)?;
let len = u16::from_be_bytes(b) as usize;
2 + len
}
Marker::Ext32 => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::other)?;
let len = u32::from_be_bytes(b) as usize;
4 + len
}
Marker::Reserved => 0,
};
if skip_len > 0 {
let mut buf = vec![0u8; skip_len];
rd.read_exact(&mut buf).map_err(Error::other)?;
}
Ok(())
}
/// msgp time format: ext8 (0xc7), len 12, type 5, 8 bytes sec (BE) + 4 bytes nsec (BE).
pub(crate) const MSGP_TIME_EXT_TYPE: i8 = 5;
pub(crate) const MSGP_TIME_LEN: u8 = 12;
/// Read msgp ext8 time - caller must have already read the marker and verified it's ext8.
/// Ext8 format: 1 byte len, 1 byte type, then data bytes.
pub(crate) fn read_msgp_ext8_time<R: Read>(rd: &mut R) -> Result<OffsetDateTime> {
let mut len_buf = [0u8; 1];
rd.read_exact(&mut len_buf).map_err(Error::other)?;
let len = len_buf[0] as usize;
if len != MSGP_TIME_LEN as usize {
return Err(Error::other(format!("invalid msgp time len: {len}")));
}
let mut type_buf = [0u8; 1];
rd.read_exact(&mut type_buf).map_err(Error::other)?;
if type_buf[0] != MSGP_TIME_EXT_TYPE as u8 {
return Err(Error::other(format!("invalid msgp time type: {}", type_buf[0])));
}
let mut buf = [0u8; 12];
rd.read_exact(&mut buf).map_err(Error::other)?;
let sec = BigEndian::read_i64(&buf[0..8]);
let nsec = BigEndian::read_u32(&buf[8..12]);
OffsetDateTime::from_unix_timestamp(sec)
.map_err(|_| Error::other("invalid timestamp"))?
.replace_nanosecond(nsec)
.map_err(|_| Error::other("invalid nanosecond"))
}
/// Write msgp time as ext8 (0xc7), len 12, type 5. Always uses ext format (never nil).
pub(crate) fn write_msgp_time<W: Write>(wr: &mut W, t: OffsetDateTime) -> Result<()> {
wr.write_all(&[0xc7, MSGP_TIME_LEN, MSGP_TIME_EXT_TYPE as u8])
.map_err(Error::other)?;
let mut buf = [0u8; 12];
BigEndian::write_i64(&mut buf[0..8], t.unix_timestamp());
BigEndian::write_u32(&mut buf[8..12], t.nanosecond());
wr.write_all(&buf).map_err(Error::other)
}
+68 -6
View File
@@ -15,7 +15,6 @@
pub mod checker;
use crate::error::Result;
use rmp_serde::Serializer as rmpSerializer;
use rustfs_config::{
QUOTA_API_PATH, QUOTA_EXCEEDED_ERROR_CODE, QUOTA_INTERNAL_ERROR_CODE, QUOTA_INVALID_CONFIG_ERROR_CODE,
QUOTA_NOT_FOUND_ERROR_CODE,
@@ -28,27 +27,35 @@ use time::OffsetDateTime;
pub enum QuotaType {
/// Hard quota: reject immediately when exceeded
#[default]
#[serde(alias = "HARD", alias = "hard")]
Hard,
}
/// Bucket quota configuration. quota_type defaults to Hard when omitted.
#[derive(Debug, Deserialize, Serialize, Default, Clone, PartialEq)]
pub struct BucketQuota {
#[serde(default)]
pub quota: Option<u64>,
/// Defaults to Hard when missing.
#[serde(default)]
pub quota_type: QuotaType,
/// Timestamp when this quota configuration was set (for audit purposes)
#[serde(default, with = "time::serde::rfc3339::option")]
pub created_at: Option<OffsetDateTime>,
/// Accept updated_at for compatibility; not used.
#[serde(default, with = "time::serde::rfc3339::option", skip_serializing_if = "Option::is_none")]
pub updated_at: Option<OffsetDateTime>,
}
impl BucketQuota {
/// Serialize to JSON bytes. Same format as parse_all_configs.
pub fn marshal_msg(&self) -> Result<Vec<u8>> {
let mut buf = Vec::new();
self.serialize(&mut rmpSerializer::new(&mut buf).with_struct_map())?;
Ok(buf)
serde_json::to_vec(self).map_err(Into::into)
}
/// Deserialize from JSON bytes. Same format as parse_all_configs.
pub fn unmarshal(buf: &[u8]) -> Result<Self> {
let t: BucketQuota = rmp_serde::from_slice(buf)?;
Ok(t)
serde_json::from_slice(buf).map_err(Into::into)
}
pub fn new(quota: Option<u64>) -> Self {
@@ -57,6 +64,7 @@ impl BucketQuota {
quota,
quota_type: QuotaType::Hard,
created_at: Some(now),
updated_at: None,
}
}
@@ -156,3 +164,57 @@ impl QuotaErrorResponse {
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Legacy format: quota, created_at, updated_at (no quota_type)
#[test]
fn deserialize_format_without_quota_type() {
let json = r#"{"quota":1073741824,"created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#;
let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
assert_eq!(q.quota, Some(1073741824));
assert_eq!(q.quota_type, QuotaType::Hard);
assert!(q.created_at.is_some());
assert!(q.updated_at.is_some());
}
/// RustFS format: quota, quota_type, created_at
#[test]
fn deserialize_rustfs_format() {
let json = r#"{"quota":1073741824,"quota_type":"Hard","created_at":"2024-01-01T00:00:00Z"}"#;
let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
assert_eq!(q.quota, Some(1073741824));
assert_eq!(q.quota_type, QuotaType::Hard);
assert!(q.created_at.is_some());
assert!(q.created_at.is_some_and(|t| t.unix_timestamp() == 1704067200));
}
/// E2E format uses "HARD" (uppercase)
#[test]
fn deserialize_quota_type_hard_uppercase() {
let json = r#"{"quota":2048,"quota_type":"HARD"}"#;
let q: BucketQuota = serde_json::from_slice(json.as_bytes()).expect("should parse");
assert_eq!(q.quota_type, QuotaType::Hard);
}
/// marshal_msg/unmarshal use JSON, same as parse_all_configs
#[test]
fn marshal_unmarshal_roundtrip() {
let q = BucketQuota::new(Some(1073741824));
let buf = q.marshal_msg().expect("marshal");
let restored = BucketQuota::unmarshal(&buf).expect("unmarshal");
assert_eq!(q.quota, restored.quota);
assert_eq!(q.quota_type, restored.quota_type);
}
/// unmarshal accepts format without quota_type
#[test]
fn unmarshal_format_without_quota_type() {
let json = r#"{"quota":1073741824,"created_at":"2024-01-01T00:00:00Z","updated_at":"2024-01-01T00:00:00Z"}"#;
let q = BucketQuota::unmarshal(json.as_bytes()).expect("should parse");
assert_eq!(q.quota, Some(1073741824));
assert_eq!(q.quota_type, QuotaType::Hard);
}
}
@@ -20,8 +20,8 @@ use crate::bucket::replication::ResyncStatusType;
use crate::bucket::replication::replicate_delete;
use crate::bucket::replication::replicate_object;
use crate::bucket::replication::replication_resyncer::{
BucketReplicationResyncStatus, DeletedObjectReplicationInfo, ReplicationConfig, ReplicationResyncer,
get_heal_replicate_object_info,
BucketReplicationResyncStatus, DeletedObjectReplicationInfo, REPLICATION_DIR, RESYNC_FILE_NAME, ReplicationConfig,
ReplicationResyncer, decode_resync_file, get_heal_replicate_object_info,
};
use crate::bucket::replication::replication_state::ReplicationStats;
use crate::config::com::read_config;
@@ -41,7 +41,7 @@ use rustfs_filemeta::VersionPurgeStatusType;
use rustfs_filemeta::replication_statuses_map;
use rustfs_filemeta::version_purge_statuses_map;
use rustfs_filemeta::{REPLICATE_EXISTING, REPLICATE_HEAL, REPLICATE_HEAL_DELETE};
use rustfs_utils::http::RESERVED_METADATA_PREFIX_LOWER;
use rustfs_utils::http::{SUFFIX_REPLICATION_TIMESTAMP, get_str};
use std::any::Any;
use std::sync::Arc;
use std::sync::atomic::AtomicI32;
@@ -861,17 +861,8 @@ async fn load_bucket_resync_metadata<S: StorageAPI>(
bucket: &str,
obj_api: Arc<S>,
) -> Result<BucketReplicationResyncStatus, EcstoreError> {
use std::convert::TryInto;
let mut brs = BucketReplicationResyncStatus::new();
// Constants that would be defined elsewhere
const REPLICATION_DIR: &str = "replication";
const RESYNC_FILE_NAME: &str = "resync.bin";
const RESYNC_META_FORMAT: u16 = 1;
const RESYNC_META_VERSION: u16 = 1;
const RESYNC_META_VERSION_V1: u16 = 1;
let resync_dir_path = format!("{BUCKET_META_PREFIX}/{bucket}/{REPLICATION_DIR}");
let resync_file_path = format!("{resync_dir_path}/{RESYNC_FILE_NAME}");
@@ -886,27 +877,7 @@ async fn load_bucket_resync_metadata<S: StorageAPI>(
return Ok(brs);
}
if data.len() <= 4 {
return Err(EcstoreError::CorruptedFormat);
}
// Read resync meta header
let format = u16::from_le_bytes(data[0..2].try_into().unwrap());
if format != RESYNC_META_FORMAT {
return Err(EcstoreError::CorruptedFormat);
}
let version = u16::from_le_bytes(data[2..4].try_into().unwrap());
if version != RESYNC_META_VERSION {
return Err(EcstoreError::CorruptedFormat);
}
// Parse data
brs = BucketReplicationResyncStatus::unmarshal_msg(&data[4..])?;
if brs.version != RESYNC_META_VERSION_V1 {
return Err(EcstoreError::CorruptedFormat);
}
brs = decode_resync_file(&data)?;
Ok(brs)
}
@@ -984,10 +955,8 @@ pub fn get_global_replication_pool() -> Option<Arc<DynReplicationPool>> {
pub async fn schedule_replication<S: StorageAPI>(oi: ObjectInfo, o: Arc<S>, dsc: ReplicateDecision, op_type: ReplicationType) {
let tgt_statuses = replication_statuses_map(&oi.replication_status_internal.clone().unwrap_or_default());
let purge_statuses = version_purge_statuses_map(&oi.version_purge_status_internal.clone().unwrap_or_default());
let tm = oi
.user_defined
.get(&format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replication-timestamp"))
.map(|v| OffsetDateTime::parse(v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH));
let tm = get_str(&oi.user_defined, SUFFIX_REPLICATION_TIMESTAMP)
.map(|v| OffsetDateTime::parse(&v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH));
let mut rstate = oi.replication_state();
rstate.replicate_decision_str = dsc.to_string();
let asz = oi.get_actual_size().unwrap_or_default();
@@ -17,6 +17,7 @@ use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, BucketTargetSys, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient,
};
use crate::bucket::metadata_sys;
use crate::bucket::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
use crate::bucket::replication::ResyncStatusType;
use crate::bucket::replication::replication_pool::GLOBAL_REPLICATION_STATS;
use crate::bucket::replication::{ObjectOpts, ReplicationConfigurationExt as _};
@@ -50,7 +51,7 @@ use http_body::Frame;
use http_body_util::StreamBody;
use regex::Regex;
use rustfs_filemeta::{
MrfReplicateEntry, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATION_RESET, ReplicateDecision, ReplicateObjectInfo,
MrfReplicateEntry, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, ReplicateDecision, ReplicateObjectInfo,
ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType,
ReplicationType, ReplicationWorkerOperation, ResyncDecision, ResyncTargetDecision, VersionPurgeStatusType,
get_replication_state, parse_replicate_decision, replication_statuses_map, target_reset_header, version_purge_statuses_map,
@@ -58,8 +59,13 @@ use rustfs_filemeta::{
use rustfs_s3_common::EventName;
use rustfs_utils::http::{
AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_TAGGING, AMZ_TAGGING_DIRECTIVE, CONTENT_ENCODING, HeaderExt as _,
RESERVED_METADATA_PREFIX, RESERVED_METADATA_PREFIX_LOWER, RUSTFS_REPLICATION_ACTUAL_OBJECT_SIZE,
RUSTFS_REPLICATION_RESET_STATUS, SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, headers,
SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP,
SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, SUFFIX_REPLICATION_RESET, SUFFIX_REPLICATION_RESET_ARN_PREFIX,
SUFFIX_REPLICATION_STATUS, SUFFIX_TAGGING_TIMESTAMP, headers,
};
use rustfs_utils::http::{
SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_RESET_STATUS, SUFFIX_REPLICATION_SSEC_CRC, get_header_map, get_str,
has_internal_suffix, insert_header_map, insert_str, internal_key_strip_suffix_prefix, is_internal_key,
};
use rustfs_utils::path::path_join_buf;
use rustfs_utils::string::strings_has_prefix_fold;
@@ -69,7 +75,8 @@ use serde::Deserialize;
use serde::Serialize;
use std::any::Any;
use std::collections::HashMap;
use std::sync::Arc;
use std::io::{Cursor, Read};
use std::sync::{Arc, LazyLock};
use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
use tokio::io::AsyncRead;
@@ -80,14 +87,29 @@ use tokio_util::io::ReaderStream;
use tokio_util::sync::CancellationToken;
use tracing::{error, info, instrument, warn};
const REPLICATION_DIR: &str = ".replication";
const RESYNC_FILE_NAME: &str = "resync.bin";
const RESYNC_META_FORMAT: u16 = 1;
const RESYNC_META_VERSION: u16 = 1;
pub(crate) const REPLICATION_DIR: &str = ".replication";
pub(crate) const RESYNC_FILE_NAME: &str = "resync.bin";
pub(crate) const RESYNC_META_FORMAT: u16 = 1;
pub(crate) const RESYNC_META_VERSION: u16 = 1;
const RESYNC_TIME_INTERVAL: TokioDuration = TokioDuration::from_secs(60);
const WIRE_ZERO_TIME_UNIX: i64 = -62_135_596_800;
static WIRE_ZERO_TIME: LazyLock<OffsetDateTime> =
LazyLock::new(|| OffsetDateTime::from_unix_timestamp(WIRE_ZERO_TIME_UNIX).unwrap_or(OffsetDateTime::UNIX_EPOCH));
static WARNED_MONITOR_UNINIT: std::sync::Once = std::sync::Once::new();
fn wire_time_or_default(value: Option<OffsetDateTime>) -> OffsetDateTime {
value.unwrap_or(*WIRE_ZERO_TIME)
}
fn normalize_wire_time(value: Option<OffsetDateTime>) -> Option<OffsetDateTime> {
match value {
Some(v) if v == *WIRE_ZERO_TIME || v == OffsetDateTime::UNIX_EPOCH => None,
other => other,
}
}
#[derive(Debug, Clone, Default)]
pub struct ResyncOpts {
pub bucket: String,
@@ -139,14 +161,199 @@ impl BucketReplicationResyncStatus {
}
pub fn marshal_msg(&self) -> Result<Vec<u8>> {
Ok(rmp_serde::to_vec(&self)?)
let mut wr = Vec::new();
rmp::encode::write_map_len(&mut wr, 4)?;
rmp::encode::write_str(&mut wr, "v")?;
rmp::encode::write_i32(&mut wr, i32::from(self.version))?;
rmp::encode::write_str(&mut wr, "brs")?;
rmp::encode::write_map_len(&mut wr, self.targets_map.len() as u32)?;
for (arn, status) in &self.targets_map {
rmp::encode::write_str(&mut wr, arn)?;
status.marshal_wire_msg(&mut wr)?;
}
rmp::encode::write_str(&mut wr, "id")?;
rmp::encode::write_i32(&mut wr, self.id)?;
rmp::encode::write_str(&mut wr, "lu")?;
write_msgp_time(&mut wr, wire_time_or_default(self.last_update))?;
Ok(wr)
}
pub fn unmarshal_msg(data: &[u8]) -> Result<Self> {
let mut rd = Cursor::new(data);
let mut out = Self::new();
let mut fields = rmp::decode::read_map_len(&mut rd)?;
while fields > 0 {
fields -= 1;
let key = read_msgp_str(&mut rd)?;
match key.as_str() {
"v" => {
let v: i32 = rmp::decode::read_int(&mut rd)?;
out.version = u16::try_from(v).map_err(|_| Error::other("invalid resync version"))?;
}
"brs" => {
let map_len = rmp::decode::read_map_len(&mut rd)?;
let mut targets = HashMap::with_capacity(map_len as usize);
for _ in 0..map_len {
let arn = read_msgp_str(&mut rd)?;
let status = TargetReplicationResyncStatus::unmarshal_wire_msg(&mut rd)?;
targets.insert(arn, status);
}
out.targets_map = targets;
}
"id" => {
out.id = rmp::decode::read_int::<i32, _>(&mut rd)?;
}
"lu" => {
out.last_update = normalize_wire_time(read_msgp_time_or_nil(&mut rd)?);
}
_ => skip_msgp_value(&mut rd)?,
}
}
Ok(out)
}
pub fn unmarshal_legacy_msg(data: &[u8]) -> Result<Self> {
Ok(rmp_serde::from_slice(data)?)
}
}
pub(crate) fn encode_resync_file(status: &BucketReplicationResyncStatus) -> Result<Vec<u8>> {
let payload = status.marshal_msg()?;
let mut data = Vec::with_capacity(4 + payload.len());
let mut major = [0u8; 2];
byteorder::LittleEndian::write_u16(&mut major, RESYNC_META_FORMAT);
data.extend_from_slice(&major);
let mut minor = [0u8; 2];
byteorder::LittleEndian::write_u16(&mut minor, RESYNC_META_VERSION);
data.extend_from_slice(&minor);
data.extend_from_slice(&payload);
Ok(data)
}
pub(crate) fn decode_resync_file(data: &[u8]) -> Result<BucketReplicationResyncStatus> {
if data.len() <= 4 {
return Err(Error::CorruptedFormat);
}
let mut major = [0u8; 2];
major.copy_from_slice(&data[0..2]);
if byteorder::LittleEndian::read_u16(&major) != RESYNC_META_FORMAT {
return Err(Error::CorruptedFormat);
}
let mut minor = [0u8; 2];
minor.copy_from_slice(&data[2..4]);
if byteorder::LittleEndian::read_u16(&minor) != RESYNC_META_VERSION {
return Err(Error::CorruptedFormat);
}
let status = match BucketReplicationResyncStatus::unmarshal_msg(&data[4..]) {
Ok(v) => v,
Err(_) => BucketReplicationResyncStatus::unmarshal_legacy_msg(&data[4..])?,
};
if status.version != RESYNC_META_VERSION {
return Err(Error::CorruptedFormat);
}
Ok(status)
}
impl TargetReplicationResyncStatus {
fn marshal_wire_msg(&self, wr: &mut Vec<u8>) -> Result<()> {
rmp::encode::write_map_len(wr, 11)?;
rmp::encode::write_str(wr, "st")?;
write_msgp_time(wr, wire_time_or_default(self.start_time))?;
rmp::encode::write_str(wr, "lst")?;
write_msgp_time(wr, wire_time_or_default(self.last_update))?;
rmp::encode::write_str(wr, "id")?;
rmp::encode::write_str(wr, &self.resync_id)?;
rmp::encode::write_str(wr, "rdt")?;
write_msgp_time(wr, wire_time_or_default(self.resync_before_date))?;
rmp::encode::write_str(wr, "rst")?;
rmp::encode::write_i32(wr, resync_status_to_i32(self.resync_status))?;
rmp::encode::write_str(wr, "fs")?;
rmp::encode::write_i64(wr, self.failed_size)?;
rmp::encode::write_str(wr, "frc")?;
rmp::encode::write_i64(wr, self.failed_count)?;
rmp::encode::write_str(wr, "rs")?;
rmp::encode::write_i64(wr, self.replicated_size)?;
rmp::encode::write_str(wr, "rrc")?;
rmp::encode::write_i64(wr, self.replicated_count)?;
rmp::encode::write_str(wr, "bkt")?;
rmp::encode::write_str(wr, &self.bucket)?;
rmp::encode::write_str(wr, "obj")?;
rmp::encode::write_str(wr, &self.object)?;
Ok(())
}
fn unmarshal_wire_msg<R: Read>(rd: &mut R) -> Result<Self> {
let mut out = Self::new();
let mut fields = rmp::decode::read_map_len(rd)?;
while fields > 0 {
fields -= 1;
let key = read_msgp_str(rd)?;
match key.as_str() {
"st" => out.start_time = normalize_wire_time(read_msgp_time_or_nil(rd)?),
"lst" => out.last_update = normalize_wire_time(read_msgp_time_or_nil(rd)?),
"id" => out.resync_id = read_msgp_str(rd)?,
"rdt" => out.resync_before_date = normalize_wire_time(read_msgp_time_or_nil(rd)?),
"rst" => {
let v: i32 = rmp::decode::read_int(rd)?;
out.resync_status = resync_status_from_i32(v)?;
}
"fs" => out.failed_size = rmp::decode::read_int(rd)?,
"frc" => out.failed_count = rmp::decode::read_int(rd)?,
"rs" => out.replicated_size = rmp::decode::read_int(rd)?,
"rrc" => out.replicated_count = rmp::decode::read_int(rd)?,
"bkt" => out.bucket = read_msgp_str(rd)?,
"obj" => out.object = read_msgp_str(rd)?,
_ => skip_msgp_value(rd)?,
}
}
Ok(out)
}
}
fn read_msgp_str<R: Read>(rd: &mut R) -> Result<String> {
let len = rmp::decode::read_str_len(rd)? as usize;
let mut buf = vec![0u8; len];
rd.read_exact(&mut buf)?;
Ok(String::from_utf8(buf)?)
}
fn read_msgp_time_or_nil<R: Read>(rd: &mut R) -> Result<Option<OffsetDateTime>> {
let marker = rmp::decode::read_marker(rd).map_err(|e| Error::other(format!("{e:?}")))?;
match marker {
rmp::Marker::Null => Ok(None),
rmp::Marker::Ext8 => Ok(Some(read_msgp_ext8_time(rd)?)),
other => Err(Error::other(format!("expected time ext or nil, got marker: {other:?}"))),
}
}
fn resync_status_to_i32(status: ResyncStatusType) -> i32 {
match status {
ResyncStatusType::NoResync => 0,
ResyncStatusType::ResyncPending => 1,
ResyncStatusType::ResyncCanceled => 2,
ResyncStatusType::ResyncStarted => 3,
ResyncStatusType::ResyncCompleted => 4,
ResyncStatusType::ResyncFailed => 5,
}
}
fn resync_status_from_i32(code: i32) -> Result<ResyncStatusType> {
match code {
0 => Ok(ResyncStatusType::NoResync),
1 => Ok(ResyncStatusType::ResyncPending),
2 => Ok(ResyncStatusType::ResyncCanceled),
3 => Ok(ResyncStatusType::ResyncStarted),
4 => Ok(ResyncStatusType::ResyncCompleted),
5 => Ok(ResyncStatusType::ResyncFailed),
_ => Err(Error::other(format!("invalid resync status code: {code}"))),
}
}
static RESYNC_WORKER_COUNT: usize = 10;
#[derive(Debug)]
@@ -617,7 +824,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
let keys_to_update: Vec<_> = user_defined
.iter()
.filter(|(k, _)| k.eq_ignore_ascii_case(format!("{RESERVED_METADATA_PREFIX_LOWER}{REPLICATION_RESET}").as_str()))
.filter(|(k, _)| has_internal_suffix(k, SUFFIX_REPLICATION_RESET))
.map(|(k, v)| (k.clone(), v.clone()))
.collect();
@@ -695,19 +902,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
}
async fn save_resync_status<S: StorageAPI>(bucket: &str, status: &BucketReplicationResyncStatus, api: Arc<S>) -> Result<()> {
let buf = status.marshal_msg()?;
let mut data = Vec::new();
let mut major = [0u8; 2];
byteorder::LittleEndian::write_u16(&mut major, RESYNC_META_FORMAT);
data.extend_from_slice(&major);
let mut minor = [0u8; 2];
byteorder::LittleEndian::write_u16(&mut minor, RESYNC_META_VERSION);
data.extend_from_slice(&minor);
data.extend_from_slice(&buf);
let data = encode_resync_file(status)?;
let config_file = path_join_buf(&[BUCKET_META_PREFIX, bucket, REPLICATION_DIR, RESYNC_FILE_NAME]);
save_config(api, &config_file, data).await?;
@@ -900,8 +1095,8 @@ pub fn resync_target(
let rs = oi
.user_defined
.get(target_reset_header(arn).as_str())
.or(oi.user_defined.get(RUSTFS_REPLICATION_RESET_STATUS))
.map(|s| s.to_string());
.cloned()
.or_else(|| get_header_map(&oi.user_defined, SUFFIX_REPLICATION_RESET_STATUS));
let mut dec = ResyncTargetDecision::default();
@@ -1132,15 +1327,7 @@ impl ObjectInfoExt for ObjectInfo {
.user_defined
.iter()
.filter_map(|(k, v)| {
if k.starts_with(&format!("{RESERVED_METADATA_PREFIX_LOWER}-{REPLICATION_RESET}")) {
Some((
k.trim_start_matches(&format!("{RESERVED_METADATA_PREFIX_LOWER}-{REPLICATION_RESET}"))
.to_string(),
v.clone(),
))
} else {
None
}
internal_key_strip_suffix_prefix(k, SUFFIX_REPLICATION_RESET_ARN_PREFIX).map(|arn| (arn, v.clone()))
})
.collect(),
..Default::default()
@@ -1893,7 +2080,7 @@ pub async fn replicate_object<S: StorageAPI>(roi: ReplicateObjectInfo, storage:
if roi.replication_status_internal != new_replication_internal || rinfos.replication_resynced() {
let mut eval_metadata = HashMap::new();
if let Some(ref s) = new_replication_internal {
eval_metadata.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}replication-status"), s.clone());
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_STATUS, s.clone());
}
let popts = ObjectOptions {
version_id: roi.version_id.map(|v| v.to_string()),
@@ -2549,8 +2736,6 @@ static VALID_SSE_REPLICATION_HEADERS: &[(&str, &str)] = &[
("X-Rustfs-Internal-Actual-Object-Size", "X-Rustfs-Replication-Actual-Object-Size"),
];
const REPLICATION_SSEC_CHECKSUM_HEADER: &str = "X-Rustfs-Replication-Ssec-Crc";
fn is_valid_sse_header(k: &str) -> Option<&str> {
VALID_SSE_REPLICATION_HEADERS
.iter()
@@ -2574,7 +2759,7 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
// In case of SSE-C objects copy the allowed internal headers as well
if !is_ssec || !has_valid_sse_header {
if strings_has_prefix_fold(k, RESERVED_METADATA_PREFIX) {
if is_internal_key(k) {
continue;
}
if is_standard_header(k) {
@@ -2598,7 +2783,7 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
// Add encrypted CRC to metadata for SSE-C objects
if is_ssec {
let encoded = BASE64_STANDARD.encode(checksum_data);
meta.insert(REPLICATION_SSEC_CHECKSUM_HEADER.to_string(), encoded);
insert_header_map(&mut meta, SUFFIX_REPLICATION_SSEC_CRC, encoded);
} else {
// Get checksum metadata for non-SSE-C objects
let (cs_meta, is_mp) = object_info.decrypt_checksums(0, &HeaderMap::new())?;
@@ -2658,11 +2843,8 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
if !tags.is_empty() {
put_op.user_tags = tags;
// set tag timestamp in opts
put_op.internal.tagging_timestamp = if let Some(ts) = object_info
.user_defined
.get(&format!("{RESERVED_METADATA_PREFIX_LOWER}tagging-timestamp"))
{
OffsetDateTime::parse(ts, &Rfc3339)
put_op.internal.tagging_timestamp = if let Some(ts) = get_str(&object_info.user_defined, SUFFIX_TAGGING_TIMESTAMP) {
OffsetDateTime::parse(&ts, &Rfc3339)
.map_err(|e| Error::other(format!("Failed to parse tagging timestamp: {}", e)))?
} else {
object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
@@ -2694,28 +2876,24 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
put_op.retain_until_date =
OffsetDateTime::parse(v, &Rfc3339).map_err(|e| Error::other(format!("Failed to parse retain until date: {}", e)))?;
// set retention timestamp in opts
put_op.internal.retention_timestamp = if let Some(v) = object_info
.user_defined
.get(&format!("{RESERVED_METADATA_PREFIX_LOWER}objectlock-retention-timestamp"))
{
OffsetDateTime::parse(v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)
} else {
object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
};
put_op.internal.retention_timestamp =
if let Some(v) = get_str(&object_info.user_defined, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP) {
OffsetDateTime::parse(&v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)
} else {
object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
};
}
if let Some(v) = lk_map.lookup(AMZ_OBJECT_LOCK_LEGAL_HOLD) {
let hold = v.to_uppercase();
put_op.legalhold = Some(ObjectLockLegalHoldStatus::from(hold.as_str()));
// set legalhold timestamp in opts
put_op.internal.legalhold_timestamp = if let Some(v) = object_info
.user_defined
.get(&format!("{RESERVED_METADATA_PREFIX_LOWER}objectlock-legalhold-timestamp"))
{
OffsetDateTime::parse(v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)
} else {
object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
};
put_op.internal.legalhold_timestamp =
if let Some(v) = get_str(&object_info.user_defined, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP) {
OffsetDateTime::parse(&v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH)
} else {
object_info.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH)
};
}
// Handle SSE-S3 encryption
@@ -2736,7 +2914,7 @@ fn put_replication_opts(sc: &str, object_info: &ObjectInfo) -> Result<(PutObject
// If KMS key ID replication is enabled (as by default)
// we include the object's KMS key ID. In any case, we
// always set the SSE-KMS header. If no KMS key ID is
// specified, MinIO is supposed to use whatever default
// specified, the server uses the default applicable
// config applies on the site or bucket.
// TODO: Implement SSE-KMS support with key ID replication
// let key_id = if kms::replicate_key_id() {
@@ -2859,13 +3037,10 @@ async fn replicate_object_with_multipart<S: StorageAPI>(ctx: MultipartReplicatio
let mut user_metadata = HashMap::new();
user_metadata.insert(
RUSTFS_REPLICATION_ACTUAL_OBJECT_SIZE.to_string(),
object_info
.user_defined
.get(&format!("{RESERVED_METADATA_PREFIX}actual-size"))
.map(|v| v.to_string())
.unwrap_or_default(),
insert_header_map(
&mut user_metadata,
SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE,
rustfs_utils::http::get_str(&object_info.user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE).unwrap_or_default(),
);
cli.complete_multipart_upload(
@@ -2994,6 +3169,9 @@ fn get_replication_action(oi1: &ObjectInfo, oi2: &HeadObjectOutput, op_type: Rep
#[cfg(test)]
mod tests {
use super::*;
use crate::bucket::msgp_decode::write_msgp_time;
use std::collections::HashMap;
use time::OffsetDateTime;
use uuid::Uuid;
#[test]
@@ -3010,6 +3188,176 @@ mod tests {
assert!(part_range_spec_from_actual_size(0, -1).is_err());
}
#[test]
fn test_unmarshal_resync_payload() {
let start = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid ts");
let last = OffsetDateTime::from_unix_timestamp(1_700_000_123).expect("valid ts");
let before = OffsetDateTime::from_unix_timestamp(1_699_000_000).expect("valid ts");
let bucket_last = OffsetDateTime::from_unix_timestamp(1_700_111_111).expect("valid ts");
let mut payload = Vec::new();
rmp::encode::write_map_len(&mut payload, 4).expect("write map");
rmp::encode::write_str(&mut payload, "v").expect("write key");
rmp::encode::write_i32(&mut payload, 1).expect("write version");
rmp::encode::write_str(&mut payload, "brs").expect("write key");
rmp::encode::write_map_len(&mut payload, 1).expect("write target map");
rmp::encode::write_str(&mut payload, "arn:replication::1:dest").expect("write arn");
rmp::encode::write_map_len(&mut payload, 11).expect("write target");
rmp::encode::write_str(&mut payload, "st").expect("write key");
write_msgp_time(&mut payload, start).expect("write time");
rmp::encode::write_str(&mut payload, "lst").expect("write key");
write_msgp_time(&mut payload, last).expect("write time");
rmp::encode::write_str(&mut payload, "id").expect("write key");
rmp::encode::write_str(&mut payload, "resync-1").expect("write id");
rmp::encode::write_str(&mut payload, "rdt").expect("write key");
write_msgp_time(&mut payload, before).expect("write time");
rmp::encode::write_str(&mut payload, "rst").expect("write key");
rmp::encode::write_i32(&mut payload, 3).expect("write status");
rmp::encode::write_str(&mut payload, "fs").expect("write key");
rmp::encode::write_i64(&mut payload, 11).expect("write fs");
rmp::encode::write_str(&mut payload, "frc").expect("write key");
rmp::encode::write_i64(&mut payload, 2).expect("write frc");
rmp::encode::write_str(&mut payload, "rs").expect("write key");
rmp::encode::write_i64(&mut payload, 101).expect("write rs");
rmp::encode::write_str(&mut payload, "rrc").expect("write key");
rmp::encode::write_i64(&mut payload, 9).expect("write rrc");
rmp::encode::write_str(&mut payload, "bkt").expect("write key");
rmp::encode::write_str(&mut payload, "bucket-a").expect("write bucket");
rmp::encode::write_str(&mut payload, "obj").expect("write key");
rmp::encode::write_str(&mut payload, "object-a").expect("write obj");
rmp::encode::write_str(&mut payload, "id").expect("write key");
rmp::encode::write_i32(&mut payload, 42).expect("write id");
rmp::encode::write_str(&mut payload, "lu").expect("write key");
write_msgp_time(&mut payload, bucket_last).expect("write lu");
let got = BucketReplicationResyncStatus::unmarshal_msg(&payload).expect("decode");
assert_eq!(got.version, 1);
assert_eq!(got.id, 42);
assert_eq!(got.last_update, Some(bucket_last));
let tgt = got.targets_map.get("arn:replication::1:dest").expect("target exists");
assert_eq!(tgt.resync_id, "resync-1");
assert_eq!(tgt.resync_status, ResyncStatusType::ResyncStarted);
assert_eq!(tgt.bucket, "bucket-a");
assert_eq!(tgt.object, "object-a");
assert_eq!(tgt.start_time, Some(start));
assert_eq!(tgt.last_update, Some(last));
assert_eq!(tgt.resync_before_date, Some(before));
}
#[test]
fn test_unmarshal_legacy_resync_payload() {
let mut status = BucketReplicationResyncStatus::new();
status.id = 7;
status.version = 1;
status.last_update = Some(OffsetDateTime::from_unix_timestamp(1_700_222_222).expect("valid ts"));
status.targets_map = HashMap::from([(
"legacy-arn".to_string(),
TargetReplicationResyncStatus {
resync_id: "legacy-1".to_string(),
resync_status: ResyncStatusType::ResyncCompleted,
..Default::default()
},
)]);
let old_payload = rmp_serde::to_vec(&status).expect("legacy encode");
let got = BucketReplicationResyncStatus::unmarshal_legacy_msg(&old_payload).expect("legacy decode");
assert_eq!(got.id, 7);
assert_eq!(got.version, 1);
assert_eq!(got.targets_map["legacy-arn"].resync_id, "legacy-1");
assert_eq!(got.targets_map["legacy-arn"].resync_status, ResyncStatusType::ResyncCompleted);
}
#[test]
fn test_resync_file_roundtrip_wire_format() {
let mut status = BucketReplicationResyncStatus::new();
status.id = 19;
status.last_update = Some(OffsetDateTime::from_unix_timestamp(1_700_333_333).expect("valid ts"));
status.targets_map = HashMap::from([(
"arn:replication::1:dest".to_string(),
TargetReplicationResyncStatus {
resync_id: "wire-1".to_string(),
resync_status: ResyncStatusType::ResyncStarted,
replicated_count: 5,
..Default::default()
},
)]);
let bytes = encode_resync_file(&status).expect("encode file");
assert_eq!(&bytes[0..2], &RESYNC_META_FORMAT.to_le_bytes());
assert_eq!(&bytes[2..4], &RESYNC_META_VERSION.to_le_bytes());
let got = decode_resync_file(&bytes).expect("decode file");
assert_eq!(got.version, RESYNC_META_VERSION);
assert_eq!(got.id, 19);
assert_eq!(got.targets_map["arn:replication::1:dest"].resync_id, "wire-1");
assert_eq!(got.targets_map["arn:replication::1:dest"].replicated_count, 5);
}
#[test]
fn test_resync_file_decodes_legacy_payload() {
let mut status = BucketReplicationResyncStatus::new();
status.id = 7;
status.version = RESYNC_META_VERSION;
status.targets_map = HashMap::from([(
"legacy-arn".to_string(),
TargetReplicationResyncStatus {
resync_id: "legacy-v1".to_string(),
resync_status: ResyncStatusType::ResyncCompleted,
..Default::default()
},
)]);
let legacy_payload = rmp_serde::to_vec(&status).expect("legacy encode");
let mut file_bytes = Vec::new();
file_bytes.extend_from_slice(&RESYNC_META_FORMAT.to_le_bytes());
file_bytes.extend_from_slice(&RESYNC_META_VERSION.to_le_bytes());
file_bytes.extend_from_slice(&legacy_payload);
let got = decode_resync_file(&file_bytes).expect("decode legacy");
assert_eq!(got.id, 7);
assert_eq!(got.targets_map["legacy-arn"].resync_id, "legacy-v1");
assert_eq!(got.targets_map["legacy-arn"].resync_status, ResyncStatusType::ResyncCompleted);
}
#[test]
fn test_resync_none_time_encodes_as_wire_zero_and_decodes_to_none() {
let wire_zero = OffsetDateTime::from_unix_timestamp(WIRE_ZERO_TIME_UNIX).expect("valid wire zero timestamp");
let mut with_none = BucketReplicationResyncStatus::new();
with_none.id = 77;
with_none.targets_map = HashMap::from([(
"arn:replication::1:dest".to_string(),
TargetReplicationResyncStatus {
resync_id: "wire-none".to_string(),
resync_status: ResyncStatusType::ResyncStarted,
replicated_count: 1,
..Default::default()
},
)]);
let mut with_zero = with_none.clone();
with_zero.last_update = Some(wire_zero);
if let Some(target) = with_zero.targets_map.get_mut("arn:replication::1:dest") {
target.start_time = Some(wire_zero);
target.last_update = Some(wire_zero);
target.resync_before_date = Some(wire_zero);
}
let encoded_none = encode_resync_file(&with_none).expect("encode with none");
let encoded_zero = encode_resync_file(&with_zero).expect("encode with zero");
assert_eq!(encoded_none, encoded_zero);
let decoded = decode_resync_file(&encoded_none).expect("decode");
let target = decoded
.targets_map
.get("arn:replication::1:dest")
.expect("target should exist");
assert_eq!(decoded.last_update, None);
assert_eq!(target.start_time, None);
assert_eq!(target.last_update, None);
assert_eq!(target.resync_before_date, None);
}
#[test]
fn test_heal_should_use_check_replicate_delete_failed_non_delete_marker() {
let oi = ObjectInfo {
+2 -2
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::disk::RUSTFS_META_BUCKET;
use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
use crate::error::{Error, Result, StorageError};
use regex::Regex;
use rustfs_utils::path::SLASH_SEPARATOR;
@@ -20,7 +20,7 @@ use s3s::xml;
use tracing::instrument;
pub fn is_meta_bucketname(name: &str) -> bool {
name.starts_with(RUSTFS_META_BUCKET)
name.starts_with(RUSTFS_META_BUCKET) || name.starts_with(MIGRATING_META_BUCKET)
}
lazy_static::lazy_static! {
+331 -8
View File
@@ -13,16 +13,17 @@
// limitations under the License.
use crate::config::{Config, GLOBAL_STORAGE_CLASS, storageclass};
use crate::disk::RUSTFS_META_BUCKET;
use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
use crate::error::{Error, Result};
use crate::store_api::{ObjectInfo, ObjectOptions, PutObjReader, StorageAPI};
use http::HeaderMap;
use rustfs_config::DEFAULT_DELIMITER;
use rustfs_config::{DEFAULT_DELIMITER, RUSTFS_REGION};
use rustfs_utils::path::SLASH_SEPARATOR;
use std::collections::HashSet;
use serde_json::{Map, Value};
use std::collections::{HashMap, HashSet};
use std::sync::Arc;
use std::sync::LazyLock;
use tracing::{error, instrument, warn};
use tracing::{debug, error, info, instrument, warn};
pub const CONFIG_PREFIX: &str = "config";
const CONFIG_FILE: &str = "config.json";
@@ -136,6 +137,226 @@ fn get_config_file() -> String {
format!("{CONFIG_PREFIX}{SLASH_SEPARATOR}{CONFIG_FILE}")
}
fn storage_class_kvs_mut(cfg: &mut Config) -> &mut crate::config::KVS {
let sub_cfg = cfg.0.entry(STORAGE_CLASS_SUB_SYS.to_string()).or_insert_with(|| {
let mut section = HashMap::new();
section.insert(DEFAULT_DELIMITER.to_string(), storageclass::DEFAULT_KVS.clone());
section
});
sub_cfg
.entry(DEFAULT_DELIMITER.to_string())
.or_insert_with(|| storageclass::DEFAULT_KVS.clone())
}
fn parse_storage_class_value(value: &Value) -> Option<String> {
match value {
Value::String(v) => Some(v.trim().to_string()),
Value::Object(m) => m
.get("parity")
.and_then(Value::as_u64)
.map(|parity| if parity == 0 { String::new() } else { format!("EC:{parity}") }),
_ => None,
}
}
fn parse_inline_block_value(value: &Value) -> Option<String> {
match value {
Value::String(v) if !v.trim().is_empty() => Some(v.trim().to_string()),
Value::Number(v) => Some(v.to_string()),
_ => None,
}
}
fn apply_external_storage_class_map(cfg: &mut Config, root: &Map<String, Value>) -> bool {
let sc = root.get("storageclass").or_else(|| root.get("storage_class"));
let Some(Value::Object(sc_obj)) = sc else {
return false;
};
let mut applied = false;
let kvs = storage_class_kvs_mut(cfg);
if let Some(v) = sc_obj.get("standard").and_then(parse_storage_class_value) {
kvs.insert(storageclass::CLASS_STANDARD.to_string(), v);
applied = true;
}
if let Some(v) = sc_obj.get("rrs").and_then(parse_storage_class_value) {
kvs.insert(storageclass::CLASS_RRS.to_string(), v);
applied = true;
}
if let Some(Value::String(v)) = sc_obj.get("optimize")
&& !v.trim().is_empty()
{
kvs.insert(storageclass::OPTIMIZE.to_string(), v.clone());
applied = true;
}
if let Some(v) = sc_obj.get("inline_block").and_then(parse_inline_block_value) {
kvs.insert(storageclass::INLINE_BLOCK.to_string(), v);
applied = true;
}
applied
}
fn decode_server_config_blob(data: &[u8]) -> Result<Config> {
if let Ok(cfg) = Config::unmarshal(data) {
return Ok(cfg);
}
let value: Value = serde_json::from_slice(data)?;
let Value::Object(root) = value else {
return Err(Error::other("unrecognized external server config shape"));
};
let mut cfg = Config::new();
let has_storage = apply_external_storage_class_map(&mut cfg, &root);
let has_header = root.contains_key("version") || root.contains_key("region") || root.contains_key("credential");
if !has_storage && !has_header {
return Err(Error::other("unrecognized external server config shape"));
}
Ok(cfg)
}
fn parse_object_seed(data: &[u8]) -> Option<Map<String, Value>> {
let value: Value = serde_json::from_slice(data).ok()?;
value.as_object().cloned()
}
fn build_storageclass_object(cfg: &Config) -> Map<String, Value> {
let kvs = cfg.get_value(STORAGE_CLASS_SUB_SYS, DEFAULT_DELIMITER).unwrap_or_default();
let mut sc_obj = Map::new();
sc_obj.insert(
"standard".to_string(),
Value::String(kvs.lookup(storageclass::CLASS_STANDARD).unwrap_or_default()),
);
sc_obj.insert("rrs".to_string(), Value::String(kvs.lookup(storageclass::CLASS_RRS).unwrap_or_default()));
let optimize = kvs
.lookup(storageclass::OPTIMIZE)
.filter(|v| !v.trim().is_empty())
.unwrap_or_else(|| "availability".to_string());
sc_obj.insert("optimize".to_string(), Value::String(optimize));
if let Some(v) = kvs.lookup(storageclass::INLINE_BLOCK).filter(|v| !v.trim().is_empty()) {
sc_obj.insert("inline_block".to_string(), Value::String(v));
}
sc_obj
}
fn encode_server_config_blob(cfg: &Config, seed: Option<&[u8]>) -> Result<Vec<u8>> {
let mut root = seed.and_then(parse_object_seed).unwrap_or_default();
if !matches!(root.get("version"), Some(Value::String(v)) if !v.trim().is_empty()) {
root.insert("version".to_string(), Value::String("33".to_string()));
}
if !matches!(root.get("region"), Some(Value::String(v)) if !v.trim().is_empty()) {
root.insert("region".to_string(), Value::String(RUSTFS_REGION.to_string()));
}
let mut sc_obj = match root.remove("storageclass") {
Some(Value::Object(v)) => v,
_ => Map::new(),
};
for (k, v) in build_storageclass_object(cfg) {
sc_obj.insert(k, v);
}
root.insert("storageclass".to_string(), Value::Object(sc_obj));
root.remove("storage_class");
Ok(serde_json::to_vec(&Value::Object(root))?)
}
fn is_standard_object_server_config(data: &[u8]) -> bool {
let Ok(value) = serde_json::from_slice::<Value>(data) else {
return false;
};
let Value::Object(root) = value else {
return false;
};
matches!(root.get("version"), Some(Value::String(v)) if !v.trim().is_empty())
&& matches!(root.get("storageclass"), Some(Value::Object(_)))
&& !root.contains_key("storage_class")
}
fn configs_semantically_equal(lhs: &Config, rhs: &Config) -> bool {
build_storageclass_object(lhs) == build_storageclass_object(rhs)
}
fn is_object_not_found(err: &Error) -> bool {
*err == Error::FileNotFound || matches!(err, Error::ObjectNotFound(_, _) | Error::BucketNotFound(_))
}
pub async fn try_migrate_server_config<S: StorageAPI>(api: Arc<S>) {
let config_file = get_config_file();
match api
.get_object_info(RUSTFS_META_BUCKET, &config_file, &ObjectOptions::default())
.await
{
Ok(_) => {
debug!("server config already exists in RustFS metadata bucket, skip migration");
return;
}
Err(err) if is_object_not_found(&err) => {}
Err(err) => {
warn!("check target server config failed, skip migration: {:?}", err);
return;
}
}
let opts = ObjectOptions {
max_parity: true,
no_lock: true,
..Default::default()
};
let mut rd = match api
.get_object_reader(MIGRATING_META_BUCKET, &config_file, None, HeaderMap::new(), &opts)
.await
{
Ok(v) => v,
Err(err) => {
if !is_object_not_found(&err) {
warn!("read legacy server config failed: {:?}", err);
}
return;
}
};
let data = match rd.read_all().await {
Ok(v) if !v.is_empty() => v,
Ok(_) => {
debug!("legacy server config is empty, skip migration");
return;
}
Err(err) => {
warn!("read legacy server config body failed: {:?}", err);
return;
}
};
let cfg = match decode_server_config_blob(&data) {
Ok(v) => v,
Err(err) => {
warn!("legacy server config format is incompatible, skip migration: {:?}", err);
return;
}
};
let normalized = match encode_server_config_blob(&cfg, Some(&data)) {
Ok(v) => v,
Err(err) => {
warn!("serialize migrated server config failed, skip migration: {:?}", err);
return;
}
};
match save_config(api, &config_file, normalized).await {
Ok(()) => {
info!("Migrated compatible server config from legacy metadata bucket");
}
Err(err) => {
warn!("write migrated server config failed: {:?}", err);
}
}
}
/// Handle the situation where the configuration file does not exist, create and save a new configuration
async fn handle_missing_config<S: StorageAPI>(api: Arc<S>, context: &str) -> Result<Config> {
warn!("Configuration not found ({}): Start initializing new configuration", context);
@@ -171,7 +392,7 @@ async fn read_server_config<S: StorageAPI>(api: Arc<S>, data: &[u8]) -> Result<C
match read_config(api.clone(), &config_file).await {
Ok(cfg_data) => {
// TODO: decrypt
let cfg = Config::unmarshal(&cfg_data)?;
let cfg = decode_server_config_blob(&cfg_data)?;
return Ok(cfg.merge());
}
Err(Error::ConfigNotFound) => return handle_missing_config(api, "Read alternate configuration").await,
@@ -180,14 +401,35 @@ async fn read_server_config<S: StorageAPI>(api: Arc<S>, data: &[u8]) -> Result<C
}
// Process non-empty configuration data
let cfg = Config::unmarshal(data)?;
let cfg = decode_server_config_blob(data)?;
Ok(cfg.merge())
}
pub async fn save_server_config<S: StorageAPI>(api: Arc<S>, cfg: &Config) -> Result<()> {
let data = cfg.marshal()?;
let config_file = get_config_file();
let existing = match read_config(api.clone(), &config_file).await {
Ok(v) => Some(v),
Err(Error::ConfigNotFound) => None,
Err(err) => {
warn!("read existing server config before save failed, continue with clean output: {:?}", err);
None
}
};
if let Some(current) = existing.as_deref()
&& is_standard_object_server_config(current)
&& let Ok(decoded_current) = decode_server_config_blob(current)
&& configs_semantically_equal(&decoded_current, cfg)
{
debug!("server config unchanged and already in standard object shape, skip write");
return Ok(());
}
let data = encode_server_config_blob(cfg, existing.as_deref())?;
if existing.as_deref().is_some_and(|current| current == data.as_slice()) {
debug!("server config bytes unchanged after encode, skip write");
return Ok(());
}
save_config(api, &config_file, data).await
}
@@ -232,3 +474,84 @@ async fn apply_dynamic_config_for_sub_sys<S: StorageAPI>(cfg: &mut Config, api:
Ok(())
}
#[cfg(test)]
mod tests {
use super::{
configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
storage_class_kvs_mut,
};
use crate::config::Config;
use serde_json::Value;
#[test]
fn test_decode_server_config_accepts_legacy_hidden_if_empty_alias() {
let input = r#"{"storage_class":{"_":[{"key":"standard","value":"EC:2","hiddenIfEmpty":true}]}}"#;
let cfg = decode_server_config_blob(input.as_bytes()).expect("decode should succeed");
let kvs = cfg.get_value("storage_class", "_").expect("storage_class should exist");
assert!(kvs.0[0].hidden_if_empty);
}
#[test]
fn test_decode_server_config_accepts_missing_hidden_if_empty() {
let input = r#"{"storage_class":{"_":[{"key":"standard","value":"EC:2"}]}}"#;
let cfg = decode_server_config_blob(input.as_bytes()).expect("decode should succeed");
let kvs = cfg.get_value("storage_class", "_").expect("storage_class should exist");
assert!(!kvs.0[0].hidden_if_empty);
}
#[test]
fn test_decode_server_config_accepts_v33_object_shape() {
let input = r#"{
"version":"33",
"credential":{"accessKey":"test","secretKey":"testtesttest"},
"region":"us-east-1",
"worm":"off",
"storageclass":{"standard":"EC:2","rrs":"EC:1"},
"notify":{},
"logger":{},
"compress":{"enabled":false},
"openid":{},
"policy":{"opa":{}},
"ldapserverconfig":{}
}"#;
let cfg = decode_server_config_blob(input.as_bytes()).expect("decode should succeed");
let kvs = cfg.get_value("storage_class", "_").expect("storage_class should exist");
assert_eq!(kvs.get("standard"), "EC:2");
assert_eq!(kvs.get("rrs"), "EC:1");
assert_eq!(kvs.get("optimize"), "availability");
}
#[test]
fn test_encode_server_config_writes_external_object_shape() {
let mut cfg = Config::new();
let kvs = storage_class_kvs_mut(&mut cfg);
kvs.insert("standard".to_string(), "EC:2".to_string());
kvs.insert("rrs".to_string(), "EC:1".to_string());
let out = encode_server_config_blob(&cfg, None).expect("encode should succeed");
let v: Value = serde_json::from_slice(&out).expect("output should be json");
assert!(v.get("version").is_some(), "external object should have version");
assert!(v.get("storageclass").is_some(), "external object should have storageclass");
assert!(v.get("storage_class").is_none(), "should not write rustfs map shape");
}
#[test]
fn test_is_standard_object_server_config_detection() {
let external = br#"{"version":"33","storageclass":{"standard":"EC:2","rrs":"EC:1"}}"#;
assert!(is_standard_object_server_config(external));
let legacy = br#"{"storage_class":{"_":[{"key":"standard","value":"EC:2"}]}}"#;
assert!(!is_standard_object_server_config(legacy));
}
#[test]
fn test_configs_semantically_equal_for_equivalent_shapes() {
let external = br#"{"version":"33","storageclass":{"standard":"EC:2","rrs":"EC:1","optimize":"availability"}}"#;
let legacy = br#"{"storage_class":{"_":[{"key":"standard","value":"EC:2"},{"key":"rrs","value":"EC:1"},{"key":"optimize","value":"availability"}]}}"#;
let lhs = decode_server_config_blob(external).expect("decode external");
let rhs = decode_server_config_blob(legacy).expect("decode legacy");
assert!(configs_semantically_equal(&lhs, &rhs));
}
}
+5
View File
@@ -75,10 +75,15 @@ pub async fn init_global_config_sys(api: Arc<ECStore>) -> Result<()> {
GLOBAL_CONFIG_SYS.init(api).await
}
pub async fn try_migrate_server_config(api: Arc<ECStore>) {
com::try_migrate_server_config(api).await
}
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct KV {
pub key: String,
pub value: String,
#[serde(default, alias = "hiddenIfEmpty")]
pub hidden_if_empty: bool,
}
+7 -3
View File
@@ -697,7 +697,6 @@ impl LocalDisk {
match self.read_metadata_with_dmtime(meta_path).await {
Ok(res) => Ok(res),
Err(err) => {
warn!("read_raw: error: {:?}", err);
if err == Error::FileNotFound
&& !skip_access_checks(volume_dir.as_ref().to_string_lossy().to_string().as_str())
&& let Err(e) = access(volume_dir.as_ref()).await
@@ -1493,6 +1492,12 @@ impl DiskAPI for LocalDisk {
let erasure = &fi.erasure;
for (i, part) in fi.parts.iter().enumerate() {
let checksum_info = erasure.get_checksum_info(part.number);
let checksum_algo =
if fi.uses_legacy_checksum && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S {
rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
} else {
checksum_info.algorithm
};
let part_path = self.get_object_path(
volume,
path_join_buf(&[
@@ -1506,7 +1511,7 @@ impl DiskAPI for LocalDisk {
.bitrot_verify(
&part_path,
erasure.shard_file_size(part.size as i64) as usize,
checksum_info.algorithm,
checksum_algo,
&checksum_info.hash,
erasure.shard_size(),
)
@@ -2058,7 +2063,6 @@ impl DiskAPI for LocalDisk {
let search_version_id = fi.version_id.or(Some(Uuid::nil()));
// Check if there's an existing version with the same version_id that has a data_dir to clean up
// Note: For non-versioned buckets, fi.version_id is None, but in xl.meta it's stored as Some(Uuid::nil())
let has_old_data_dir = {
xlmeta.find_version(search_version_id).ok().and_then(|(_, ver)| {
// shard_count == 0 means no other version shares this data_dir
+1
View File
@@ -23,6 +23,7 @@ pub mod local;
pub mod os;
pub const RUSTFS_META_BUCKET: &str = ".rustfs.sys";
pub const MIGRATING_META_BUCKET: &str = ".minio.sys";
pub const RUSTFS_META_MULTIPART_BUCKET: &str = ".rustfs.sys/multipart";
pub const RUSTFS_META_TMP_BUCKET: &str = ".rustfs.sys/tmp";
pub const RUSTFS_META_TMP_DELETED_BUCKET: &str = ".rustfs.sys/tmp/.trash";
+1 -1
View File
@@ -173,7 +173,7 @@ where
}
pub fn bitrot_shard_file_size(size: usize, shard_size: usize, algo: HashAlgorithm) -> usize {
if algo != HashAlgorithm::HighwayHash256S {
if algo != HashAlgorithm::HighwayHash256S && algo != HashAlgorithm::HighwayHash256SLegacy {
return size;
}
size.div_ceil(shard_size) * algo.size() + size
+274 -117
View File
@@ -12,31 +12,12 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Erasure coding implementation using Reed-Solomon SIMD backend.
//! Erasure coding implementation using reed-solomon-erasure (GF(2^8)).
//! Supports legacy (reed-solomon-simd) for reading/healing old-version files.
//!
//! This module provides erasure coding functionality with high-performance SIMD
//! Reed-Solomon implementation:
//!
//! ## Reed-Solomon Implementation
//!
//! ### SIMD Mode (Only)
//! - **Performance**: Uses SIMD optimization for high-performance encoding/decoding
//! - **Compatibility**: Works with any shard size through SIMD implementation
//! - **Reliability**: High-performance SIMD implementation for large data processing
//! - **Use case**: Optimized for maximum performance in large data processing scenarios
//!
//! ## Example
//!
//! ```ignore
//! use rustfs_ecstore::erasure_coding::Erasure;
//!
//! let erasure = Erasure::new(4, 2, 1024); // 4 data shards, 2 parity shards, 1KB block size
//! let data = b"hello world";
//! let shards = erasure.encode_data(data).unwrap();
//! // Simulate loss and recovery...
//! ```
use bytes::{Bytes, BytesMut};
use reed_solomon_erasure::galois_8::ReedSolomon;
use reed_solomon_simd;
use smallvec::SmallVec;
use std::io;
@@ -44,132 +25,88 @@ use tokio::io::AsyncRead;
use tracing::warn;
use uuid::Uuid;
/// Reed-Solomon encoder using SIMD implementation.
pub struct ReedSolomonEncoder {
/// Legacy calc_shard_size formula: (block_size.div_ceil(data_shards) + 1) & !1
/// Matches main branch and filemeta::ErasureInfo for old-version files.
pub fn calc_shard_size_legacy(block_size: usize, data_shards: usize) -> usize {
(block_size.div_ceil(data_shards) + 1) & !1
}
/// Reed-Solomon encoder for legacy (main branch) format using reed-solomon-simd.
/// Used when decoding/encoding files with uses_legacy_checksum == true.
struct LegacyReedSolomonEncoder {
data_shards: usize,
parity_shards: usize,
// Use RwLock to ensure thread safety, implementing Send + Sync
encoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
decoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
}
impl Clone for ReedSolomonEncoder {
impl Clone for LegacyReedSolomonEncoder {
fn clone(&self) -> Self {
Self {
data_shards: self.data_shards,
parity_shards: self.parity_shards,
// Create an empty cache for the new instance instead of sharing one
encoder_cache: std::sync::RwLock::new(None),
decoder_cache: std::sync::RwLock::new(None),
}
}
}
impl ReedSolomonEncoder {
/// Create a new Reed-Solomon encoder with specified data and parity shards.
pub fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
Ok(ReedSolomonEncoder {
data_shards,
parity_shards,
impl LegacyReedSolomonEncoder {
fn new(_data_shards: usize, _parity_shards: usize) -> io::Result<Self> {
Ok(Self {
data_shards: _data_shards,
parity_shards: _parity_shards,
encoder_cache: std::sync::RwLock::new(None),
decoder_cache: std::sync::RwLock::new(None),
})
}
/// Encode data shards with parity.
pub fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
if shards_vec.is_empty() {
return Ok(());
}
let simd_result = self.encode_with_simd(&mut shards_vec);
match simd_result {
Ok(()) => Ok(()),
Err(simd_error) => {
warn!("SIMD encoding failed: {}", simd_error);
Err(simd_error)
}
}
}
fn encode_with_simd(&self, shards_vec: &mut [&mut [u8]]) -> io::Result<()> {
let shard_len = shards_vec[0].len();
// Get or create encoder
let mut encoder = {
let mut cache_guard = self
.encoder_cache
.write()
.map_err(|_| io::Error::other("Failed to acquire encoder cache lock"))?;
match cache_guard.take() {
Some(mut cached_encoder) => {
// Use reset method to reset existing encoder to adapt to new parameters
if let Err(e) = cached_encoder.reset(self.data_shards, self.parity_shards, shard_len) {
warn!("Failed to reset SIMD encoder: {:?}, creating new one", e);
// If reset fails, create new encoder
Some(mut cached) => {
if cached.reset(self.data_shards, self.parity_shards, shard_len).is_err() {
reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
.map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {e:?}")))?
} else {
cached_encoder
cached
}
}
None => {
// First use, create new encoder
reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
.map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {e:?}")))?
}
None => reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
.map_err(|e| io::Error::other(format!("Failed to create SIMD encoder: {e:?}")))?,
}
};
// Add original shards
for (i, shard) in shards_vec.iter().enumerate().take(self.data_shards) {
encoder
.add_original_shard(shard)
.map_err(|e| io::Error::other(format!("Failed to add shard {i}: {e:?}")))?;
}
// Encode and get recovery shards
let result = encoder
.encode()
.map_err(|e| io::Error::other(format!("SIMD encoding failed: {e:?}")))?;
// Copy recovery shards to output buffer
for (i, recovery_shard) in result.recovery_iter().enumerate() {
if i + self.data_shards < shards_vec.len() {
shards_vec[i + self.data_shards].copy_from_slice(recovery_shard);
}
}
// Return encoder to cache (encoder is automatically reset after result is dropped, can be reused)
drop(result); // Explicitly drop result to ensure encoder is reset
drop(result);
*self
.encoder_cache
.write()
.map_err(|_| io::Error::other("Failed to return encoder to cache"))? = Some(encoder);
Ok(())
}
/// Reconstruct missing shards.
pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
// Use SIMD for reconstruction
let simd_result = self.reconstruct_with_simd(shards);
match simd_result {
Ok(()) => Ok(()),
Err(simd_error) => {
warn!("SIMD reconstruction failed: {}", simd_error);
Err(simd_error)
}
}
}
fn reconstruct_with_simd(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
// Find a valid shard to determine length
fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
let shard_len = shards
.iter()
.find_map(|s| s.as_ref().map(|v| v.len()))
@@ -185,7 +122,6 @@ impl ReedSolomonEncoder {
Some(mut cached_decoder) => {
if let Err(e) = cached_decoder.reset(self.data_shards, self.parity_shards, shard_len) {
warn!("Failed to reset SIMD decoder: {:?}, creating new one", e);
reed_solomon_simd::ReedSolomonDecoder::new(self.data_shards, self.parity_shards, shard_len)
.map_err(|e| io::Error::other(format!("Failed to create SIMD decoder: {e:?}")))?
} else {
@@ -197,7 +133,6 @@ impl ReedSolomonEncoder {
}
};
// Add available shards (both data and parity)
for (i, shard_opt) in shards.iter().enumerate() {
if let Some(shard) = shard_opt {
if i < self.data_shards {
@@ -217,7 +152,6 @@ impl ReedSolomonEncoder {
.decode()
.map_err(|e| io::Error::other(format!("SIMD decode error: {e:?}")))?;
// Fill in missing data shards from reconstruction result
for (i, shard_opt) in shards.iter_mut().enumerate() {
if shard_opt.is_none() && i < self.data_shards {
for (restored_index, restored_data) in result.restored_original_iter() {
@@ -240,6 +174,67 @@ impl ReedSolomonEncoder {
}
}
/// Reed-Solomon encoder using reed-solomon-erasure
pub struct ReedSolomonEncoder {
data_shards: usize,
parity_shards: usize,
encoder: Option<ReedSolomon>,
}
impl Clone for ReedSolomonEncoder {
fn clone(&self) -> Self {
Self {
data_shards: self.data_shards,
parity_shards: self.parity_shards,
encoder: self.encoder.clone(),
}
}
}
impl ReedSolomonEncoder {
/// Create a new Reed-Solomon encoder with specified data and parity shards.
pub fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
let encoder = if parity_shards > 0 {
ReedSolomon::new(data_shards, parity_shards)
.map_err(|e| io::Error::other(format!("Failed to create Reed-Solomon encoder: {e:?}")))
.map(Some)?
} else {
None
};
Ok(ReedSolomonEncoder {
data_shards,
parity_shards,
encoder,
})
}
/// Encode data shards with parity.
pub fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
if shards_vec.is_empty() {
return Ok(());
}
if let Some(ref rs) = self.encoder {
rs.encode(&mut shards_vec)
.map_err(|e| io::Error::other(format!("Reed-Solomon encode failed: {e:?}")))
} else {
Ok(())
}
}
/// Reconstruct missing shards.
pub fn reconstruct(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
if let Some(ref rs) = self.encoder {
rs.reconstruct_data(shards)
.map_err(|e| io::Error::other(format!("Reed-Solomon reconstruct failed: {e:?}")))
} else {
Ok(())
}
}
}
/// Erasure coding utility for data reliability using Reed-Solomon codes.
///
/// This struct provides encoding and decoding of data into data and parity shards.
@@ -262,24 +257,41 @@ impl ReedSolomonEncoder {
/// let shards = erasure.encode_data(data).unwrap();
/// // Simulate loss and recovery...
/// ```
#[derive(Default)]
pub struct Erasure {
pub data_shards: usize,
pub parity_shards: usize,
encoder: Option<ReedSolomonEncoder>,
legacy_encoder: Option<LegacyReedSolomonEncoder>,
pub block_size: usize,
uses_legacy: bool,
_id: Uuid,
_buf: Vec<u8>,
}
impl Default for Erasure {
fn default() -> Self {
Self {
data_shards: 0,
parity_shards: 0,
encoder: None,
legacy_encoder: None,
block_size: 0,
uses_legacy: false,
_id: Uuid::nil(),
_buf: vec![],
}
}
}
impl Clone for Erasure {
fn clone(&self) -> Self {
Self {
data_shards: self.data_shards,
parity_shards: self.parity_shards,
encoder: self.encoder.clone(),
legacy_encoder: self.legacy_encoder.clone(),
block_size: self.block_size,
uses_legacy: self.uses_legacy,
_id: Uuid::new_v4(), // Generate new ID for clone
_buf: vec![0u8; self.block_size],
}
@@ -287,28 +299,44 @@ impl Clone for Erasure {
}
pub fn calc_shard_size(block_size: usize, data_shards: usize) -> usize {
(block_size.div_ceil(data_shards) + 1) & !1
block_size.div_ceil(data_shards)
}
impl Erasure {
/// Create a new Erasure instance.
/// Create a new Erasure instance
///
/// # Arguments
/// * `data_shards` - Number of data shards.
/// * `parity_shards` - Number of parity shards.
/// * `block_size` - Block size for each shard.
pub fn new(data_shards: usize, parity_shards: usize, block_size: usize) -> Self {
let encoder = if parity_shards > 0 {
Self::new_with_options(data_shards, parity_shards, block_size, false)
}
/// Create a new Erasure instance with legacy format support.
///
/// When `uses_legacy` is true, uses main-branch shard_size formula and reed-solomon-simd
/// for decode/reconstruct (for reading and healing old-version files).
pub fn new_with_options(data_shards: usize, parity_shards: usize, block_size: usize, uses_legacy: bool) -> Self {
let encoder = if !uses_legacy && parity_shards > 0 {
Some(ReedSolomonEncoder::new(data_shards, parity_shards).unwrap())
} else {
None
};
let legacy_encoder = if uses_legacy && parity_shards > 0 {
Some(LegacyReedSolomonEncoder::new(data_shards, parity_shards).unwrap())
} else {
None
};
Erasure {
data_shards,
parity_shards,
block_size,
encoder,
legacy_encoder,
uses_legacy,
_id: Uuid::new_v4(),
_buf: vec![0u8; block_size],
}
@@ -323,28 +351,29 @@ impl Erasure {
/// A vector of encoded shards as `Bytes`.
#[tracing::instrument(level = "debug", skip_all, fields(data_len=data.len()))]
pub fn encode_data(&self, data: &[u8]) -> io::Result<Vec<Bytes>> {
// let shard_size = self.shard_size();
// let total_size = shard_size * self.total_shard_count();
// Data shard count
let per_shard_size = calc_shard_size(data.len(), self.data_shards);
// Total required size
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
} else {
calc_shard_size
};
let per_shard_size = shard_size_fn(data.len(), self.data_shards);
let need_total_size = per_shard_size * self.total_shard_count();
// Create a new buffer with the required total length for all shards
let mut data_buffer = BytesMut::with_capacity(need_total_size);
// Copy source data
data_buffer.extend_from_slice(data);
data_buffer.resize(need_total_size, 0u8);
{
// EC encode, the result will be written into data_buffer
let data_slices: SmallVec<[&mut [u8]; 16]> = data_buffer.chunks_exact_mut(per_shard_size).collect();
// Only do EC if parity_shards > 0
if self.parity_shards > 0 {
if let Some(encoder) = self.encoder.as_ref() {
if self.uses_legacy {
if let Some(encoder) = self.legacy_encoder.as_ref() {
encoder.encode(data_slices)?;
} else {
warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
}
} else if let Some(encoder) = self.encoder.as_ref() {
encoder.encode(data_slices)?;
} else {
warn!("parity_shards > 0, but encoder is None");
@@ -372,7 +401,13 @@ impl Erasure {
/// Ok if reconstruction succeeds, error otherwise.
pub fn decode_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
if self.parity_shards > 0 {
if let Some(encoder) = self.encoder.as_ref() {
if self.uses_legacy {
if let Some(encoder) = self.legacy_encoder.as_ref() {
encoder.reconstruct(shards)?;
} else {
warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
}
} else if let Some(encoder) = self.encoder.as_ref() {
encoder.reconstruct(shards)?;
} else {
warn!("parity_shards > 0, but encoder is None");
@@ -395,7 +430,11 @@ impl Erasure {
/// Calculate the size of each shard.
pub fn shard_size(&self) -> usize {
calc_shard_size(self.block_size, self.data_shards)
if self.uses_legacy {
calc_shard_size_legacy(self.block_size, self.data_shards)
} else {
calc_shard_size(self.block_size, self.data_shards)
}
}
/// Calculate the total erasure file size for a given original size.
// Returns the final erasure size from the original size
@@ -408,10 +447,15 @@ impl Erasure {
}
let total_length = total_length as usize;
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
} else {
calc_shard_size
};
let num_shards = total_length / self.block_size;
let last_block_size = total_length % self.block_size;
let last_shard_size = calc_shard_size(last_block_size, self.data_shards);
let last_shard_size = shard_size_fn(last_block_size, self.data_shards);
(num_shards * self.shard_size() + last_shard_size) as i64
}
@@ -494,8 +538,7 @@ mod tests {
#[test]
fn test_shard_file_size_cases2() {
let erasure = Erasure::new(12, 4, 1024 * 1024);
assert_eq!(erasure.shard_file_size(1572864), 131074);
assert_eq!(erasure.shard_file_size(1572864), 131073);
}
#[test]
@@ -517,11 +560,14 @@ mod tests {
// Case 5: total_length > block_size, aligned
assert_eq!(erasure.shard_file_size(16), 4); // 16/8=2, last=0, 2*2+0=4
assert_eq!(erasure.shard_file_size(1248739), 312186); // 1248739/8=156092, last=3, 3 div_ceil 4=1, 156092*2+1=312185
// MinIO-compatible: 1248739/8=156092, last=3, ceil(3/4)=1, 156092*2+1=312185
assert_eq!(erasure.shard_file_size(1248739), 312185);
assert_eq!(erasure.shard_file_size(43), 12); // 43/8=5, last=3, 3 div_ceil 4=1, 5*2+1=11
// MinIO-compatible: 43/8=5, last=3, ceil(3/4)=1, 5*2+1=11
assert_eq!(erasure.shard_file_size(43), 11);
assert_eq!(erasure.shard_file_size(1572864), 393216); // 43/8=5, last=3, 3 div_ceil 4=1, 5*2+1=11
// 1572864 with block_size=8: 196608 full blocks, last=0, 196608*2+0=393216
assert_eq!(erasure.shard_file_size(1572864), 393216);
}
#[test]
@@ -601,10 +647,70 @@ mod tests {
#[test]
fn test_shard_size_and_file_size() {
let erasure = Erasure::new(4, 2, 8);
assert_eq!(erasure.shard_file_size(33), 9);
assert_eq!(erasure.shard_file_size(0), 0);
}
#[test]
fn test_legacy_shard_size_and_file_size() {
let erasure = Erasure::new_with_options(4, 2, 8, true);
assert_eq!(erasure.shard_size(), 2);
assert_eq!(calc_shard_size_legacy(8, 4), 2);
assert_eq!(calc_shard_size_legacy(1, 4), 2);
assert_eq!(erasure.shard_file_size(33), 10);
assert_eq!(erasure.shard_file_size(0), 0);
}
#[test]
fn test_legacy_encode_decode_roundtrip() {
let data_shards = 4;
let parity_shards = 2;
let block_size = 1024;
let erasure = Erasure::new_with_options(data_shards, parity_shards, block_size, true);
let data = b"Legacy encode/decode roundtrip test data with sufficient length.".repeat(20);
let encoded_shards = erasure.encode_data(&data).unwrap();
assert_eq!(encoded_shards.len(), data_shards + parity_shards);
let mut decode_input: Vec<Option<Vec<u8>>> = vec![None; data_shards + parity_shards];
for i in 0..data_shards {
decode_input[i] = Some(encoded_shards[i].to_vec());
}
erasure.decode_data(&mut decode_input).unwrap();
let mut recovered = Vec::new();
for shard in decode_input.iter().take(data_shards) {
recovered.extend_from_slice(shard.as_ref().unwrap());
}
recovered.truncate(data.len());
assert_eq!(&recovered, &data);
}
#[test]
fn test_legacy_decode_with_missing_shards() {
let data_shards = 4;
let parity_shards = 2;
let block_size = 256;
let erasure = Erasure::new_with_options(data_shards, parity_shards, block_size, true);
let data = b"Legacy decode with missing shards test.".repeat(10);
let encoded_shards = erasure.encode_data(&data).unwrap();
let mut shards_opt: Vec<Option<Vec<u8>>> = encoded_shards.iter().map(|s| Some(s.to_vec())).collect();
shards_opt[1] = None;
shards_opt[5] = None;
erasure.decode_data(&mut shards_opt).unwrap();
let mut recovered = Vec::new();
for shard in shards_opt.iter().take(data_shards) {
recovered.extend_from_slice(shard.as_ref().unwrap());
}
recovered.truncate(data.len());
assert_eq!(&recovered, &data);
}
#[test]
fn test_shard_file_offset() {
let erasure = Erasure::new(8, 8, 1024 * 1024);
@@ -887,6 +993,57 @@ mod tests {
assert_eq!(&recovered, &data);
}
/// Generates 7557 bytes identical to MinIO generateCompatTestData.
fn generate_compat_test_data(size: usize) -> Vec<u8> {
(0..size).map(|i| ((i * 7 + 13) % 256) as u8).collect()
}
/// Verifies reed-solomon-simd produces same shards.
/// Data shards (0-3) must match for MinIO to read RustFS part files.
/// Parity shards (4-5) differ: reed-solomon-simd vs klauspost use different RS encoding.
/// Run: cargo test -p rustfs-ecstore test_reed_solomon_compat
#[test]
fn test_reed_solomon_compat() {
let data = generate_compat_test_data(7557);
let erasure = Erasure::new(4, 2, 7557);
let shards = erasure.encode_data(&data).unwrap();
assert_eq!(shards.len(), 6, "expected 6 shards (4 data + 2 parity)");
// Per-shard HighwayHash
let expected_hashes: [&str; 6] = [
"fb3db9338e610cec541504ddae4b0bfd54445bcbd45318cf21f35f024240914d", // data 0
"a545269a3196e18e77ef9f5ec6e735a4f4ebe82d342db666b11a5256eb305720", // data 1
"2adbf0058f36c4cbcb5c9c16c38a6530c54198dfe504179a6f92d2349f245318", // data 2
"898e6d060b0cb4f0e830add7e1f936bc8b78442bf582283ee244a3a058602db8", // data 3
"4a20460bca044b3a777b26f2b0bcd371e3eab2f156f84778be3ccd8edd521ef2", // parity 4
"eb8ba4c0db15ca910d58d031f74e4601ba2fed62ad03ec29cadde3367ab0d415", // parity 5
];
let mut data_shards_match = true;
let mut parity_shards_match = true;
for (i, shard) in shards.iter().enumerate() {
let hash = rustfs_utils::HashAlgorithm::HighwayHash256S.hash_encode(shard);
let got = hex_simd::encode_to_string(hash.as_ref(), hex_simd::AsciiCase::Lower);
let matches = got == expected_hashes[i];
if i < 4 {
data_shards_match &= matches;
} else {
parity_shards_match &= matches;
}
if !matches {
eprintln!(
"Shard {} ({}): got {} want {}",
i,
if i < 4 { "data" } else { "parity" },
got,
expected_hashes[i]
);
}
}
assert!(data_shards_match, "Data shards (0-3) must match");
assert!(parity_shards_match, "Parity shards (4-5): reed-solomon-simd differs");
}
#[test]
fn test_simd_small_data_handling() {
let data_shards = 4;
+1 -1
View File
@@ -19,4 +19,4 @@ pub mod erasure;
pub mod heal;
pub use bitrot::*;
pub use erasure::{Erasure, ReedSolomonEncoder, calc_shard_size};
pub use erasure::{Erasure, ReedSolomonEncoder, calc_shard_size, calc_shard_size_legacy};
+698 -62
View File
@@ -13,6 +13,17 @@
// limitations under the License.
use crate::bucket::versioning_sys::BucketVersioningSys;
use crate::bucket::{
lifecycle::{
bucket_lifecycle_audit::LcEventSrc,
bucket_lifecycle_ops::{
LifecycleOps, apply_expiry_on_transitioned_object, apply_expiry_rule, eval_action_from_lifecycle,
},
lifecycle::IlmAction,
},
metadata_sys,
object_lock::objectlock_sys::BucketObjectLockSys,
};
use crate::cache_value::metacache_set::{ListPathRawOptions, list_path_raw};
use crate::config::com::{CONFIG_PREFIX, read_config, save_config};
use crate::data_usage::DATA_USAGE_CACHE_NAME;
@@ -30,25 +41,32 @@ use crate::store_api::{
BucketOperations, BucketOptions, CompletePart, GetObjectReader, HealOperations, MakeBucketOptions, MultipartOperations,
ObjectIO, ObjectOperations, ObjectOptions, PutObjReader, StorageAPI,
};
use crate::{sets::Sets, store::ECStore};
use crate::{global::GLOBAL_LifecycleSys, sets::Sets, store::ECStore};
use byteorder::{ByteOrder, LittleEndian, WriteBytesExt};
use bytes::Bytes;
use futures::future::BoxFuture;
use http::HeaderMap;
use rmp_serde::{Deserializer, Serializer};
use rustfs_common::defer;
use rustfs_common::heal_channel::HealOpts;
use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
use rustfs_rio::{HashReader, WarpReader};
use rustfs_filemeta::{FileInfoVersions, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
use rustfs_rio::{EtagResolvable, HashReader, HashReaderDetector, Index, Reader, TryGetIndex, WarpReader};
use rustfs_utils::path::{SLASH_SEPARATOR, encode_dir_object, path_join};
use rustfs_workers::workers::Workers;
use s3s::dto::{BucketLifecycleConfiguration, DefaultRetention, ReplicationConfiguration};
use serde::{Deserialize, Serialize};
use std::collections::{HashMap, HashSet};
use std::fmt::Display;
use std::io::{Cursor, Write};
use std::path::PathBuf;
use std::sync::Arc;
use std::pin::Pin;
use std::sync::{
Arc,
atomic::{AtomicUsize, Ordering},
};
use std::task::{Context, Poll};
use time::{Duration, OffsetDateTime};
use tokio::io::{AsyncReadExt, BufReader};
use tokio::io::{AsyncRead, AsyncReadExt, BufReader, ReadBuf};
use tokio_util::sync::CancellationToken;
use tracing::{debug, error, info, warn};
@@ -75,6 +93,147 @@ pub struct PoolMeta {
pub dont_save: bool,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
struct PersistedPoolMeta {
pub version: u16,
pub pools: Vec<PersistedPoolStatus>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
struct PersistedPoolStatus {
#[serde(rename = "id")]
pub id: usize,
#[serde(rename = "cmdline")]
pub cmd_line: String,
#[serde(rename = "lastUpdate", with = "time::serde::rfc3339")]
pub last_update: OffsetDateTime,
#[serde(rename = "decommissionInfo")]
pub decommission: Option<PersistedPoolDecommissionInfo>,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
struct PersistedPoolDecommissionInfo {
#[serde(rename = "startTime", with = "time::serde::rfc3339::option")]
pub start_time: Option<OffsetDateTime>,
#[serde(rename = "startSize")]
pub start_size: usize,
#[serde(rename = "totalSize")]
pub total_size: usize,
#[serde(rename = "currentSize")]
pub current_size: usize,
#[serde(rename = "complete")]
pub complete: bool,
#[serde(rename = "failed")]
pub failed: bool,
#[serde(rename = "canceled")]
pub canceled: bool,
#[serde(rename = "queuedBuckets", default)]
pub queued_buckets: Vec<String>,
#[serde(rename = "decommissionedBuckets", default)]
pub decommissioned_buckets: Vec<String>,
#[serde(rename = "bucket", default)]
pub bucket: String,
#[serde(rename = "prefix", default)]
pub prefix: String,
#[serde(rename = "object", default)]
pub object: String,
#[serde(rename = "objectsDecommissioned")]
pub items_decommissioned: usize,
#[serde(rename = "objectsDecommissionedFailed")]
pub items_decommission_failed: usize,
#[serde(rename = "bytesDecommissioned")]
pub bytes_done: usize,
#[serde(rename = "bytesDecommissionedFailed")]
pub bytes_failed: usize,
}
impl From<PersistedPoolMeta> for PoolMeta {
fn from(value: PersistedPoolMeta) -> Self {
Self {
version: value.version,
pools: value.pools.into_iter().map(Into::into).collect(),
dont_save: false,
}
}
}
impl From<PersistedPoolStatus> for PoolStatus {
fn from(value: PersistedPoolStatus) -> Self {
Self {
id: value.id,
cmd_line: value.cmd_line,
last_update: value.last_update,
decommission: value.decommission.map(Into::into),
}
}
}
impl From<PersistedPoolDecommissionInfo> for PoolDecommissionInfo {
fn from(value: PersistedPoolDecommissionInfo) -> Self {
Self {
start_time: value.start_time,
start_size: value.start_size,
total_size: value.total_size,
current_size: value.current_size,
complete: value.complete,
failed: value.failed,
canceled: value.canceled,
queued_buckets: value.queued_buckets,
decommissioned_buckets: value.decommissioned_buckets,
bucket: value.bucket,
prefix: value.prefix,
object: value.object,
items_decommissioned: value.items_decommissioned,
items_decommission_failed: value.items_decommission_failed,
bytes_done: value.bytes_done,
bytes_failed: value.bytes_failed,
}
}
}
impl From<&PoolMeta> for PersistedPoolMeta {
fn from(value: &PoolMeta) -> Self {
Self {
version: value.version,
pools: value.pools.iter().map(Into::into).collect(),
}
}
}
impl From<&PoolStatus> for PersistedPoolStatus {
fn from(value: &PoolStatus) -> Self {
Self {
id: value.id,
cmd_line: value.cmd_line.clone(),
last_update: value.last_update,
decommission: value.decommission.as_ref().map(Into::into),
}
}
}
impl From<&PoolDecommissionInfo> for PersistedPoolDecommissionInfo {
fn from(value: &PoolDecommissionInfo) -> Self {
Self {
start_time: value.start_time,
start_size: value.start_size,
total_size: value.total_size,
current_size: value.current_size,
complete: value.complete,
failed: value.failed,
canceled: value.canceled,
queued_buckets: value.queued_buckets.clone(),
decommissioned_buckets: value.decommissioned_buckets.clone(),
bucket: value.bucket.clone(),
prefix: value.prefix.clone(),
object: value.object.clone(),
items_decommissioned: value.items_decommissioned,
items_decommission_failed: value.items_decommission_failed,
bytes_done: value.bytes_done,
bytes_failed: value.bytes_failed,
}
}
}
impl PoolMeta {
pub fn new(pools: &[Arc<Sets>], prev_meta: &PoolMeta) -> Self {
let mut new_meta = Self {
@@ -144,8 +303,8 @@ impl PoolMeta {
}
let mut buf = Deserializer::new(Cursor::new(&data[4..]));
let meta: PoolMeta = Deserialize::deserialize(&mut buf)?;
*self = meta;
let meta: PersistedPoolMeta = Deserialize::deserialize(&mut buf)?;
*self = meta.into();
if self.version != POOL_META_VERSION {
return Err(Error::other(format!("unexpected PoolMeta version: {}", self.version)));
@@ -161,7 +320,7 @@ impl PoolMeta {
data.write_u16::<LittleEndian>(POOL_META_FORMAT)?;
data.write_u16::<LittleEndian>(POOL_META_VERSION)?;
let mut buf = Vec::new();
self.serialize(&mut Serializer::new(&mut buf))?;
PersistedPoolMeta::from(self).serialize(&mut Serializer::new(&mut buf))?;
data.write_all(&buf)?;
for pool in pools {
@@ -178,7 +337,6 @@ impl PoolMeta {
stats.last_update = OffsetDateTime::now_utc();
let mut pd = d.clone();
pd.start_time = None;
pd.canceled = true;
pd.failed = false;
pd.complete = false;
@@ -202,7 +360,6 @@ impl PoolMeta {
stats.last_update = OffsetDateTime::now_utc();
let mut pd = d.clone();
pd.start_time = None;
pd.canceled = false;
pd.failed = true;
pd.complete = false;
@@ -226,7 +383,6 @@ impl PoolMeta {
stats.last_update = OffsetDateTime::now_utc();
let mut pd = d.clone();
pd.start_time = None;
pd.canceled = false;
pd.failed = false;
pd.complete = true;
@@ -576,6 +732,132 @@ impl Display for DecomBucketInfo {
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum DecommissionFinalState {
Complete,
Failed,
}
fn determine_decommission_final_state(items_failed: usize, was_cancelled: bool) -> DecommissionFinalState {
if items_failed > 0 || was_cancelled {
DecommissionFinalState::Failed
} else {
DecommissionFinalState::Complete
}
}
fn remaining_versions_after_decommission(fivs: &FileInfoVersions) -> usize {
fivs.versions.iter().filter(|version| !version.deleted).count()
}
fn decommission_delete_marker_opts(
version: &rustfs_filemeta::FileInfo,
version_id: Option<String>,
src_pool_idx: usize,
) -> ObjectOptions {
ObjectOptions {
versioned: true,
version_id,
mod_time: version.mod_time,
src_pool_idx,
data_movement: true,
delete_marker: true,
skip_decommissioned: true,
delete_replication: version.replication_state_internal.clone(),
..Default::default()
}
}
async fn should_skip_lifecycle_for_decommission(
store: Arc<ECStore>,
bucket: &str,
version: &rustfs_filemeta::FileInfo,
lifecycle_config: Option<&BucketLifecycleConfiguration>,
lock_retention: Option<DefaultRetention>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
apply_actions: bool,
) -> bool {
let Some(lifecycle_config) = lifecycle_config else {
return false;
};
let versioned = BucketVersioningSys::prefix_enabled(bucket, &version.name).await;
let object_info = crate::store_api::ObjectInfo::from_file_info(version, bucket, &version.name, versioned);
let event = eval_action_from_lifecycle(lifecycle_config, lock_retention, replication_config, &object_info).await;
match event.action {
IlmAction::DeleteRestoredAction | IlmAction::DeleteRestoredVersionAction => {
if apply_actions && object_info.is_remote() {
let _ = apply_expiry_on_transitioned_object(store, &object_info, &event, &LcEventSrc::Decom).await;
}
false
}
IlmAction::DeleteAction
| IlmAction::DeleteVersionAction
| IlmAction::DeleteAllVersionsAction
| IlmAction::DelMarkerDeleteAllVersionsAction => {
if apply_actions {
let _ = apply_expiry_rule(&event, &LcEventSrc::Decom, &object_info).await;
}
true
}
_ => false,
}
}
struct IndexedDecommissionReader<R> {
inner: R,
index: Option<Index>,
}
impl<R> IndexedDecommissionReader<R> {
fn new(inner: R, index: Option<Index>) -> Self {
Self { inner, index }
}
}
impl<R: AsyncRead + Unpin + Send + Sync> AsyncRead for IndexedDecommissionReader<R> {
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
Pin::new(&mut self.inner).poll_read(cx, buf)
}
}
impl<R: AsyncRead + Unpin + Send + Sync> EtagResolvable for IndexedDecommissionReader<R> {}
impl<R: AsyncRead + Unpin + Send + Sync> HashReaderDetector for IndexedDecommissionReader<R> {}
impl<R: AsyncRead + Unpin + Send + Sync> TryGetIndex for IndexedDecommissionReader<R> {
fn try_get_index(&self) -> Option<&Index> {
self.index.as_ref()
}
}
impl<R: AsyncRead + Unpin + Send + Sync> Reader for IndexedDecommissionReader<R> {}
fn decode_part_index(index: Option<&Bytes>) -> Option<Index> {
let bytes = index?;
let mut decoded = Index::new();
if decoded.load(bytes.as_ref()).is_ok() {
Some(decoded)
} else {
None
}
}
fn put_obj_reader_from_chunk(chunk: Vec<u8>, size: i64, actual_size: i64, index: Option<Index>) -> Result<PutObjReader> {
use sha2::{Digest, Sha256};
let sha256hex = if !chunk.is_empty() {
Some(hex_simd::encode_to_string(Sha256::digest(&chunk), hex_simd::AsciiCase::Lower))
} else {
None
};
let reader = IndexedDecommissionReader::new(WarpReader::new(Cursor::new(chunk)), index);
let hash_reader = HashReader::new(Box::new(reader), size, actual_size, None, sha256hex, false)?;
Ok(PutObjReader::new(hash_reader))
}
impl ECStore {
pub async fn status(&self, idx: usize) -> Result<PoolStatus> {
let space_info = self.get_decommission_pool_space_info(idx).await?;
@@ -621,13 +903,18 @@ impl ECStore {
return Err(Error::other("InvalidArgument"));
}
let Some(has_canceler) = self.decommission_cancelers.get(idx) else {
return Err(Error::other("InvalidArgument"));
};
let canceler = {
let mut cancelers = self.decommission_cancelers.write().await;
let Some(slot) = cancelers.get_mut(idx) else {
return Err(Error::other("InvalidArgument"));
};
if has_canceler.is_none() {
return Err(StorageError::DecommissionNotStarted);
}
let Some(canceler) = slot.take() else {
return Err(StorageError::DecommissionNotStarted);
};
canceler
};
let mut lock = self.pool_meta.write().await;
if lock.decommission_cancel(idx) {
@@ -640,6 +927,8 @@ impl ECStore {
}
}
canceler.cancel();
Ok(())
}
pub async fn is_decommission_running(&self) -> bool {
@@ -684,8 +973,8 @@ impl ECStore {
Ok(())
}
#[allow(unused_assignments)]
#[tracing::instrument(skip(self, set, wk, rcfg))]
#[allow(unused_assignments, clippy::too_many_arguments)]
#[tracing::instrument(skip(self, set, wk, lifecycle_config, lock_retention, replication_config))]
async fn decommission_entry(
self: &Arc<Self>,
idx: usize,
@@ -693,7 +982,9 @@ impl ECStore {
bucket: String,
set: Arc<SetDisks>,
wk: Arc<Workers>,
rcfg: Option<String>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lock_retention: Option<DefaultRetention>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
) {
warn!("decommission_entry: {} {}", &bucket, &entry.name);
wk.give().await;
@@ -713,12 +1004,27 @@ impl ECStore {
fivs.versions.sort_by(|a, b| b.mod_time.cmp(&a.mod_time));
let mut decommissioned: usize = 0;
let expired: usize = 0;
let mut expired: usize = 0;
for version in fivs.versions.iter() {
// TODO: filterLifecycle
if should_skip_lifecycle_for_decommission(
self.clone(),
&bucket,
version,
lifecycle_config.as_ref(),
lock_retention.clone(),
replication_config.clone(),
true,
)
.await
{
expired += 1;
decommissioned += 1;
continue;
}
let remaining_versions = fivs.versions.len() - expired;
if version.deleted && remaining_versions == 1 && rcfg.is_none() {
if version.deleted && remaining_versions == 1 && replication_config.is_none() {
//
decommissioned += 1;
info!("decommission_pool: DELETE marked object with no other non-current versions will be skipped");
@@ -731,25 +1037,19 @@ impl ECStore {
let mut failure = false;
let mut error = None;
if version.deleted {
// TODO: other params
if let Err(err) = self
.delete_object(
bucket.as_str(),
&version.name,
ObjectOptions {
versioned: true,
version_id: version_id.clone(),
mod_time: version.mod_time,
src_pool_idx: idx,
data_movement: true,
delete_marker: true,
skip_decommissioned: true,
..Default::default()
},
decommission_delete_marker_opts(version, version_id.clone(), idx),
)
.await
{
if is_err_object_not_found(&err) || is_err_version_not_found(&err) || is_err_data_movement_overwrite(&err) {
warn!(
"decommission_pool: ignore delete-marker copy for {}/{} version {:?}: {:?}",
&bucket, &version.name, &version_id, &err
);
ignore = true;
continue;
}
@@ -776,7 +1076,33 @@ impl ECStore {
for _i in 0..3 {
if version.is_remote() {
// TODO: DecomTieredObject
if let Err(err) = self
.decommission_tiered_object(
bucket.as_str(),
&version.name,
version,
&ObjectOptions {
version_id: version_id.clone(),
mod_time: version.mod_time,
user_defined: version.metadata.clone(),
src_pool_idx: idx,
data_movement: true,
..Default::default()
},
)
.await
{
if is_err_object_not_found(&err) || is_err_version_not_found(&err) || is_err_data_movement_overwrite(&err)
{
ignore = true;
break;
}
failure = true;
error!("decommission_pool: decommission_tiered_object err {:?}", &err);
error = Some(err);
}
break;
}
let bucket = bucket.clone();
@@ -847,7 +1173,8 @@ impl ECStore {
}
{
self.pool_meta.write().await.count_item(idx, decommissioned, failure);
let size = usize::try_from(version.size).unwrap_or_default();
self.pool_meta.write().await.count_item(idx, size, failure);
}
if failure {
@@ -872,6 +1199,14 @@ impl ECStore {
.await
{
error!("decommission_pool: delete_object err {:?}", &err);
} else if decommissioned != fivs.versions.len() {
warn!(
"decommission_pool: source object retained for {}/{} because only {}/{} versions were decommissioned",
&bucket,
&entry.name,
decommissioned,
fivs.versions.len()
);
}
{
@@ -903,16 +1238,19 @@ impl ECStore {
) -> Result<()> {
let wk = Workers::new(pool.disk_set.len() * 2).map_err(Error::other)?;
// let mut vc = None;
// replication
let rcfg: Option<String> = None;
let mut lifecycle_config = None;
let mut lock_retention = None;
let mut replication_config = None;
if bi.name != RUSTFS_META_BUCKET {
let _versioning = BucketVersioningSys::get(&bi.name).await?;
// vc = Some(versioning);
// TODO: LifecycleSys
// TODO: BucketObjectLockSys
// TODO: ReplicationConfig
let _ = BucketVersioningSys::get(&bi.name).await?;
lifecycle_config = GLOBAL_LifecycleSys.get(&bi.name).await;
lock_retention = BucketObjectLockSys::get(&bi.name).await;
replication_config = match metadata_sys::get_replication_config(&bi.name).await {
Ok(config) => Some(config),
Err(Error::ConfigNotFound) => None,
Err(err) => return Err(err),
};
}
for (set_idx, set) in pool.disk_set.iter().enumerate() {
@@ -925,17 +1263,22 @@ impl ECStore {
let bucket = bi.name.clone();
let wk = wk.clone();
let set = set.clone();
let rcfg = rcfg.clone();
let lifecycle_config = lifecycle_config.clone();
let lock_retention = lock_retention.clone();
let replication_config = replication_config.clone();
move |entry: MetaCacheEntry| {
let this = this.clone();
let bucket = bucket.clone();
let wk = wk.clone();
let set = set.clone();
let rcfg = rcfg.clone();
let lifecycle_config = lifecycle_config.clone();
let lock_retention = lock_retention.clone();
let replication_config = replication_config.clone();
Box::pin(async move {
wk.take().await;
this.decommission_entry(idx, entry, bucket, set, wk, rcfg).await
this.decommission_entry(idx, entry, bucket, set, wk, lifecycle_config, lock_retention, replication_config)
.await
})
}
});
@@ -988,7 +1331,15 @@ impl ECStore {
#[tracing::instrument(skip(self, rx))]
pub async fn do_decommission_in_routine(self: &Arc<Self>, rx: CancellationToken, idx: usize) {
if let Err(err) = self.decommission_in_background(rx, idx).await {
let decommission_token = rx.child_token();
{
let mut cancelers = self.decommission_cancelers.write().await;
if let Some(slot) = cancelers.get_mut(idx) {
*slot = Some(decommission_token.clone());
}
}
if let Err(err) = self.decommission_in_background(decommission_token.clone(), idx).await {
error!("decom err {:?}", &err);
if let Err(er) = self.decommission_failed(idx).await {
error!("decom failed err {:?}", &er);
@@ -1001,29 +1352,49 @@ impl ECStore {
warn!("decommission: decommission_in_background complete {}", idx);
let (failed, cmd_line) = {
let (final_state, cmd_line) = {
let pool_meta = self.pool_meta.read().await;
let failed = {
let final_state = {
if let Some(info) = &pool_meta.pools[idx].decommission {
info.items_decommission_failed > 0
determine_decommission_final_state(info.items_decommission_failed, info.canceled)
} else {
false
DecommissionFinalState::Failed
}
};
let cmd_line = pool_meta.pools[idx].cmd_line.clone();
(failed, cmd_line)
(final_state, cmd_line)
};
if !failed {
let mut completed_successfully = false;
if final_state == DecommissionFinalState::Complete {
warn!("Decommissioning complete for pool {}, verifying for any pending objects", cmd_line);
if let Err(er) = self.decommission_failed(idx).await {
error!("decom failed err {:?}", &er);
if let Err(err) = self.check_after_decommission(idx).await {
error!("decom post-check err {:?}", &err);
if let Err(er) = self.decommission_failed(idx).await {
error!("decom failed err {:?}", &er);
}
} else if let Err(er) = self.complete_decommission(idx).await {
error!("decom complete err {:?}", &er);
} else {
completed_successfully = true;
}
} else if let Err(er) = self.complete_decommission(idx).await {
error!("decom complete err {:?}", &er);
} else if let Err(er) = self.decommission_failed(idx).await {
error!("decom failed err {:?}", &er);
}
warn!("Decommissioning complete for pool {}", cmd_line);
{
let mut cancelers = self.decommission_cancelers.write().await;
if let Some(slot) = cancelers.get_mut(idx) {
*slot = None;
}
}
if completed_successfully {
warn!("Decommissioning complete for pool {}", cmd_line);
} else {
warn!("Decommissioning finished in failed state for pool {}", cmd_line);
}
}
#[tracing::instrument(skip(self))]
@@ -1043,6 +1414,14 @@ impl ECStore {
}
}
let canceler = {
let mut cancelers = self.decommission_cancelers.write().await;
cancelers.get_mut(idx).and_then(Option::take)
};
if let Some(canceler) = canceler {
canceler.cancel();
}
Ok(())
}
@@ -1061,6 +1440,14 @@ impl ECStore {
}
}
let canceler = {
let mut cancelers = self.decommission_cancelers.write().await;
cancelers.get_mut(idx).and_then(Option::take)
};
if let Some(canceler) = canceler {
canceler.cancel();
}
Ok(())
}
@@ -1190,6 +1577,94 @@ impl ECStore {
Ok(ret)
}
async fn check_after_decommission(self: &Arc<Self>, idx: usize) -> Result<()> {
let buckets = self.get_buckets_to_decommission().await?;
let pool = self.pools[idx].clone();
for set in &pool.disk_set {
for bucket_info in &buckets {
let mut lifecycle_config = None;
let mut lock_retention = None;
let mut replication_config = None;
if bucket_info.name != RUSTFS_META_BUCKET {
lifecycle_config = GLOBAL_LifecycleSys.get(&bucket_info.name).await;
lock_retention = BucketObjectLockSys::get(&bucket_info.name).await;
replication_config = match metadata_sys::get_replication_config(&bucket_info.name).await {
Ok(config) => Some(config),
Err(Error::ConfigNotFound) => None,
Err(err) => return Err(err),
};
}
let versions_found = Arc::new(AtomicUsize::new(0));
let versions_found_cb = versions_found.clone();
let bucket_name = bucket_info.name.clone();
let lifecycle_config_cb = lifecycle_config.clone();
let lock_retention_cb = lock_retention.clone();
let replication_config_cb = replication_config.clone();
let store = Arc::clone(self);
let callback: ListCallback = Arc::new(move |entry: MetaCacheEntry| {
let versions_found = versions_found_cb.clone();
let bucket_name = bucket_name.clone();
let lifecycle_config = lifecycle_config_cb.clone();
let lock_retention = lock_retention_cb.clone();
let replication_config = replication_config_cb.clone();
let store = Arc::clone(&store);
Box::pin(async move {
if !entry.is_object() {
return;
}
if bucket_name == RUSTFS_META_BUCKET && entry.name.contains(DATA_USAGE_CACHE_NAME) {
return;
}
let Ok(fivs) = entry.file_info_versions(&bucket_name) else {
return;
};
let mut remaining = 0;
for version in &fivs.versions {
if version.deleted {
continue;
}
if should_skip_lifecycle_for_decommission(
Arc::clone(&store),
&bucket_name,
version,
lifecycle_config.as_ref(),
lock_retention.clone(),
replication_config.clone(),
false,
)
.await
{
continue;
}
remaining += 1;
}
versions_found.fetch_add(remaining, Ordering::Relaxed);
})
});
set.list_objects_to_decommission(CancellationToken::new(), bucket_info.clone(), callback)
.await?;
let versions_found = versions_found.load(Ordering::Relaxed);
if versions_found > 0 {
return Err(Error::other(format!(
"at least {versions_found} object(s)/version(s) were found in bucket `{}` after decommissioning",
bucket_info.name
)));
}
}
}
Ok(())
}
#[tracing::instrument(skip(self, rd))]
async fn decommission_object(self: Arc<Self>, pool_idx: usize, bucket: String, rd: GetObjectReader) -> Result<()> {
warn!("decommission_object: start {} {}", &bucket, &rd.object_info.name);
@@ -1238,7 +1713,10 @@ impl ECStore {
reader.read_exact(&mut chunk).await?;
let mut data = PutObjReader::from_vec(chunk);
let part_size = i64::try_from(part.size).map_err(|_| Error::other("part size overflow"))?;
let part_actual_size = if part.actual_size > 0 { part.actual_size } else { part_size };
let index = decode_part_index(part.index.as_ref());
let mut data = put_obj_reader_from_chunk(chunk, part_size, part_actual_size, index)?;
let pi = match self
.put_object_part(
@@ -1294,8 +1772,13 @@ impl ECStore {
return Ok(());
}
let reader = BufReader::new(rd.stream);
let hrd = HashReader::new(Box::new(WarpReader::new(reader)), object_info.size, object_info.size, None, None, false)?;
let actual_size = object_info.get_actual_size()?;
let index = object_info
.parts
.first()
.and_then(|part| decode_part_index(part.index.as_ref()));
let reader = IndexedDecommissionReader::new(WarpReader::new(BufReader::new(rd.stream)), index);
let hrd = HashReader::new(Box::new(reader), object_info.size, actual_size, object_info.etag.clone(), None, false)?;
let mut data = PutObjReader::new(hrd);
if let Err(err) = self
@@ -1325,6 +1808,159 @@ impl ECStore {
}
}
#[cfg(test)]
#[allow(clippy::items_after_test_module)]
mod tests {
use super::*;
#[test]
fn determine_decommission_final_state_marks_failures_and_cancellations() {
assert_eq!(determine_decommission_final_state(0, false), DecommissionFinalState::Complete);
assert_eq!(determine_decommission_final_state(1, false), DecommissionFinalState::Failed);
assert_eq!(determine_decommission_final_state(0, true), DecommissionFinalState::Failed);
}
#[test]
fn remaining_versions_after_decommission_ignores_delete_markers() {
let fivs = FileInfoVersions {
versions: vec![
rustfs_filemeta::FileInfo {
deleted: false,
size: 128,
..Default::default()
},
rustfs_filemeta::FileInfo {
deleted: true,
size: 0,
..Default::default()
},
],
..Default::default()
};
assert_eq!(remaining_versions_after_decommission(&fivs), 1);
}
#[test]
fn decommission_delete_marker_opts_preserves_replication_state() {
let mod_time = OffsetDateTime::now_utc();
let version = rustfs_filemeta::FileInfo {
mod_time: Some(mod_time),
replication_state_internal: Some(rustfs_filemeta::ReplicationState {
replica_status: rustfs_filemeta::ReplicationStatusType::Replica,
delete_marker: true,
replicate_decision_str: "existing".to_string(),
..Default::default()
}),
..Default::default()
};
let opts = decommission_delete_marker_opts(&version, Some("version-id".to_string()), 7);
let replication = opts.delete_replication.expect("replication state should be preserved");
assert!(opts.versioned);
assert!(opts.data_movement);
assert!(opts.delete_marker);
assert!(opts.skip_decommissioned);
assert_eq!(opts.src_pool_idx, 7);
assert_eq!(opts.version_id.as_deref(), Some("version-id"));
assert_eq!(opts.mod_time, Some(mod_time));
assert_eq!(replication.replica_status, rustfs_filemeta::ReplicationStatusType::Replica);
assert!(replication.delete_marker);
assert_eq!(replication.replicate_decision_str, "existing");
}
#[test]
fn decommission_state_transitions_preserve_start_time() {
let start_time = OffsetDateTime::now_utc();
let mut pool_meta = PoolMeta {
version: POOL_META_VERSION,
pools: vec![PoolStatus {
id: 0,
cmd_line: "/tmp/pool".to_string(),
last_update: start_time,
decommission: Some(PoolDecommissionInfo {
start_time: Some(start_time),
..Default::default()
}),
}],
dont_save: true,
};
assert!(pool_meta.decommission_failed(0));
assert_eq!(
pool_meta.pools[0].decommission.as_ref().and_then(|info| info.start_time),
Some(start_time)
);
assert!(pool_meta.decommission_complete(0));
assert_eq!(
pool_meta.pools[0].decommission.as_ref().and_then(|info| info.start_time),
Some(start_time)
);
assert!(pool_meta.decommission_cancel(0));
assert_eq!(
pool_meta.pools[0].decommission.as_ref().and_then(|info| info.start_time),
Some(start_time)
);
}
#[test]
fn pool_meta_persists_decommission_resume_queues() {
let start_time = OffsetDateTime::now_utc();
let pool_meta = PoolMeta {
version: POOL_META_VERSION,
pools: vec![PoolStatus {
id: 1,
cmd_line: "/data/pool1/disk{1...4}".to_string(),
last_update: start_time,
decommission: Some(PoolDecommissionInfo {
start_time: Some(start_time),
queued_buckets: vec!["bucket-a".to_string(), "bucket-b/prefix".to_string()],
decommissioned_buckets: vec!["bucket-done".to_string()],
bucket: "bucket-b".to_string(),
prefix: "prefix".to_string(),
object: "object.txt".to_string(),
items_decommissioned: 7,
items_decommission_failed: 1,
bytes_done: 1024,
bytes_failed: 128,
..Default::default()
}),
}],
dont_save: false,
};
let mut buf = Vec::new();
PersistedPoolMeta::from(&pool_meta)
.serialize(&mut Serializer::new(&mut buf))
.expect("pool meta should serialize");
let mut deserializer = Deserializer::new(Cursor::new(&buf));
let restored: PoolMeta = PersistedPoolMeta::deserialize(&mut deserializer)
.expect("pool meta should deserialize")
.into();
let restored_decommission = restored.pools[0]
.decommission
.as_ref()
.expect("decommission info should survive round-trip");
assert_eq!(
restored_decommission.queued_buckets,
vec!["bucket-a".to_string(), "bucket-b/prefix".to_string()]
);
assert_eq!(restored_decommission.decommissioned_buckets, vec!["bucket-done".to_string()]);
assert_eq!(restored_decommission.bucket, "bucket-b");
assert_eq!(restored_decommission.prefix, "prefix");
assert_eq!(restored_decommission.object, "object.txt");
assert_eq!(restored_decommission.items_decommissioned, 7);
assert_eq!(restored_decommission.items_decommission_failed, 1);
assert_eq!(restored_decommission.bytes_done, 1024);
assert_eq!(restored_decommission.bytes_failed, 128);
}
}
// impl Fn(MetaCacheEntry) -> impl Future<Output = Result<(), Error>>
pub type ListCallback = Arc<dyn Fn(MetaCacheEntry) -> BoxFuture<'static, ()> + Send + Sync + 'static>;
+147 -25
View File
@@ -80,9 +80,12 @@ use rustfs_lock::{FastLockGuard, NamespaceLock, NamespaceLockGuard, NamespaceLoc
use rustfs_madmin::heal_commands::{HealDriveInfo, HealResultItem};
use rustfs_rio::{EtagResolvable, HashReader, HashReaderMut, TryGetIndex as _, WarpReader};
use rustfs_s3_common::EventName;
use rustfs_utils::http::RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM;
use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
use rustfs_utils::http::headers::AMZ_STORAGE_CLASS;
use rustfs_utils::http::headers::{AMZ_OBJECT_TAGGING, RESERVED_METADATA_PREFIX, RESERVED_METADATA_PREFIX_LOWER};
use rustfs_utils::http::{
SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
contains_key_str, get_header_map, get_str, insert_str, remove_header_map,
};
use rustfs_utils::{
HashAlgorithm,
crypto::hex,
@@ -136,6 +139,30 @@ pub fn get_lock_acquire_timeout() -> Duration {
Duration::from_secs(rustfs_utils::get_env_u64("RUSTFS_LOCK_ACQUIRE_TIMEOUT", 5))
}
fn build_tiered_decommission_file_info(
bucket: &str,
object: &str,
fi: &FileInfo,
disk_count: usize,
default_parity_count: usize,
storage_class: Option<&str>,
) -> (FileInfo, usize) {
let parity_drives = GLOBAL_STORAGE_CLASS
.get()
.and_then(|sc| sc.get_parity_for_sc(storage_class.unwrap_or_default()))
.unwrap_or(default_parity_count);
let data_drives = disk_count - parity_drives;
let mut write_quorum = data_drives;
if data_drives == parity_drives {
write_quorum += 1;
}
let mut updated = fi.clone();
updated.erasure = FileInfo::new([bucket, object].join("/").as_str(), data_drives, parity_drives).erasure;
(updated, write_quorum)
}
#[derive(Clone, Debug)]
pub struct SetDisks {
pub locker_owner: String,
@@ -620,7 +647,7 @@ impl ObjectIO for SetDisks {
&tmp_object,
erasure.shard_file_size(data.size()),
erasure.shard_size(),
HashAlgorithm::HighwayHash256,
HashAlgorithm::HighwayHash256S,
)
.await
{
@@ -678,8 +705,8 @@ impl ObjectIO for SetDisks {
)));
}
if user_defined.contains_key(&format!("{RESERVED_METADATA_PREFIX_LOWER}compression")) {
user_defined.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}compression-size"), w_size.to_string());
if contains_key_str(&user_defined, SUFFIX_COMPRESSION) {
insert_str(&mut user_defined, SUFFIX_COMPRESSION_SIZE, w_size.to_string());
}
let index_op = data.stream.try_get_index().map(|v| v.clone().into_vec());
@@ -765,7 +792,7 @@ impl ObjectIO for SetDisks {
.await?;
if let Some(old_dir) = op_old_dir {
self.commit_rename_data_dir(&shuffle_disks, bucket, object, &old_dir.to_string(), write_quorum)
self.commit_rename_data_dir(&online_disks, bucket, object, &old_dir.to_string(), write_quorum)
.await?;
}
@@ -1336,6 +1363,12 @@ impl ObjectOperations for SetDisks {
let mut delete_marker = opts.versioned;
if opts.version_id.is_some() {
// Decommission/rebalance may recreate a delete marker on a new pool before that
// exact version exists there, so we must still treat it as a mark-delete write.
if opts.data_movement && opts.delete_marker && !version_found {
mark_delete = true;
}
if version_found && opts.delete_marker_replication_status() == ReplicationStatusType::Replica {
mark_delete = false;
}
@@ -1889,6 +1922,68 @@ impl ObjectOperations for SetDisks {
}
}
impl SetDisks {
#[tracing::instrument(skip(self, fi, opts))]
pub(crate) async fn decommission_tiered_object(
&self,
bucket: &str,
object: &str,
fi: &FileInfo,
opts: &ObjectOptions,
) -> Result<()> {
let _lock_guard = if !opts.no_lock {
Some(
self.new_ns_lock(bucket, object)
.await?
.get_write_lock(get_lock_acquire_timeout())
.await
.map_err(|e| {
Error::other(format!(
"Failed to acquire write lock: {}",
self.format_lock_error_from_error(bucket, object, "write", &e)
))
})?,
)
} else {
None
};
let disks = self.disks.read().await.clone();
let storage_class = opts.user_defined.get(AMZ_STORAGE_CLASS).map(String::as_str);
let (fi, write_quorum) =
build_tiered_decommission_file_info(bucket, object, fi, disks.len(), self.default_parity_count, storage_class);
let parts_metadata = vec![fi.clone(); disks.len()];
let (shuffle_disks, parts_metadata) = Self::shuffle_disks_and_parts_metadata(&disks, &parts_metadata, &fi);
let mut errs = Vec::with_capacity(shuffle_disks.len());
let mut futures = Vec::with_capacity(shuffle_disks.len());
for (index, disk) in shuffle_disks.iter().enumerate() {
let mut file_info = parts_metadata[index].clone();
file_info.erasure.index = index + 1;
futures.push(async move {
if let Some(disk) = disk {
disk.write_metadata("", bucket, object, file_info).await
} else {
Err(DiskError::DiskNotFound)
}
});
}
for result in join_all(futures).await {
match result {
Ok(_) => errs.push(None),
Err(err) => errs.push(Some(err)),
}
}
if let Some(err) = reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, write_quorum) {
return Err(to_object_err(err.into(), vec![bucket, object]));
}
Ok(())
}
}
#[async_trait::async_trait]
impl ListOperations for SetDisks {
#[tracing::instrument(skip(self))]
@@ -2007,7 +2102,7 @@ impl MultipartOperations for SetDisks {
&tmp_part_path,
erasure.shard_file_size(data.size()),
erasure.shard_size(),
HashAlgorithm::HighwayHash256,
HashAlgorithm::HighwayHash256S,
)
.await
{
@@ -2454,11 +2549,11 @@ impl MultipartOperations for SetDisks {
fi.data_dir = Some(Uuid::new_v4());
if let Some(cssum) = user_defined.get(RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM)
if let Some(cssum) = get_header_map(&user_defined, SUFFIX_REPLICATION_SSEC_CRC)
&& !cssum.is_empty()
{
fi.checksum = base64_simd::STANDARD.decode_to_vec(cssum).ok().map(Bytes::from);
user_defined.remove(RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM);
fi.checksum = base64_simd::STANDARD.decode_to_vec(&cssum).ok().map(Bytes::from);
remove_header_map(&mut user_defined, SUFFIX_REPLICATION_SSEC_CRC);
}
let parts_metadata = vec![fi.clone(); disks.len()];
@@ -2809,8 +2904,8 @@ impl MultipartOperations for SetDisks {
}
}
if let Some(rc_crc) = opts.user_defined.get(RUSTFS_BUCKET_REPLICATION_SSEC_CHECKSUM) {
if let Ok(rc_crc_bytes) = base64_simd::STANDARD.decode_to_vec(rc_crc) {
if let Some(rc_crc) = get_header_map(&opts.user_defined, SUFFIX_REPLICATION_SSEC_CRC) {
if let Ok(rc_crc_bytes) = base64_simd::STANDARD.decode_to_vec(&rc_crc) {
fi.checksum = Some(Bytes::from(rc_crc_bytes));
} else {
error!("complete_multipart_upload decode rc_crc failed rc_crc={}", rc_crc);
@@ -2849,25 +2944,19 @@ impl MultipartOperations for SetDisks {
fi.metadata.insert("etag".to_owned(), etag);
if opts.replication_request {
if let Some(actual_size) = opts
.user_defined
.get(format!("{RESERVED_METADATA_PREFIX_LOWER}Actual-Object-Size").as_str())
{
if let Some(actual_size) = get_str(&opts.user_defined, SUFFIX_ACTUAL_OBJECT_SIZE_CAP) {
insert_str(&mut fi.metadata, SUFFIX_ACTUAL_SIZE, actual_size.clone());
fi.metadata
.insert(format!("{RESERVED_METADATA_PREFIX}actual-size"), actual_size.clone());
fi.metadata
.insert("x-rustfs-encryption-original-size".to_string(), actual_size.to_string());
.insert("x-rustfs-encryption-original-size".to_string(), actual_size);
}
} else {
fi.metadata
.insert(format!("{RESERVED_METADATA_PREFIX}actual-size"), object_actual_size.to_string());
insert_str(&mut fi.metadata, SUFFIX_ACTUAL_SIZE, object_actual_size.to_string());
fi.metadata
.insert("x-rustfs-encryption-original-size".to_string(), object_actual_size.to_string());
}
if fi.is_compressed() {
fi.metadata
.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}compression-size"), object_size.to_string());
insert_str(&mut fi.metadata, SUFFIX_COMPRESSION_SIZE, object_size.to_string());
}
if opts.data_movement {
@@ -2927,7 +3016,7 @@ impl MultipartOperations for SetDisks {
.await?;
if let Some(old_dir) = op_old_dir {
self.commit_rename_data_dir(&shuffle_disks, bucket, object, &old_dir.to_string(), write_quorum)
self.commit_rename_data_dir(&online_disks, bucket, object, &old_dir.to_string(), write_quorum)
.await?;
}
@@ -3356,12 +3445,18 @@ async fn disks_with_all_parts(
if (meta.data.is_some() || meta.size == 0) && !meta.parts.is_empty() {
if let Some(data) = &meta.data {
let checksum_info = meta.erasure.get_checksum_info(meta.parts[0].number);
let checksum_algo =
if meta.uses_legacy_checksum && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S {
rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
} else {
checksum_info.algorithm
};
let data_len = data.len();
let verify_err = bitrot_verify(
Box::new(Cursor::new(data.clone())),
data_len,
meta.erasure.shard_file_size(meta.size) as usize,
checksum_info.algorithm,
checksum_algo,
checksum_info.hash,
meta.erasure.shard_size(),
)
@@ -4161,6 +4256,33 @@ mod tests {
assert!(e_tag_matches("\"abc\"", "*"));
}
#[test]
fn test_build_tiered_decommission_file_info_preserves_transition_metadata() {
let version_id = Uuid::new_v4();
let transition_version_id = Uuid::new_v4();
let original = FileInfo {
version_id: Some(version_id),
transition_status: TRANSITION_COMPLETE.to_string(),
transitioned_objname: "remote/object".to_string(),
transition_tier: "WARM-TIER".to_string(),
transition_version_id: Some(transition_version_id),
erasure: FileInfo::new("old-bucket/old-object", 8, 8).erasure,
..Default::default()
};
let (updated, write_quorum) = build_tiered_decommission_file_info("bucket", "object", &original, 16, 4, None);
assert_eq!(updated.version_id, original.version_id);
assert_eq!(updated.transition_status, original.transition_status);
assert_eq!(updated.transitioned_objname, original.transitioned_objname);
assert_eq!(updated.transition_tier, original.transition_tier);
assert_eq!(updated.transition_version_id, original.transition_version_id);
assert_eq!(updated.erasure.data_blocks, 12);
assert_eq!(updated.erasure.parity_blocks, 4);
assert_eq!(write_quorum, 12);
assert_ne!(updated.erasure.distribution, original.erasure.distribution);
}
#[test]
fn test_should_prevent_write() {
let oi = ObjectInfo {
+12 -4
View File
@@ -124,11 +124,12 @@ impl SetDisks {
);
let erasure = if !latest_meta.deleted && !latest_meta.is_remote() {
// Initialize erasure coding
erasure_coding::Erasure::new(
// Initialize erasure coding; use legacy mode for old-version files
erasure_coding::Erasure::new_with_options(
latest_meta.erasure.data_blocks,
latest_meta.erasure.parity_blocks,
latest_meta.erasure.block_size,
latest_meta.uses_legacy_checksum,
)
} else {
erasure_coding::Erasure::default()
@@ -347,7 +348,14 @@ impl SetDisks {
for (part_index, part) in latest_meta.parts.iter().enumerate() {
let till_offset = erasure.shard_file_offset(0, part.size, part.size);
let checksum_algo = erasure_info.get_checksum_info(part.number).algorithm;
let checksum_info = erasure_info.get_checksum_info(part.number);
let checksum_algo = if latest_meta.uses_legacy_checksum
&& checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S
{
rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
} else {
checksum_info.algorithm
};
let mut readers = Vec::with_capacity(latest_disks.len());
let mut writers = Vec::with_capacity(out_dated_disks.len());
// let mut errors = Vec::with_capacity(out_dated_disks.len());
@@ -420,7 +428,7 @@ impl SetDisks {
]),
erasure.shard_file_size(part.size as i64),
erasure.shard_size(),
HashAlgorithm::HighwayHash256,
HashAlgorithm::HighwayHash256S,
)
.await
{
+15 -2
View File
@@ -603,7 +603,12 @@ impl SetDisks {
object, offset, length, end_offset, part_index, last_part_index, last_part_relative_offset, "Multipart read bounds"
);
let erasure = erasure_coding::Erasure::new(fi.erasure.data_blocks, fi.erasure.parity_blocks, fi.erasure.block_size);
let erasure = erasure_coding::Erasure::new_with_options(
fi.erasure.data_blocks,
fi.erasure.parity_blocks,
fi.erasure.block_size,
fi.uses_legacy_checksum,
);
let part_indices: Vec<usize> = (part_index..=last_part_index).collect();
debug!(bucket, object, ?part_indices, "Multipart part indices to stream");
@@ -648,6 +653,14 @@ impl SetDisks {
"Streaming multipart part"
);
let checksum_info = fi.erasure.get_checksum_info(part_number);
let checksum_algo =
if fi.uses_legacy_checksum && checksum_info.algorithm == rustfs_utils::HashAlgorithm::HighwayHash256S {
rustfs_utils::HashAlgorithm::HighwayHash256SLegacy
} else {
checksum_info.algorithm
};
let mut readers = Vec::with_capacity(disks.len());
let mut errors = Vec::with_capacity(disks.len());
for (idx, disk_op) in disks.iter().enumerate() {
@@ -659,7 +672,7 @@ impl SetDisks {
read_offset,
till_offset,
erasure.shard_size(),
HashAlgorithm::HighwayHash256,
checksum_algo.clone(),
skip_verify_bitrot,
)
.await
+1 -1
View File
@@ -156,7 +156,7 @@ pub struct ECStore {
// pub local_disks: Vec<DiskStore>,
pub pool_meta: RwLock<PoolMeta>,
pub rebalance_meta: RwLock<Option<RebalanceMeta>>,
pub decommission_cancelers: Vec<Option<usize>>,
pub decommission_cancelers: RwLock<Vec<Option<CancellationToken>>>,
}
// impl Clone for ECStore {
+2 -1
View File
@@ -149,7 +149,7 @@ impl ECStore {
let mut pool_meta = PoolMeta::new(&pools, &PoolMeta::default());
pool_meta.dont_save = true;
let decommission_cancelers = vec![None; pools.len()];
let decommission_cancelers = RwLock::new(vec![None; pools.len()]);
let ec = Arc::new(ECStore {
id: deployment_id.unwrap(),
disk_map,
@@ -265,6 +265,7 @@ impl ECStore {
init_background_expiry(self.clone()).await;
TransitionState::init(self.clone()).await;
crate::tier::tier::try_migrate_tiering_config(self.clone()).await;
if let Err(err) = GLOBAL_TierConfigMgr.write().await.init(self.clone()).await {
info!("TierConfigMgr init error: {}", err);
+105 -6
View File
@@ -14,7 +14,64 @@
use super::*;
fn select_data_movement_target_pool(
existing_pool_idx: Result<usize>,
src_pool_idx: usize,
delete_marker: bool,
) -> Result<Option<usize>> {
match existing_pool_idx {
Ok(pool_idx) => {
if delete_marker && pool_idx == src_pool_idx {
Ok(None)
} else {
Ok(Some(pool_idx))
}
}
Err(err) => {
if is_err_read_quorum(&err) {
return Err(StorageError::ErasureWriteQuorum);
}
if delete_marker && (is_err_object_not_found(&err) || is_err_version_not_found(&err)) {
Ok(None)
} else {
Err(err)
}
}
}
}
impl ECStore {
#[instrument(skip(self, fi, opts))]
pub(crate) async fn decommission_tiered_object(
&self,
bucket: &str,
object: &str,
fi: &rustfs_filemeta::FileInfo,
opts: &ObjectOptions,
) -> Result<()> {
check_put_object_args(bucket, object)?;
let object = encode_dir_object(object);
if self.single_pool() {
return Err(Error::other(format!("error decommissioning {bucket}/{object}")));
}
let idx = self.get_pool_idx_no_lock(bucket, &object, fi.size).await?;
if opts.data_movement && idx == opts.src_pool_idx {
return Err(StorageError::DataMovementOverwriteErr(
bucket.to_owned(),
object.to_owned(),
opts.version_id.clone().unwrap_or_default(),
));
}
self.pools[idx]
.get_disks_by_key(&object)
.decommission_tiered_object(bucket, &object, fi, opts)
.await
}
#[instrument(level = "debug", skip(self))]
pub(super) async fn handle_get_object_reader(
&self,
@@ -179,6 +236,30 @@ impl ECStore {
let mut gopts = opts.clone();
gopts.no_lock = true;
if opts.data_movement {
let existing_pool_idx = self
.get_pool_info_existing_with_opts(bucket, object, &gopts)
.await
.map(|(pinfo, _)| pinfo.index);
let target_pool_idx =
match select_data_movement_target_pool(existing_pool_idx, opts.src_pool_idx, opts.delete_marker)? {
Some(pool_idx) => pool_idx,
None => self.get_pool_idx_no_lock(bucket, object, 0).await?,
};
if opts.src_pool_idx == target_pool_idx {
return Err(StorageError::DataMovementOverwriteErr(
bucket.to_owned(),
object.to_owned(),
opts.version_id.unwrap_or_default(),
));
}
let mut obj = self.pools[target_pool_idx].delete_object(bucket, object, opts).await?;
obj.name = decode_dir_object(obj.name.as_str());
return Ok(obj);
}
// Determine which pool contains it
let (mut pinfo, errs) = self
.get_pool_info_existing_with_opts(bucket, object, &gopts)
@@ -204,12 +285,6 @@ impl ECStore {
));
}
if opts.data_movement {
let mut obj = self.pools[pinfo.index].delete_object(bucket, object, opts).await?;
obj.name = decode_dir_object(obj.name.as_str());
return Ok(obj);
}
if !errs.is_empty() && !opts.versioned && !opts.version_suspended {
return self.delete_object_from_all_pools(bucket, object, &opts, errs).await;
}
@@ -565,3 +640,27 @@ impl ECStore {
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn delete_marker_data_movement_falls_back_when_only_source_pool_has_object() {
let target = select_data_movement_target_pool(Ok(1), 1, true).unwrap();
assert_eq!(target, None);
}
#[test]
fn delete_marker_data_movement_falls_back_when_version_does_not_exist_yet() {
let err = StorageError::ObjectNotFound("bucket".to_string(), "object".to_string());
let target = select_data_movement_target_pool(Err(err), 1, true).unwrap();
assert_eq!(target, None);
}
#[test]
fn non_delete_marker_data_movement_keeps_existing_pool() {
let target = select_data_movement_target_pool(Ok(0), 1, false).unwrap();
assert_eq!(target, Some(0));
}
}
+3 -3
View File
@@ -27,8 +27,8 @@ use bytes::Bytes;
use http::{HeaderMap, HeaderValue};
use rustfs_common::heal_channel::HealOpts;
use rustfs_filemeta::{
FileInfo, MetaCacheEntriesSorted, ObjectPartInfo, REPLICATION_RESET, REPLICATION_STATUS, ReplicateDecision, ReplicationState,
ReplicationStatusType, RestoreStatusOps as _, VersionPurgeStatusType, parse_restore_obj_status, replication_statuses_map,
FileInfo, MetaCacheEntriesSorted, ObjectPartInfo, ReplicateDecision, ReplicationState, ReplicationStatusType,
RestoreStatusOps as _, VersionPurgeStatusType, parse_restore_obj_status, replication_statuses_map,
version_purge_statuses_map,
};
use rustfs_lock::NamespaceLockWrapper;
@@ -36,7 +36,7 @@ use rustfs_madmin::heal_commands::HealResultItem;
use rustfs_rio::Checksum;
use rustfs_rio::{DecompressReader, HashReader, LimitReader, WarpReader};
use rustfs_utils::CompressionAlgorithm;
use rustfs_utils::http::headers::{AMZ_OBJECT_TAGGING, RESERVED_METADATA_PREFIX_LOWER};
use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
use rustfs_utils::http::{AMZ_BUCKET_REPLICATION_STATUS, AMZ_RESTORE, AMZ_STORAGE_CLASS};
use rustfs_utils::path::decode_dir_object;
use serde::{Deserialize, Serialize};
+14 -25
View File
@@ -118,11 +118,8 @@ impl ObjectOptions {
}
pub fn put_replication_state(&self) -> ReplicationState {
let rs = match self
.user_defined
.get(format!("{RESERVED_METADATA_PREFIX_LOWER}{REPLICATION_STATUS}").as_str())
{
Some(v) => v.to_string(),
let rs = match rustfs_utils::http::get_str(&self.user_defined, rustfs_utils::http::SUFFIX_REPLICATION_STATUS) {
Some(v) => v,
None => return ReplicationState::default(),
};
@@ -341,15 +338,11 @@ impl Clone for ObjectInfo {
impl ObjectInfo {
pub fn is_compressed(&self) -> bool {
self.user_defined
.contains_key(&format!("{RESERVED_METADATA_PREFIX_LOWER}compression"))
rustfs_utils::http::contains_key_str(&self.user_defined, rustfs_utils::http::SUFFIX_COMPRESSION)
}
pub fn is_compressed_ok(&self) -> Result<(CompressionAlgorithm, bool)> {
let scheme = self
.user_defined
.get(&format!("{RESERVED_METADATA_PREFIX_LOWER}compression"))
.cloned();
let scheme = rustfs_utils::http::get_str(&self.user_defined, rustfs_utils::http::SUFFIX_COMPRESSION);
if let Some(scheme) = scheme {
let algorithm = CompressionAlgorithm::from_str(&scheme)?;
@@ -369,7 +362,7 @@ impl ObjectInfo {
}
if self.is_compressed() {
if let Some(size_str) = self.user_defined.get(&format!("{RESERVED_METADATA_PREFIX_LOWER}actual-size"))
if let Some(size_str) = rustfs_utils::http::get_str(&self.user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE)
&& !size_str.is_empty()
{
// Todo: deal with error
@@ -745,15 +738,11 @@ impl ObjectInfo {
.user_defined
.iter()
.filter_map(|(k, v)| {
if k.starts_with(&format!("{RESERVED_METADATA_PREFIX_LOWER}{REPLICATION_RESET}")) {
Some((
k.trim_start_matches(&format!("{RESERVED_METADATA_PREFIX_LOWER}{REPLICATION_RESET}-"))
.to_string(),
v.clone(),
))
} else {
None
}
rustfs_utils::http::internal_key_strip_suffix_prefix(
k,
rustfs_utils::http::SUFFIX_REPLICATION_RESET_ARN_PREFIX,
)
.map(|arn| (arn, v.clone()))
})
.collect(),
..Default::default()
@@ -1035,8 +1024,8 @@ mod tests {
fn get_actual_size_uses_compressed_metadata_size() {
let user_defined = {
let mut map = HashMap::new();
map.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}compression"), "zstd".to_string());
map.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}actual-size"), "42".to_string());
rustfs_utils::http::insert_str(&mut map, rustfs_utils::http::SUFFIX_COMPRESSION, "zstd".to_string());
rustfs_utils::http::insert_str(&mut map, rustfs_utils::http::SUFFIX_ACTUAL_SIZE, "42".to_string());
map
};
@@ -1072,7 +1061,7 @@ mod tests {
fn get_actual_size_uses_compressed_parts_actual_size_when_metadata_missing() {
let user_defined = {
let mut map = HashMap::new();
map.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}compression"), "zstd".to_string());
rustfs_utils::http::insert_str(&mut map, rustfs_utils::http::SUFFIX_COMPRESSION, "zstd".to_string());
map
};
@@ -1100,7 +1089,7 @@ mod tests {
fn get_actual_size_returns_error_when_compressed_parts_missing_and_size_mismatch() {
let user_defined = {
let mut map = HashMap::new();
map.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}compression"), "zstd".to_string());
rustfs_utils::http::insert_str(&mut map, rustfs_utils::http::SUFFIX_COMPRESSION, "zstd".to_string());
map
};
+60 -4
View File
@@ -18,7 +18,7 @@ use crate::disk::{self, DiskAPI};
use crate::error::{Error, Result};
use crate::{
disk::{
DiskInfoOptions, DiskOption, DiskStore, FORMAT_CONFIG_FILE, RUSTFS_META_BUCKET,
DiskInfoOptions, DiskOption, DiskStore, FORMAT_CONFIG_FILE, MIGRATING_META_BUCKET, RUSTFS_META_BUCKET,
error::DiskError,
format::{FormatErasureVersion, FormatMetaVersion, FormatV3},
new_disk,
@@ -71,11 +71,13 @@ pub async fn connect_load_init_formats(
check_format_erasure_values(&formats, set_drive_count)?;
if first_disk && should_init_erasure_disks(&errs) {
// UnformattedDisk, not format file create
// UnformattedDisk, try migrate from MinIO format first, else create new format
info!("first_disk && should_init_erasure_disks");
// new format and save
if let Ok(fm) = try_migrate_format(disks, set_count, set_drive_count).await {
info!("Migrated format from MinIO config");
return Ok(fm);
}
let fm = init_format_erasure(disks, set_count, set_drive_count, deployment_id).await?;
return Ok(fm);
}
@@ -149,6 +151,60 @@ async fn init_format_erasure(
get_format_erasure_in_quorum(&fms)
}
/// Tries to migrate format
/// Returns Ok(FormatV3) if migration succeeds, Err otherwise.
async fn try_migrate_format(disks: &[Option<DiskStore>], set_count: usize, set_drive_count: usize) -> Result<FormatV3> {
for disk in disks.iter().flatten() {
let data = match disk.read_all(MIGRATING_META_BUCKET, FORMAT_CONFIG_FILE).await {
Ok(d) if !d.is_empty() => d,
_ => continue,
};
let fm = FormatV3::try_from(data.as_ref()).map_err(|e| Error::other(format!("parse MinIO format: {e}")))?;
let first_set = fm
.erasure
.sets
.first()
.ok_or_else(|| Error::other("MinIO format: erasure.sets is empty"))?;
if fm.erasure.sets.len() != set_count || first_set.len() != set_drive_count {
debug!(
"MinIO format set count mismatch: got {}x{}, expected {}x{}",
fm.erasure.sets.len(),
first_set.len(),
set_count,
set_drive_count
);
continue;
}
if fm.erasure.version != FormatErasureVersion::V3 {
debug!("MinIO format erasure version not V3: {:?}", fm.erasure.version);
continue;
}
let mut fms = vec![None; disks.len()];
for (idx, disk_opt) in disks.iter().enumerate() {
if disk_opt.is_none() {
continue;
}
let set_idx = idx / set_drive_count;
let disk_idx = idx % set_drive_count;
if set_idx >= fm.erasure.sets.len() || disk_idx >= fm.erasure.sets[set_idx].len() {
continue;
}
let mut newfm = fm.clone();
newfm.erasure.this = fm.erasure.sets[set_idx][disk_idx];
fms[idx] = Some(newfm);
}
save_format_file_all(disks, &fms).await?;
return get_format_erasure_in_quorum(&fms);
}
Err(Error::other("no MinIO format to migrate"))
}
pub fn get_format_erasure_in_quorum(formats: &[Option<FormatV3>]) -> Result<FormatV3> {
let mut countmap = HashMap::new();
+1 -1
View File
@@ -492,7 +492,7 @@ impl ECStore {
}
pub async fn list_path(self: Arc<Self>, o: &ListPathOptions) -> Result<MetaCacheEntriesSortedResult> {
// warn!("list_path opt {:?}", &o);
// tracing::warn!("list_path opt {:?}", &o);
check_list_objs_args(&o.bucket, &o.prefix, &o.marker)?;
// if opts.prefix.ends_with(SLASH_SEPARATOR) {
+4 -2
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use crate::config::storageclass::STANDARD;
use crate::disk::RUSTFS_META_BUCKET;
use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
use regex::Regex;
use rustfs_utils::http::headers::{AMZ_OBJECT_TAGGING, AMZ_STORAGE_CLASS};
use std::collections::HashMap;
@@ -45,7 +45,7 @@ pub fn clean_metadata_keys(metadata: &mut HashMap<String, String>, key_names: &[
// Check whether the bucket is the metadata bucket
fn is_meta_bucket(bucket_name: &str) -> bool {
bucket_name == RUSTFS_META_BUCKET
bucket_name == RUSTFS_META_BUCKET || bucket_name == MIGRATING_META_BUCKET
}
// Check whether the bucket is reserved
@@ -164,6 +164,8 @@ mod tests {
fn test_meta_bucket_is_invalid() {
assert!(is_reserved_or_invalid_bucket(RUSTFS_META_BUCKET, false));
assert!(is_reserved_or_invalid_bucket(RUSTFS_META_BUCKET, true));
assert!(is_reserved_or_invalid_bucket(MIGRATING_META_BUCKET, false));
assert!(is_reserved_or_invalid_bucket(MIGRATING_META_BUCKET, true));
}
#[test]
+839 -37
View File
@@ -18,14 +18,16 @@
#![allow(unused_must_use)]
#![allow(clippy::all)]
use byteorder::{ByteOrder, LittleEndian};
use bytes::Bytes;
use http::HeaderMap;
use http::status::StatusCode;
use lazy_static::lazy_static;
use rand::{Rng, RngExt};
use serde::{Deserialize, Serialize};
use std::{
collections::{HashMap, hash_map::Entry},
io::Cursor,
io::{self, Cursor},
sync::Arc,
time::Duration,
};
@@ -46,9 +48,9 @@ use crate::tier::{
use crate::{
StorageAPI,
config::com::{CONFIG_PREFIX, read_config},
disk::RUSTFS_META_BUCKET,
disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET},
store::ECStore,
store_api::{ObjectOptions, PutObjReader},
store_api::{ObjectIO as _, ObjectOptions, PutObjReader},
};
use rustfs_rio::HashReader;
use rustfs_utils::path::{SLASH_SEPARATOR, path_join};
@@ -61,10 +63,17 @@ use super::{
const TIER_CFG_REFRESH: Duration = Duration::from_secs(15 * 60);
pub const TIER_CONFIG_FILE: &str = "tier-config.json";
const TIER_CONFIG_LEGACY_FILE: &str = "tier-config.json";
pub const TIER_CONFIG_FILE: &str = "tier-config.bin";
pub const TIER_CONFIG_FORMAT: u16 = 1;
pub const TIER_CONFIG_V1: u16 = 1;
pub const TIER_CONFIG_VERSION: u16 = 1;
pub const TIER_CONFIG_VERSION: u16 = 2;
const EXTERNAL_TIER_TYPE_UNSUPPORTED: i32 = 0;
const EXTERNAL_TIER_TYPE_S3: i32 = 1;
const EXTERNAL_TIER_TYPE_AZURE: i32 = 2;
const EXTERNAL_TIER_TYPE_GCS: i32 = 3;
const EXTERNAL_TIER_TYPE_MINIO: i32 = 4;
const _TIER_CFG_REFRESH_AT_HDR: &str = "X-RustFS-TierCfg-RefreshedAt";
@@ -104,6 +113,609 @@ pub struct TierConfigMgr {
pub last_refreshed_at: OffsetDateTime,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[serde(default)]
struct ExternalTierConfigMgr {
#[serde(rename = "Tiers")]
tiers: HashMap<String, ExternalTierConfig>,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[serde(default)]
struct ExternalTierConfig {
#[serde(rename = "Version")]
version: String,
#[serde(rename = "Type")]
tier_type: i32,
#[serde(rename = "Name")]
name: String,
#[serde(rename = "S3")]
s3: Option<ExternalTierS3>,
#[serde(rename = "Azure")]
azure: Option<ExternalTierAzure>,
#[serde(rename = "GCS")]
gcs: Option<ExternalTierGcs>,
#[serde(rename = "MinIO", alias = "Compatible")]
compatible_backend: Option<ExternalTierCompatible>,
#[serde(rename = "XTierType", skip_serializing_if = "Option::is_none")]
tier_type_hint: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[serde(default)]
struct ExternalTierS3 {
#[serde(rename = "Endpoint")]
endpoint: String,
#[serde(rename = "AccessKey")]
access_key: String,
#[serde(rename = "SecretKey")]
secret_key: String,
#[serde(rename = "Bucket")]
bucket: String,
#[serde(rename = "Prefix")]
prefix: String,
#[serde(rename = "Region")]
region: String,
#[serde(rename = "StorageClass")]
storage_class: String,
#[serde(rename = "AWSRole")]
aws_role: bool,
#[serde(rename = "AWSRoleWebIdentityTokenFile")]
aws_role_web_identity_token_file: String,
#[serde(rename = "AWSRoleARN")]
aws_role_arn: String,
#[serde(rename = "AWSRoleSessionName")]
aws_role_session_name: String,
#[serde(rename = "AWSRoleDurationSeconds")]
aws_role_duration_seconds: i32,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[serde(default)]
struct ExternalServicePrincipalAuth {
#[serde(rename = "TenantID")]
tenant_id: String,
#[serde(rename = "ClientID")]
client_id: String,
#[serde(rename = "ClientSecret")]
client_secret: String,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[serde(default)]
struct ExternalTierAzure {
#[serde(rename = "Endpoint")]
endpoint: String,
#[serde(rename = "AccountName")]
account_name: String,
#[serde(rename = "AccountKey")]
account_key: String,
#[serde(rename = "Bucket")]
bucket: String,
#[serde(rename = "Prefix")]
prefix: String,
#[serde(rename = "Region")]
region: String,
#[serde(rename = "StorageClass")]
storage_class: String,
#[serde(rename = "SPAuth")]
sp_auth: ExternalServicePrincipalAuth,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[serde(default)]
struct ExternalTierGcs {
#[serde(rename = "Endpoint")]
endpoint: String,
#[serde(rename = "Creds")]
creds: String,
#[serde(rename = "Bucket")]
bucket: String,
#[serde(rename = "Prefix")]
prefix: String,
#[serde(rename = "Region")]
region: String,
#[serde(rename = "StorageClass")]
storage_class: String,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[serde(default)]
struct ExternalTierCompatible {
#[serde(rename = "Endpoint")]
endpoint: String,
#[serde(rename = "AccessKey")]
access_key: String,
#[serde(rename = "SecretKey")]
secret_key: String,
#[serde(rename = "Bucket")]
bucket: String,
#[serde(rename = "Prefix")]
prefix: String,
#[serde(rename = "Region")]
region: String,
}
fn tier_config_path(file: &str) -> String {
format!("{}{}{}", CONFIG_PREFIX, SLASH_SEPARATOR, file)
}
fn tier_hint_for_type(tier_type: TierType) -> Option<&'static str> {
match tier_type {
TierType::RustFS => Some("rustfs"),
TierType::Aliyun => Some("aliyun"),
TierType::Tencent => Some("tencent"),
TierType::Huaweicloud => Some("huaweicloud"),
TierType::R2 => Some("r2"),
_ => None,
}
}
fn tier_type_from_hint(hint: Option<&str>) -> Option<TierType> {
match hint {
Some("rustfs") => Some(TierType::RustFS),
Some("aliyun") => Some(TierType::Aliyun),
Some("tencent") => Some(TierType::Tencent),
Some("huaweicloud") => Some(TierType::Huaweicloud),
Some("r2") => Some(TierType::R2),
_ => None,
}
}
fn external_tier_s3_from_internal(s3: &crate::tier::tier_config::TierS3) -> ExternalTierS3 {
ExternalTierS3 {
endpoint: s3.endpoint.clone(),
access_key: s3.access_key.clone(),
secret_key: s3.secret_key.clone(),
bucket: s3.bucket.clone(),
prefix: s3.prefix.clone(),
region: s3.region.clone(),
storage_class: s3.storage_class.clone(),
aws_role: s3.aws_role,
aws_role_web_identity_token_file: s3.aws_role_web_identity_token_file.clone(),
aws_role_arn: s3.aws_role_arn.clone(),
aws_role_session_name: s3.aws_role_session_name.clone(),
aws_role_duration_seconds: s3.aws_role_duration_seconds,
}
}
fn external_tier_s3_from_compatible_payload(
endpoint: String,
access_key: String,
secret_key: String,
bucket: String,
prefix: String,
region: String,
) -> ExternalTierS3 {
ExternalTierS3 {
endpoint,
access_key,
secret_key,
bucket,
prefix,
region,
storage_class: String::new(),
aws_role: false,
aws_role_web_identity_token_file: String::new(),
aws_role_arn: String::new(),
aws_role_session_name: String::new(),
aws_role_duration_seconds: 0,
}
}
fn external_tier_alias_from_compatible_payload(
endpoint: String,
access_key: String,
secret_key: String,
bucket: String,
prefix: String,
region: String,
) -> ExternalTierCompatible {
ExternalTierCompatible {
endpoint,
access_key,
secret_key,
bucket,
prefix,
region,
}
}
fn to_external_tier_config(name: &str, tier: &TierConfig) -> io::Result<ExternalTierConfig> {
let mut out = ExternalTierConfig {
version: if tier.version.is_empty() {
"v1".to_string()
} else {
tier.version.clone()
},
name: if tier.name.is_empty() {
name.to_string()
} else {
tier.name.clone()
},
..Default::default()
};
match tier.tier_type {
TierType::S3 => {
let s3 = tier
.s3
.as_ref()
.ok_or_else(|| io::Error::other("tier config missing s3 backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_S3;
out.s3 = Some(external_tier_s3_from_internal(s3));
}
TierType::Azure => {
let az = tier
.azure
.as_ref()
.ok_or_else(|| io::Error::other("tier config missing azure backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_AZURE;
out.azure = Some(ExternalTierAzure {
endpoint: az.endpoint.clone(),
account_name: az.access_key.clone(),
account_key: az.secret_key.clone(),
bucket: az.bucket.clone(),
prefix: az.prefix.clone(),
region: az.region.clone(),
storage_class: az.storage_class.clone(),
sp_auth: ExternalServicePrincipalAuth {
tenant_id: az.sp_auth.tenant_id.clone(),
client_id: az.sp_auth.client_id.clone(),
client_secret: az.sp_auth.client_secret.clone(),
},
});
}
TierType::GCS => {
let gcs = tier
.gcs
.as_ref()
.ok_or_else(|| io::Error::other("tier config missing gcs backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_GCS;
out.gcs = Some(ExternalTierGcs {
endpoint: gcs.endpoint.clone(),
creds: gcs.creds.clone(),
bucket: gcs.bucket.clone(),
prefix: gcs.prefix.clone(),
region: gcs.region.clone(),
storage_class: gcs.storage_class.clone(),
});
}
TierType::MinIO => {
let backend = tier
.minio
.as_ref()
.ok_or_else(|| io::Error::other("tier config missing compatible backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_MINIO;
out.compatible_backend = Some(external_tier_alias_from_compatible_payload(
backend.endpoint.clone(),
backend.access_key.clone(),
backend.secret_key.clone(),
backend.bucket.clone(),
backend.prefix.clone(),
backend.region.clone(),
));
}
TierType::RustFS => {
let backend = tier
.rustfs
.as_ref()
.ok_or_else(|| io::Error::other("tier config missing compatible backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_S3;
out.tier_type_hint = tier_hint_for_type(tier.tier_type.clone()).map(ToString::to_string);
out.s3 = Some(external_tier_s3_from_compatible_payload(
backend.endpoint.clone(),
backend.access_key.clone(),
backend.secret_key.clone(),
backend.bucket.clone(),
backend.prefix.clone(),
backend.region.clone(),
));
}
TierType::Aliyun => {
let backend = tier
.aliyun
.as_ref()
.ok_or_else(|| io::Error::other("tier config missing compatible backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_S3;
out.tier_type_hint = tier_hint_for_type(tier.tier_type.clone()).map(ToString::to_string);
out.s3 = Some(external_tier_s3_from_compatible_payload(
backend.endpoint.clone(),
backend.access_key.clone(),
backend.secret_key.clone(),
backend.bucket.clone(),
backend.prefix.clone(),
backend.region.clone(),
));
}
TierType::Tencent => {
let backend = tier
.tencent
.as_ref()
.ok_or_else(|| io::Error::other("tier config missing compatible backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_S3;
out.tier_type_hint = tier_hint_for_type(tier.tier_type.clone()).map(ToString::to_string);
out.s3 = Some(external_tier_s3_from_compatible_payload(
backend.endpoint.clone(),
backend.access_key.clone(),
backend.secret_key.clone(),
backend.bucket.clone(),
backend.prefix.clone(),
backend.region.clone(),
));
}
TierType::Huaweicloud => {
let backend = tier
.huaweicloud
.as_ref()
.ok_or_else(|| io::Error::other("tier config missing compatible backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_S3;
out.tier_type_hint = tier_hint_for_type(tier.tier_type.clone()).map(ToString::to_string);
out.s3 = Some(external_tier_s3_from_compatible_payload(
backend.endpoint.clone(),
backend.access_key.clone(),
backend.secret_key.clone(),
backend.bucket.clone(),
backend.prefix.clone(),
backend.region.clone(),
));
}
TierType::R2 => {
let backend = tier
.r2
.as_ref()
.ok_or_else(|| io::Error::other("tier config missing compatible backend payload"))?;
out.tier_type = EXTERNAL_TIER_TYPE_S3;
out.tier_type_hint = tier_hint_for_type(tier.tier_type.clone()).map(ToString::to_string);
out.s3 = Some(external_tier_s3_from_compatible_payload(
backend.endpoint.clone(),
backend.access_key.clone(),
backend.secret_key.clone(),
backend.bucket.clone(),
backend.prefix.clone(),
backend.region.clone(),
));
}
TierType::Unsupported => {
out.tier_type = EXTERNAL_TIER_TYPE_UNSUPPORTED;
}
}
Ok(out)
}
fn decode_legacy_s3_like(name: &str, ext: &ExternalTierConfig) -> io::Result<ExternalTierS3> {
if let Some(s3) = ext.s3.as_ref() {
return Ok(s3.clone());
}
if let Some(m) = ext.compatible_backend.as_ref() {
return Ok(external_tier_s3_from_compatible_payload(
m.endpoint.clone(),
m.access_key.clone(),
m.secret_key.clone(),
m.bucket.clone(),
m.prefix.clone(),
m.region.clone(),
));
}
Err(io::Error::other(format!("tier config '{name}' missing compatible backend payload")))
}
fn from_external_tier_config(name: String, ext: ExternalTierConfig) -> io::Result<TierConfig> {
let mut cfg = TierConfig {
version: if ext.version.is_empty() {
"v1".to_string()
} else {
ext.version.clone()
},
name: if ext.name.is_empty() { name.clone() } else { ext.name.clone() },
..Default::default()
};
let hinted = tier_type_from_hint(ext.tier_type_hint.as_deref());
let tier_type = if let Some(h) = hinted {
h
} else {
match ext.tier_type {
EXTERNAL_TIER_TYPE_S3 => TierType::S3,
EXTERNAL_TIER_TYPE_AZURE => TierType::Azure,
EXTERNAL_TIER_TYPE_GCS => TierType::GCS,
EXTERNAL_TIER_TYPE_MINIO => TierType::MinIO,
_ => TierType::Unsupported,
}
};
cfg.tier_type = tier_type.clone();
match tier_type {
TierType::S3 => {
let s3 = ext
.s3
.as_ref()
.ok_or_else(|| io::Error::other(format!("tier config '{}' missing s3 backend payload", cfg.name)))?;
cfg.s3 = Some(crate::tier::tier_config::TierS3 {
name: cfg.name.clone(),
endpoint: s3.endpoint.clone(),
access_key: s3.access_key.clone(),
secret_key: s3.secret_key.clone(),
bucket: s3.bucket.clone(),
prefix: s3.prefix.clone(),
region: s3.region.clone(),
storage_class: s3.storage_class.clone(),
aws_role: s3.aws_role,
aws_role_web_identity_token_file: s3.aws_role_web_identity_token_file.clone(),
aws_role_arn: s3.aws_role_arn.clone(),
aws_role_session_name: s3.aws_role_session_name.clone(),
aws_role_duration_seconds: s3.aws_role_duration_seconds,
});
}
TierType::Azure => {
let az = ext
.azure
.as_ref()
.ok_or_else(|| io::Error::other(format!("tier config '{}' missing azure backend payload", cfg.name)))?;
cfg.azure = Some(crate::tier::tier_config::TierAzure {
name: cfg.name.clone(),
endpoint: az.endpoint.clone(),
access_key: az.account_name.clone(),
secret_key: az.account_key.clone(),
bucket: az.bucket.clone(),
prefix: az.prefix.clone(),
region: az.region.clone(),
storage_class: az.storage_class.clone(),
sp_auth: crate::tier::tier_config::ServicePrincipalAuth {
tenant_id: az.sp_auth.tenant_id.clone(),
client_id: az.sp_auth.client_id.clone(),
client_secret: az.sp_auth.client_secret.clone(),
},
});
}
TierType::GCS => {
let gcs = ext
.gcs
.as_ref()
.ok_or_else(|| io::Error::other(format!("tier config '{}' missing gcs backend payload", cfg.name)))?;
cfg.gcs = Some(crate::tier::tier_config::TierGCS {
name: cfg.name.clone(),
endpoint: gcs.endpoint.clone(),
creds: gcs.creds.clone(),
bucket: gcs.bucket.clone(),
prefix: gcs.prefix.clone(),
region: gcs.region.clone(),
storage_class: gcs.storage_class.clone(),
});
}
TierType::MinIO => {
let m = ext
.compatible_backend
.as_ref()
.ok_or_else(|| io::Error::other(format!("tier config '{}' missing compatible backend payload", cfg.name)))?;
cfg.minio = Some(crate::tier::tier_config::TierMinIO {
name: cfg.name.clone(),
endpoint: m.endpoint.clone(),
access_key: m.access_key.clone(),
secret_key: m.secret_key.clone(),
bucket: m.bucket.clone(),
prefix: m.prefix.clone(),
region: m.region.clone(),
});
}
TierType::RustFS => {
let m = decode_legacy_s3_like(&cfg.name, &ext)?;
cfg.rustfs = Some(crate::tier::tier_config::TierRustFS {
name: cfg.name.clone(),
endpoint: m.endpoint,
access_key: m.access_key,
secret_key: m.secret_key,
bucket: m.bucket,
prefix: m.prefix,
region: m.region,
storage_class: m.storage_class,
});
}
TierType::Aliyun => {
let m = decode_legacy_s3_like(&cfg.name, &ext)?;
cfg.aliyun = Some(crate::tier::tier_config::TierAliyun {
name: cfg.name.clone(),
endpoint: m.endpoint,
access_key: m.access_key,
secret_key: m.secret_key,
bucket: m.bucket,
prefix: m.prefix,
region: m.region,
});
}
TierType::Tencent => {
let m = decode_legacy_s3_like(&cfg.name, &ext)?;
cfg.tencent = Some(crate::tier::tier_config::TierTencent {
name: cfg.name.clone(),
endpoint: m.endpoint,
access_key: m.access_key,
secret_key: m.secret_key,
bucket: m.bucket,
prefix: m.prefix,
region: m.region,
});
}
TierType::Huaweicloud => {
let m = decode_legacy_s3_like(&cfg.name, &ext)?;
cfg.huaweicloud = Some(crate::tier::tier_config::TierHuaweicloud {
name: cfg.name.clone(),
endpoint: m.endpoint,
access_key: m.access_key,
secret_key: m.secret_key,
bucket: m.bucket,
prefix: m.prefix,
region: m.region,
});
}
TierType::R2 => {
let m = decode_legacy_s3_like(&cfg.name, &ext)?;
cfg.r2 = Some(crate::tier::tier_config::TierR2 {
name: cfg.name.clone(),
endpoint: m.endpoint,
access_key: m.access_key,
secret_key: m.secret_key,
bucket: m.bucket,
prefix: m.prefix,
region: m.region,
});
}
TierType::Unsupported => {}
}
Ok(cfg)
}
fn encode_external_tiering_config_blob(cfg: &TierConfigMgr) -> io::Result<Bytes> {
let mut tiers = HashMap::with_capacity(cfg.tiers.len());
for (name, tier_cfg) in &cfg.tiers {
tiers.insert(name.clone(), to_external_tier_config(name, tier_cfg)?);
}
let payload = rmp_serde::to_vec(&ExternalTierConfigMgr { tiers })
.map_err(|err| io::Error::other(format!("serialize tier config payload failed: {err}")))?;
let mut data = Vec::with_capacity(4 + payload.len());
let mut format = [0u8; 2];
LittleEndian::write_u16(&mut format, TIER_CONFIG_FORMAT);
data.extend_from_slice(&format);
let mut version = [0u8; 2];
LittleEndian::write_u16(&mut version, TIER_CONFIG_VERSION);
data.extend_from_slice(&version);
data.extend_from_slice(&payload);
Ok(Bytes::from(data))
}
fn decode_external_tiering_config_blob(data: &[u8]) -> io::Result<TierConfigMgr> {
if data.len() <= 4 {
return Err(io::Error::other("tierConfigInit: no data"));
}
let format = LittleEndian::read_u16(&data[0..2]);
if format != TIER_CONFIG_FORMAT {
return Err(io::Error::other(format!("tierConfigInit: unknown format: {format}")));
}
let version = LittleEndian::read_u16(&data[2..4]);
if version != TIER_CONFIG_V1 && version != TIER_CONFIG_VERSION {
return Err(io::Error::other(format!("tierConfigInit: unknown version: {version}")));
}
let external: ExternalTierConfigMgr =
rmp_serde::from_slice(&data[4..]).map_err(|err| io::Error::other(format!("decode tier config payload failed: {err}")))?;
let mut tiers = HashMap::with_capacity(external.tiers.len());
for (name, ext_cfg) in external.tiers {
tiers.insert(name.clone(), from_external_tier_config(name, ext_cfg)?);
}
Ok(TierConfigMgr {
driver_cache: HashMap::new(),
tiers,
last_refreshed_at: OffsetDateTime::now_utc(),
})
}
fn decode_tiering_config_blob(data: &[u8]) -> io::Result<TierConfigMgr> {
if let Ok(cfg) = TierConfigMgr::unmarshal(data) {
return Ok(cfg);
}
decode_external_tiering_config_blob(data)
}
impl TierConfigMgr {
pub fn new() -> Arc<RwLock<Self>> {
Arc::new(RwLock::new(Self {
@@ -285,12 +897,12 @@ impl TierConfigMgr {
rustfs.secret_key = creds.secret_key;
}
TierType::MinIO => {
let mut minio = tier_config.minio.as_mut().expect("err");
let compatible_backend = tier_config.minio.as_mut().expect("err");
if creds.access_key == "" || creds.secret_key == "" {
return Err(ERR_TIER_MISSING_CREDENTIALS.clone());
}
minio.access_key = creds.access_key;
minio.secret_key = creds.secret_key;
compatible_backend.access_key = creds.access_key;
compatible_backend.secret_key = creds.secret_key;
}
TierType::Aliyun => {
let mut aliyun = tier_config.aliyun.as_mut().expect("err");
@@ -401,9 +1013,8 @@ impl TierConfigMgr {
}
pub async fn save_tiering_config<S: StorageAPI>(&self, api: Arc<S>) -> std::result::Result<(), std::io::Error> {
let data = self.marshal()?;
let config_file = format!("{}{}{}", CONFIG_PREFIX, SLASH_SEPARATOR, TIER_CONFIG_FILE);
let data = encode_external_tiering_config_blob(self)?;
let config_file = tier_config_path(TIER_CONFIG_FILE);
self.save_config(api, &config_file, data).await
}
@@ -483,38 +1094,229 @@ async fn new_and_save_tiering_config<S: StorageAPI>(api: Arc<S>) -> Result<TierC
#[tracing::instrument(level = "debug", name = "load_tier_config", skip(api))]
async fn load_tier_config(api: Arc<ECStore>) -> std::result::Result<TierConfigMgr, std::io::Error> {
let config_file = format!("{}{}{}", CONFIG_PREFIX, SLASH_SEPARATOR, TIER_CONFIG_FILE);
let data = read_config(api.clone(), config_file.as_str()).await;
if let Err(err) = data {
if is_err_config_not_found(&err) {
warn!("config not found, start to init");
let cfg = new_and_save_tiering_config(api).await?;
return Ok(cfg);
} else {
error!("read config err {:?}", &err);
return Err(std::io::Error::other(err));
}
}
let cfg;
let version = 1; //LittleEndian::read_u16(&data[2..4]);
match version {
TIER_CONFIG_V1/* | TIER_CONFIG_VERSION */ => {
cfg = match TierConfigMgr::unmarshal(&data.unwrap()) {
Ok(cfg) => cfg,
Err(err) => {
return Err(std::io::Error::other(err.to_string()));
let config_file = tier_config_path(TIER_CONFIG_FILE);
match read_config(api.clone(), config_file.as_str()).await {
Ok(data) => decode_tiering_config_blob(&data),
Err(err) if is_err_config_not_found(&err) => {
let legacy_file = tier_config_path(TIER_CONFIG_LEGACY_FILE);
match read_config(api.clone(), legacy_file.as_str()).await {
Ok(data) => {
let cfg = TierConfigMgr::unmarshal(&data)?;
let normalized = encode_external_tiering_config_blob(&cfg)?;
let _ = api
.put_object(
RUSTFS_META_BUCKET,
&config_file,
&mut PutObjReader::from_vec(normalized.to_vec()),
&ObjectOptions {
max_parity: true,
..Default::default()
},
)
.await;
Ok(cfg)
}
};
Err(legacy_err) if is_err_config_not_found(&legacy_err) => {
warn!("config not found, start to init");
new_and_save_tiering_config(api).await.map_err(io::Error::other)
}
Err(legacy_err) => Err(io::Error::other(legacy_err)),
}
}
_ => {
return Err(std::io::Error::other(format!("tierConfigInit: unknown version: {}", version)));
Err(err) => {
error!("read config err {:?}", &err);
Err(io::Error::other(err))
}
}
}
Ok(cfg)
async fn read_tier_config_from_bucket<S: StorageAPI>(
api: Arc<S>,
bucket: &str,
path: &str,
opts: &ObjectOptions,
) -> io::Result<Option<Vec<u8>>> {
let mut rd = match api.get_object_reader(bucket, path, None, HeaderMap::new(), opts).await {
Ok(v) => v,
Err(err) if is_err_config_not_found(&err) => return Ok(None),
Err(err) => return Err(io::Error::other(err)),
};
let data = rd.read_all().await.map_err(io::Error::other)?;
if data.is_empty() {
return Ok(None);
}
Ok(Some(data))
}
async fn write_tier_config_to_rustfs<S: StorageAPI>(api: Arc<S>, path: &str, data: Bytes) -> io::Result<()> {
api.put_object(
RUSTFS_META_BUCKET,
path,
&mut PutObjReader::from_vec(data.to_vec()),
&ObjectOptions {
max_parity: true,
..Default::default()
},
)
.await
.map_err(io::Error::other)?;
Ok(())
}
pub async fn try_migrate_tiering_config<S: StorageAPI>(api: Arc<S>) {
let target_path = tier_config_path(TIER_CONFIG_FILE);
if api
.get_object_info(RUSTFS_META_BUCKET, &target_path, &ObjectOptions::default())
.await
.is_ok()
{
debug!("tier config already exists in RustFS metadata bucket, skip migration");
return;
}
let opts = ObjectOptions {
max_parity: true,
no_lock: true,
..Default::default()
};
let legacy_path = tier_config_path(TIER_CONFIG_LEGACY_FILE);
match read_tier_config_from_bucket(api.clone(), RUSTFS_META_BUCKET, &legacy_path, &opts).await {
Ok(Some(data)) => match TierConfigMgr::unmarshal(&data)
.and_then(|cfg| encode_external_tiering_config_blob(&cfg).map_err(io::Error::other))
{
Ok(out) => {
if write_tier_config_to_rustfs(api.clone(), &target_path, out).await.is_ok() {
info!("Migrated tier config from legacy RustFS metadata format");
return;
}
}
Err(err) => warn!("legacy tier config is incompatible, skip local migration: {}", err),
},
Ok(None) => {}
Err(err) => warn!("read legacy local tier config failed: {}", err),
}
match read_tier_config_from_bucket(api.clone(), MIGRATING_META_BUCKET, &target_path, &opts).await {
Ok(Some(data)) => match decode_tiering_config_blob(&data).and_then(|cfg| encode_external_tiering_config_blob(&cfg)) {
Ok(out) => {
if write_tier_config_to_rustfs(api.clone(), &target_path, out).await.is_ok() {
info!("Migrated compatible tier config from migrating metadata bucket");
}
}
Err(err) => warn!("migrating tier config is incompatible, skip migration: {}", err),
},
Ok(None) => {}
Err(err) => warn!("read migrating tier config failed: {}", err),
}
}
pub fn is_err_config_not_found(err: &StorageError) -> bool {
matches!(err, StorageError::ObjectNotFound(_, _)) || err == &StorageError::ConfigNotFound
matches!(err, StorageError::ObjectNotFound(_, _) | StorageError::BucketNotFound(_)) || err == &StorageError::ConfigNotFound
}
#[cfg(test)]
mod tests {
use super::*;
fn build_s3_tier(name: &str) -> TierConfig {
TierConfig {
version: "v1".to_string(),
tier_type: TierType::S3,
name: name.to_string(),
s3: Some(crate::tier::tier_config::TierS3 {
name: name.to_string(),
endpoint: "https://example-s3.invalid".to_string(),
access_key: "ak".to_string(),
secret_key: "sk".to_string(),
bucket: "bucket-a".to_string(),
prefix: "prefix-a".to_string(),
region: "us-east-1".to_string(),
storage_class: "STANDARD".to_string(),
aws_role: false,
aws_role_web_identity_token_file: String::new(),
aws_role_arn: String::new(),
aws_role_session_name: String::new(),
aws_role_duration_seconds: 0,
}),
..Default::default()
}
}
#[test]
fn test_tiering_external_blob_roundtrip_for_standard_type() {
let mut cfg = TierConfigMgr {
driver_cache: HashMap::new(),
tiers: HashMap::new(),
last_refreshed_at: OffsetDateTime::now_utc(),
};
cfg.tiers.insert("COLD-A".to_string(), build_s3_tier("COLD-A"));
let bytes = encode_external_tiering_config_blob(&cfg).expect("encode should succeed");
assert_eq!(&bytes[0..2], &TIER_CONFIG_FORMAT.to_le_bytes());
assert_eq!(&bytes[2..4], &TIER_CONFIG_VERSION.to_le_bytes());
let decoded = decode_external_tiering_config_blob(&bytes).expect("decode should succeed");
let tier = decoded.tiers.get("COLD-A").expect("tier should exist");
assert_eq!(tier.tier_type.as_lowercase(), "s3");
assert_eq!(tier.s3.as_ref().expect("s3 should exist").endpoint, "https://example-s3.invalid");
}
#[test]
fn test_tiering_external_blob_roundtrip_for_extended_type_hint() {
let mut cfg = TierConfigMgr {
driver_cache: HashMap::new(),
tiers: HashMap::new(),
last_refreshed_at: OffsetDateTime::now_utc(),
};
cfg.tiers.insert(
"COLD-B".to_string(),
TierConfig {
version: "v1".to_string(),
tier_type: TierType::RustFS,
name: "COLD-B".to_string(),
rustfs: Some(crate::tier::tier_config::TierRustFS {
name: "COLD-B".to_string(),
endpoint: "https://example-compat.invalid".to_string(),
access_key: "ak".to_string(),
secret_key: "sk".to_string(),
bucket: "bucket-b".to_string(),
prefix: "prefix-b".to_string(),
region: "us-east-1".to_string(),
storage_class: "STANDARD".to_string(),
}),
..Default::default()
},
);
let bytes = encode_external_tiering_config_blob(&cfg).expect("encode should succeed");
let decoded = decode_external_tiering_config_blob(&bytes).expect("decode should succeed");
let tier = decoded.tiers.get("COLD-B").expect("tier should exist");
assert_eq!(tier.tier_type.as_lowercase(), "rustfs");
assert_eq!(
tier.rustfs.as_ref().expect("backend should exist").endpoint,
"https://example-compat.invalid"
);
}
#[test]
fn test_decode_tiering_config_blob_accepts_legacy_json() {
let mut cfg = TierConfigMgr {
driver_cache: HashMap::new(),
tiers: HashMap::new(),
last_refreshed_at: OffsetDateTime::now_utc(),
};
cfg.tiers.insert("COLD-A".to_string(), build_s3_tier("COLD-A"));
let data = serde_json::to_vec(&cfg).expect("legacy json should encode");
let decoded = decode_tiering_config_blob(&data).expect("legacy json should decode");
assert_eq!(
decoded
.tiers
.get("COLD-A")
.and_then(|tier| tier.s3.as_ref())
.map(|s3| s3.bucket.as_str()),
Some("bucket-a")
);
}
}
+8 -8
View File
@@ -146,7 +146,7 @@ impl Clone for TierConfig {
fn clone(&self) -> TierConfig {
let mut s3 = None;
let mut r = None;
let mut m = None;
let mut compatible_backend = None;
let mut aliyun = None;
let mut tencent = None;
let mut huaweicloud = None;
@@ -165,9 +165,9 @@ impl Clone for TierConfig {
r = Some(r_);
}
TierType::MinIO => {
let mut m_ = self.minio.as_ref().expect("err").clone();
m_.secret_key = "REDACTED".to_string();
m = Some(m_);
let mut compatible_backend_ = self.minio.as_ref().expect("err").clone();
compatible_backend_.secret_key = "REDACTED".to_string();
compatible_backend = Some(compatible_backend_);
}
TierType::Aliyun => {
let mut aliyun_ = self.aliyun.as_ref().expect("err").clone();
@@ -207,7 +207,7 @@ impl Clone for TierConfig {
name: self.name.clone(),
s3,
rustfs: r,
minio: m,
minio: compatible_backend,
aliyun,
tencent,
huaweicloud,
@@ -408,7 +408,7 @@ impl TierMinIO {
if name.is_empty() {
return Err(std::io::Error::other(ERR_TIER_NAME_EMPTY));
}
let m = TierMinIO {
let backend = TierMinIO {
access_key: access_key.to_string(),
secret_key: secret_key.to_string(),
bucket: bucket.to_string(),
@@ -417,7 +417,7 @@ impl TierMinIO {
};
for option in options {
let option = option(m.clone());
let option = option(backend.clone());
let option = *option;
option?;
}
@@ -426,7 +426,7 @@ impl TierMinIO {
version: C_TIER_CONFIG_VER.to_string(),
tier_type: TierType::MinIO,
name: name.to_string(),
minio: Some(m),
minio: Some(backend),
..Default::default()
})
}
@@ -0,0 +1,238 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Integration test: verify create_bitrot_reader with HighwayHash256SLegacy reads main-version objects.
//!
//! Supports both EC (part files) and inline objects. Reads xl.meta to compute params.
//! Uses create_bitrot_reader only (no ECStore/get_object_reader) to isolate bitrot compatibility.
//!
//! Run from workspace root:
//! cargo test -p rustfs-ecstore test_legacy_bitrot_read -- --nocapture
//!
//! Environment:
//! RUSTFS_LEGACY_TEST_ROOT - Test data root (default: workspace root)
//! RUSTFS_LEGACY_TEST_DISK - Disk name under root (default: "test1" for rustfs, "test" for minio)
//! RUSTFS_SKIP_LEGACY_TEST - Set to 1 to skip
//!
//! For MinIO data: RUSTFS_LEGACY_TEST_ROOT=/path/to/minio (disk "test" and .minio.sys auto-detected).
use rustfs_ecstore::bitrot::create_bitrot_reader;
use rustfs_ecstore::disk::endpoint::Endpoint;
use rustfs_ecstore::disk::{DiskOption, STORAGE_FORMAT_FILE, new_disk};
use rustfs_filemeta::{FileInfoOpts, get_file_info};
use rustfs_utils::HashAlgorithm;
use std::path::PathBuf;
use tokio::fs;
fn workspace_root() -> PathBuf {
let manifest = std::env::var("CARGO_MANIFEST_DIR").unwrap_or_else(|_| ".".into());
PathBuf::from(&manifest)
.ancestors()
.nth(2)
.unwrap_or(std::path::Path::new("."))
.to_path_buf()
}
fn legacy_test_data_exists() -> bool {
if std::env::var("RUSTFS_SKIP_LEGACY_TEST").unwrap_or_default() == "1" {
return false;
}
let root = workspace_root();
let format_path = root.join("test1/.rustfs.sys/format.json");
let ktvzip_meta = root.join("test1/vvvv/ktvzip.tar.gz").join(STORAGE_FORMAT_FILE);
let path_traversal_meta = root.join("test1/vvvv/path_traversal.md").join(STORAGE_FORMAT_FILE);
format_path.exists() && (ktvzip_meta.exists() || path_traversal_meta.exists())
}
async fn run_legacy_bitrot_test_for_object(root: &std::path::Path, disk_name: &str, bucket: &str, object: &str) -> bool {
let xl_meta_path = root.join(disk_name).join(bucket).join(object).join(STORAGE_FORMAT_FILE);
if !xl_meta_path.exists() {
eprintln!("xl_meta_path not found: {:?}", xl_meta_path);
return false;
}
let buf = match fs::read(&xl_meta_path).await {
Ok(b) => b,
Err(_) => {
eprintln!("Failed to read xl_meta_path: {:?}", xl_meta_path);
return false;
}
};
let fi = match get_file_info(
&buf,
bucket,
object,
"",
FileInfoOpts {
data: true, // need inline data for inline objects
include_free_versions: false,
},
) {
Ok(f) => f,
Err(_) => {
eprintln!("Failed to get file info: {:?}", xl_meta_path);
return false;
}
};
if fi.deleted || fi.parts.is_empty() {
eprintln!("File is deleted or has no parts: {:?}", xl_meta_path);
return false;
}
let shard_size = fi.erasure.shard_size();
let part_number = 1;
let checksum_info = fi.erasure.get_checksum_info(part_number);
let checksum_algo = if fi.uses_legacy_checksum && checksum_info.algorithm == HashAlgorithm::HighwayHash256S {
HashAlgorithm::HighwayHash256SLegacy
} else {
checksum_info.algorithm
};
eprintln!("fi.inline_data(): {:?}", fi.inline_data());
eprintln!("fi.is_remote(): {:?}", fi.metadata);
// Inline path: use fi.data
if fi.inline_data() {
let inline_bytes = match &fi.data {
Some(b) if !b.is_empty() => b.as_ref(),
_ => {
eprintln!("Inline data is empty: {:?}", xl_meta_path);
return false;
}
};
let read_length = fi.size as usize;
if read_length == 0 {
eprintln!("Read length is 0: {:?}", xl_meta_path);
return false;
}
let mut reader = match create_bitrot_reader(
Some(inline_bytes),
None,
bucket,
"",
0,
read_length,
shard_size,
checksum_algo.clone(),
false,
)
.await
{
Ok(Some(r)) => r,
_ => {
eprintln!("Failed to create bitrot reader for inline data: {:?}", xl_meta_path);
return false;
}
};
let mut buf = vec![0u8; shard_size];
match reader.read(&mut buf).await {
Ok(n) if n > 0 => {
eprintln!("Successfully read {} bytes (inline) via create_bitrot_reader with {:?}", n, checksum_algo);
return true;
}
_ => {
eprintln!("Failed to read inline data: {:?}", xl_meta_path);
return false;
}
}
}
// EC path: use disk + part file
let data_dir = match fi.data_dir {
Some(d) => d,
None => {
eprintln!("Data dir is empty: {:?}", xl_meta_path);
return false;
}
};
let path = format!("{object}/{data_dir}/part.{}", part_number);
let part_path = root.join(disk_name).join(bucket).join(&path);
if !part_path.exists() {
eprintln!("Part file not found: {:?}", part_path);
return false;
}
let disk_path = root.join(disk_name);
let path_str = disk_path.to_str().expect("path");
let mut ep = Endpoint::try_from(path_str).expect("endpoint");
ep.set_pool_index(0);
ep.set_set_index(0);
ep.set_disk_index(0);
let opt = DiskOption {
cleanup: false,
health_check: false,
};
let disk = match new_disk(&ep, &opt).await {
Ok(d) => d,
Err(_) => {
eprintln!("Failed to create disk: {:?}", disk_path);
return false;
}
};
let read_length = shard_size;
let mut reader =
match create_bitrot_reader(None, Some(&disk), bucket, &path, 0, read_length, shard_size, checksum_algo.clone(), false)
.await
{
Ok(Some(r)) => r,
_ => {
eprintln!("Failed to create bitrot reader for EC part: {:?}", part_path);
return false;
}
};
let mut buf = vec![0u8; shard_size];
match reader.read(&mut buf).await {
Ok(n) if n > 0 => {
eprintln!(
"Successfully read {} bytes (EC part) via create_bitrot_reader with {:?}",
n, checksum_algo
);
true
}
_ => {
eprintln!("Failed to read EC part: {:?}", part_path);
false
}
}
}
#[tokio::test]
async fn test_legacy_bitrot_read() {
if !legacy_test_data_exists() {
eprintln!("Skipping legacy bitrot test: test1/vvvv xl.meta or RUSTFS_SKIP_LEGACY_TEST");
return;
}
// let root = workspace_root();
let root = PathBuf::from("/Users/weisd/project/minio");
let disk_name = "test";
let bucket = "vvvv";
// Try both EC (part files) and inline objects
let ok = run_legacy_bitrot_test_for_object(&root, disk_name, bucket, "ktvzip.tar.gz").await;
eprintln!("ok: {:?}", ok);
assert!(ok, "create_bitrot_reader failed for both ktvzip.tar.gz and path_traversal.md");
let ok = run_legacy_bitrot_test_for_object(&root, disk_name, bucket, "path_traversal.md").await;
assert!(ok, "create_bitrot_reader failed for path_traversal.md");
}
+24
View File
@@ -0,0 +1,24 @@
use rustfs_filemeta::FileMeta;
use std::{env, fs, path::PathBuf};
fn main() {
let path = env::args()
.nth(1)
.map(PathBuf::from)
.expect("usage: dump_versions <xl.meta path>");
let data = fs::read(&path).expect("read xl.meta");
let meta = FileMeta::load(&data).expect("load xl.meta");
let versions = meta
.into_file_info_versions("debug-bucket", "debug-object", true)
.expect("decode versions");
println!("path: {}", path.display());
println!("versions: {}", versions.versions.len());
for (idx, version) in versions.versions.iter().enumerate() {
println!(
"#{idx}: version_id={:?} deleted={} mark_deleted={} is_latest={} size={} mod_time={:?}",
version.version_id, version.deleted, version.mark_deleted, version.is_latest, version.size, version.mod_time
);
}
}
+17 -22
View File
@@ -16,7 +16,10 @@ use crate::{Error, ReplicationState, ReplicationStatusType, Result, TRANSITION_C
use bytes::Bytes;
use rmp_serde::Serializer;
use rustfs_utils::HashAlgorithm;
use rustfs_utils::http::headers::{RESERVED_METADATA_PREFIX_LOWER, RUSTFS_HEALING};
use rustfs_utils::http::{
SUFFIX_COMPRESSION, SUFFIX_DATA_MOVED, SUFFIX_HEALING, SUFFIX_INLINE_DATA, SUFFIX_TIER_FV_ID, SUFFIX_TIER_FV_MARKER,
SUFFIX_TIER_SKIP_FV_ID, contains_key_str, get_str, insert_str,
};
use s3s::dto::{RestoreStatus, Timestamp};
use s3s::header::X_AMZ_RESTORE;
use serde::{Deserialize, Serialize};
@@ -236,6 +239,8 @@ pub struct FileInfo {
// Combined checksum when object was uploaded
pub checksum: Option<Bytes>,
pub versioned: bool,
/// True when version meta was parsed via rmp_serde fallback (legacy format).
pub uses_legacy_checksum: bool,
}
impl FileInfo {
@@ -367,58 +372,48 @@ impl FileInfo {
}
pub fn set_healing(&mut self) {
self.metadata.insert(RUSTFS_HEALING.to_string(), "true".to_string());
insert_str(&mut self.metadata, SUFFIX_HEALING, "true".to_string());
}
pub fn set_tier_free_version_id(&mut self, version_id: &str) {
self.metadata
.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}{TIER_FV_ID}"), version_id.to_string());
insert_str(&mut self.metadata, SUFFIX_TIER_FV_ID, version_id.to_string());
}
pub fn tier_free_version_id(&self) -> String {
self.metadata[&format!("{RESERVED_METADATA_PREFIX_LOWER}{TIER_FV_ID}")].clone()
get_str(&self.metadata, SUFFIX_TIER_FV_ID).unwrap_or_default()
}
pub fn set_tier_free_version(&mut self) {
self.metadata
.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}{TIER_FV_MARKER}"), "".to_string());
insert_str(&mut self.metadata, SUFFIX_TIER_FV_MARKER, "".to_string());
}
pub fn set_skip_tier_free_version(&mut self) {
self.metadata
.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}{TIER_SKIP_FV_ID}"), "".to_string());
insert_str(&mut self.metadata, SUFFIX_TIER_SKIP_FV_ID, "".to_string());
}
pub fn skip_tier_free_version(&self) -> bool {
self.metadata
.contains_key(&format!("{RESERVED_METADATA_PREFIX_LOWER}{TIER_SKIP_FV_ID}"))
contains_key_str(&self.metadata, SUFFIX_TIER_SKIP_FV_ID)
}
pub fn tier_free_version(&self) -> bool {
self.metadata
.contains_key(&format!("{RESERVED_METADATA_PREFIX_LOWER}{TIER_FV_MARKER}"))
contains_key_str(&self.metadata, SUFFIX_TIER_FV_MARKER)
}
pub fn set_inline_data(&mut self) {
self.metadata
.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}inline-data").to_owned(), "true".to_owned());
insert_str(&mut self.metadata, SUFFIX_INLINE_DATA, "true".to_string());
}
pub fn set_data_moved(&mut self) {
self.metadata
.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}data-moved").to_owned(), "true".to_owned());
insert_str(&mut self.metadata, SUFFIX_DATA_MOVED, "true".to_string());
}
pub fn inline_data(&self) -> bool {
self.metadata
.contains_key(format!("{RESERVED_METADATA_PREFIX_LOWER}inline-data").as_str())
&& !self.is_remote()
contains_key_str(&self.metadata, SUFFIX_INLINE_DATA) && !self.is_remote()
}
/// Check if the object is compressed
pub fn is_compressed(&self) -> bool {
self.metadata
.contains_key(&format!("{RESERVED_METADATA_PREFIX_LOWER}compression"))
contains_key_str(&self.metadata, SUFFIX_COMPRESSION)
}
/// Check if the object is remote (transitioned to another tier)
+185 -95
View File
@@ -13,17 +13,20 @@
// limitations under the License.
use crate::{
ErasureAlgo, ErasureInfo, Error, FileInfo, FileInfoVersions, InlineData, ObjectPartInfo, RawFileInfo, ReplicationState,
ReplicationStatusType, Result, TIER_FV_ID, TIER_FV_MARKER, VersionPurgeStatusType, is_restored_object_on_disk,
ErasureAlgo, ErasureInfo, Error, FileInfo, FileInfoVersions, InlineData, NULL_VERSION_ID, ObjectPartInfo, RawFileInfo,
ReplicationState, ReplicationStatusType, Result, VersionPurgeStatusType, is_restored_object_on_disk,
replication_statuses_map, version_purge_statuses_map,
};
use byteorder::ByteOrder;
use bytes::Bytes;
use rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS;
use rustfs_utils::http::headers::{
self, AMZ_META_UNENCRYPTED_CONTENT_LENGTH, AMZ_META_UNENCRYPTED_CONTENT_MD5, AMZ_RESTORE_EXPIRY_DAYS,
AMZ_RESTORE_REQUEST_DATE, AMZ_STORAGE_CLASS, RESERVED_METADATA_PREFIX, RESERVED_METADATA_PREFIX_LOWER,
VERSION_PURGE_STATUS_KEY,
AMZ_META_UNENCRYPTED_CONTENT_LENGTH, AMZ_META_UNENCRYPTED_CONTENT_MD5, AMZ_RESTORE_EXPIRY_DAYS, AMZ_RESTORE_REQUEST_DATE,
AMZ_STORAGE_CLASS,
};
use rustfs_utils::http::{
AMZ_BUCKET_REPLICATION_STATUS, SUFFIX_DATA_MOV, SUFFIX_HEALING, SUFFIX_PURGESTATUS, SUFFIX_REPLICA_STATUS,
SUFFIX_REPLICA_TIMESTAMP, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TIMESTAMP, has_internal_suffix, insert_bytes,
is_internal_key,
};
use s3s::header::X_AMZ_RESTORE;
use serde::{Deserialize, Serialize};
@@ -48,7 +51,8 @@ pub static XL_FILE_HEADER: [u8; 4] = [b'X', b'L', b'2', b' '];
static XL_FILE_VERSION_MAJOR: u16 = 1;
static XL_FILE_VERSION_MINOR: u16 = 3;
static XL_HEADER_VERSION: u8 = 3;
pub static XL_META_VERSION: u8 = 2;
pub static XL_META_VERSION: u8 = 3;
/// Legacy format (main branch): meta_ver=2 with file/header versions 1.3.3.
static XXHASH_SEED: u64 = 0;
const XL_FLAG_FREE_VERSION: u8 = 1 << 0;
@@ -58,6 +62,18 @@ const _XL_FLAG_INLINE_DATA: u8 = 1 << 2;
const META_DATA_READ_DEFAULT: usize = 4 << 10;
const MSGP_UINT32_SIZE: usize = 5;
/// Max object versions per object, default is 10000
const DEFAULT_OBJECT_MAX_VERSIONS: usize = 10000;
/// Returns the inline data map key for a version_id. "null" for null version.
pub(crate) fn data_key_for_version(version_id: Option<Uuid>) -> String {
if version_id.is_none() || version_id == Some(Uuid::nil()) {
NULL_VERSION_ID.to_string()
} else {
version_id.unwrap_or_default().to_string()
}
}
pub const TRANSITION_COMPLETE: &str = "complete";
pub const TRANSITION_PENDING: &str = "pending";
@@ -68,8 +84,14 @@ pub const TRANSITIONED_OBJECTNAME: &str = "transitioned-object";
pub const TRANSITIONED_VERSION_ID: &str = "transitioned-versionID";
pub const TRANSITION_TIER: &str = "transition-tier";
/// Returns true if the key is a transient internal flag that should not be persisted to meta_sys.
pub fn is_skip_meta_key(key: &str) -> bool {
has_internal_suffix(key, SUFFIX_HEALING) || has_internal_suffix(key, SUFFIX_DATA_MOV)
}
mod codec;
mod inline_data;
mod msgp_decode;
mod validation;
mod version;
@@ -171,11 +193,10 @@ impl FileMeta {
for (k, v) in fi.metadata.iter() {
// Split metadata into meta_user and meta_sys based on prefix
// This logic must match From<FileInfo> for MetaObject
if k.len() > RESERVED_METADATA_PREFIX.len()
&& (k.starts_with(RESERVED_METADATA_PREFIX) || k.starts_with(RESERVED_METADATA_PREFIX_LOWER))
{
let is_system = is_internal_key(k);
if is_system {
// Skip internal flags that shouldn't be persisted
if k == headers::X_RUSTFS_HEALING || k == headers::X_RUSTFS_DATA_MOV {
if is_skip_meta_key(k) {
continue;
}
// Insert into meta_sys
@@ -221,18 +242,26 @@ impl FileMeta {
}
pub fn add_version(&mut self, mut fi: FileInfo) -> Result<()> {
// empty version_id means "null" (versioning disabled/suspended)
if fi.version_id.is_none() {
fi.version_id = Some(Uuid::nil());
}
let version_key = data_key_for_version(fi.version_id);
let mut next_data = self.data.clone();
if let Some(ref data) = fi.data {
let key = fi.version_id.unwrap_or_default().to_string();
self.data.replace(&key, data.to_vec())?;
next_data.replace(&version_key, data.to_vec())?;
} else {
let _ = next_data.remove_key(&version_key)?;
}
let version = FileMetaVersion::from(fi);
self.add_version_filemata(version)
self.add_version_filemata(version)?;
self.data = next_data;
Ok(())
}
pub fn add_version_filemata(&mut self, version: FileMetaVersion) -> Result<()> {
@@ -240,13 +269,12 @@ impl FileMeta {
return Err(Error::other("file meta version invalid"));
}
// TODO: make it configurable
// 1000 is the limit of versions
// if self.versions.len() + 1 > 1000 {
// return Err(Error::other(
// "You've exceeded the limit on the number of versions you can create on this object",
// ));
// }
// check max versions limit
if self.versions.len() + 1 > DEFAULT_OBJECT_MAX_VERSIONS {
return Err(Error::other(
"You've exceeded the limit on the number of versions you can create on this object",
));
}
if self.versions.is_empty() {
self.versions.push(FileMetaShallowVersion::try_from(version)?);
@@ -255,21 +283,44 @@ impl FileMeta {
let vid = version.get_version_id();
if let Some(fidx) = self.versions.iter().position(|v| v.header.version_id == vid) {
// Match existing version for replace; null version: None and Some(nil) are equivalent
let matches = |h: &Option<Uuid>| {
let v_null = vid.is_none() || vid == Some(Uuid::nil());
let h_null = h.is_none() || *h == Some(Uuid::nil());
(v_null && h_null) || (vid == *h)
};
if let Some(fidx) = self.versions.iter().position(|v| matches(&v.header.version_id)) {
return self.set_idx(fidx, version);
}
// append placeholder to find insert position
let placeholder = FileMetaShallowVersion {
header: FileMetaVersionHeader {
mod_time: None, // None sorts before any real mod_time
..Default::default()
},
meta: Vec::new(),
};
self.versions.push(placeholder);
let mod_time = version.get_mod_time();
let new_shallow = FileMetaShallowVersion::try_from(version)?;
for (idx, exist) in self.versions.iter().enumerate() {
if let Some(ref ex_mt) = exist.header.mod_time
&& let Some(ref in_md) = mod_time
&& ex_mt <= in_md
{
self.versions.insert(idx, FileMetaShallowVersion::try_from(version)?);
let ex_mt = exist.header.mod_time;
let insert_here = match (ex_mt, mod_time) {
(None, _) => true, // placeholder: always insert before
(Some(em), Some(nm)) => em <= nm,
(Some(_), None) => false,
};
if insert_here {
self.versions.insert(idx, new_shallow);
self.versions.pop(); // remove placeholder
return Ok(());
}
}
self.versions.pop(); // remove placeholder on fallback
Err(Error::other("add_version failed"))
// if !ver.valid() {
@@ -343,8 +394,9 @@ impl FileMeta {
&& let Some(delete_marker) = ventry.delete_marker.as_mut()
{
if fi.delete_marker_replication_status() == ReplicationStatusType::Replica {
delete_marker.meta_sys.insert(
format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replica-status"),
insert_bytes(
&mut delete_marker.meta_sys,
SUFFIX_REPLICA_STATUS,
fi.replication_state_internal
.as_ref()
.map(|v| v.replica_status.clone())
@@ -353,8 +405,9 @@ impl FileMeta {
.as_bytes()
.to_vec(),
);
delete_marker.meta_sys.insert(
format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replica-timestamp"),
insert_bytes(
&mut delete_marker.meta_sys,
SUFFIX_REPLICA_TIMESTAMP,
fi.replication_state_internal
.as_ref()
.map(|v| v.replica_timestamp.unwrap_or(OffsetDateTime::UNIX_EPOCH).to_string())
@@ -363,8 +416,9 @@ impl FileMeta {
.to_vec(),
);
} else {
delete_marker.meta_sys.insert(
format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replication-status"),
insert_bytes(
&mut delete_marker.meta_sys,
SUFFIX_REPLICATION_STATUS,
fi.replication_state_internal
.as_ref()
.map(|v| v.replication_status_internal.clone().unwrap_or_default())
@@ -372,8 +426,9 @@ impl FileMeta {
.as_bytes()
.to_vec(),
);
delete_marker.meta_sys.insert(
format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replication-timestamp"),
insert_bytes(
&mut delete_marker.meta_sys,
SUFFIX_REPLICATION_TIMESTAMP,
fi.replication_state_internal
.as_ref()
.map(|v| v.replication_timestamp.unwrap_or(OffsetDateTime::UNIX_EPOCH).to_string())
@@ -387,15 +442,14 @@ impl FileMeta {
if !fi.version_purge_status().is_empty()
&& let Some(delete_marker) = ventry.delete_marker.as_mut()
{
delete_marker.meta_sys.insert(
VERSION_PURGE_STATUS_KEY.to_string(),
fi.replication_state_internal
.as_ref()
.map(|v| v.version_purge_status_internal.clone().unwrap_or_default())
.unwrap_or_default()
.as_bytes()
.to_vec(),
);
let value = fi
.replication_state_internal
.as_ref()
.map(|v| v.version_purge_status_internal.clone().unwrap_or_default())
.unwrap_or_default()
.as_bytes()
.to_vec();
insert_bytes(&mut delete_marker.meta_sys, SUFFIX_PURGESTATUS, value);
}
if let Some(delete_marker) = ventry.delete_marker.as_mut() {
@@ -433,8 +487,9 @@ impl FileMeta {
if let Some(delete_marker) = v.delete_marker.as_mut() {
if !fi.delete_marker_replication_status().is_empty() {
if fi.delete_marker_replication_status() == ReplicationStatusType::Replica {
delete_marker.meta_sys.insert(
format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replica-status"),
insert_bytes(
&mut delete_marker.meta_sys,
SUFFIX_REPLICA_STATUS,
fi.replication_state_internal
.as_ref()
.map(|v| v.replica_status.clone())
@@ -443,8 +498,9 @@ impl FileMeta {
.as_bytes()
.to_vec(),
);
delete_marker.meta_sys.insert(
format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replica-timestamp"),
insert_bytes(
&mut delete_marker.meta_sys,
SUFFIX_REPLICA_TIMESTAMP,
fi.replication_state_internal
.as_ref()
.map(|v| v.replica_timestamp.unwrap_or(OffsetDateTime::UNIX_EPOCH).to_string())
@@ -453,8 +509,9 @@ impl FileMeta {
.to_vec(),
);
} else {
delete_marker.meta_sys.insert(
format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replication-status"),
insert_bytes(
&mut delete_marker.meta_sys,
SUFFIX_REPLICATION_STATUS,
fi.replication_state_internal
.as_ref()
.map(|v| v.replication_status_internal.clone().unwrap_or_default())
@@ -462,8 +519,9 @@ impl FileMeta {
.as_bytes()
.to_vec(),
);
delete_marker.meta_sys.insert(
format!("{}{}", RESERVED_METADATA_PREFIX_LOWER, "replication-timestamp"),
insert_bytes(
&mut delete_marker.meta_sys,
SUFFIX_REPLICATION_TIMESTAMP,
fi.replication_state_internal
.as_ref()
.map(|v| v.replication_timestamp.unwrap_or(OffsetDateTime::UNIX_EPOCH).to_string())
@@ -502,15 +560,14 @@ impl FileMeta {
let mut v = self.get_idx(i)?;
if let Some(obj) = v.object.as_mut() {
obj.meta_sys.insert(
VERSION_PURGE_STATUS_KEY.to_string(),
fi.replication_state_internal
.as_ref()
.map(|v| v.version_purge_status_internal.clone().unwrap_or_default())
.unwrap_or_default()
.as_bytes()
.to_vec(),
);
let value = fi
.replication_state_internal
.as_ref()
.map(|v| v.version_purge_status_internal.clone().unwrap_or_default())
.unwrap_or_default()
.as_bytes()
.to_vec();
insert_bytes(&mut obj.meta_sys, SUFFIX_PURGESTATUS, value);
for (k, v) in fi
.replication_state_internal
.as_ref()
@@ -617,13 +674,14 @@ impl FileMeta {
// TODO: freeVersion
if header.free_version() {
non_free_versions -= 1;
if include_free_versions && found_free_version.is_none() {
let mut found_free_fi = FileMetaVersion::default();
if found_free_fi.unmarshal_msg(&ver.meta).is_ok() && found_free_fi.version_type != VersionType::Invalid {
let mut free_fi = found_free_fi.into_fileinfo(volume, path, all_parts);
free_fi.is_latest = true;
found_free_version = Some(free_fi);
}
if include_free_versions
&& found_free_version.is_none()
&& let Ok(found_free_fi) = ver.parse_version_meta()
&& found_free_fi.version_type != VersionType::Invalid
{
let mut free_fi = found_free_fi.into_fileinfo(volume, path, all_parts);
free_fi.is_latest = true;
found_free_version = Some(free_fi);
}
if header.version_id != Some(vid) {
@@ -650,10 +708,10 @@ impl FileMeta {
fi.successor_mod_time = succ_mod_time;
}
if read_data {
if read_data && fi.inline_data() {
fi.data = self
.data
.find(fi.version_id.unwrap_or_default().to_string().as_str())?
.find(data_key_for_version(fi.version_id).as_str())?
.map(bytes::Bytes::from);
}
@@ -725,9 +783,7 @@ impl FileMeta {
pub fn into_file_info_versions(&self, volume: &str, path: &str, all_parts: bool) -> Result<FileInfoVersions> {
let mut versions = Vec::new();
for version in self.versions.iter() {
let mut file_version = FileMetaVersion::default();
file_version.unmarshal_msg(&version.meta)?;
let fi = file_version.into_fileinfo(volume, path, all_parts);
let fi = version.into_fileinfo(volume, path, all_parts)?;
versions.push(fi);
}
@@ -770,29 +826,9 @@ impl FileMeta {
self.versions.first().unwrap().header.mod_time
}
/// Load or convert from buffer
/// Load or convert from buffer. Handles both current (meta_ver=3) and legacy (meta_ver=2) formats.
pub fn load_or_convert(buf: &[u8]) -> Result<Self> {
// Try to load as current format first
match Self::load(buf) {
Ok(meta) => Ok(meta),
Err(_) => {
// Try to convert from legacy format
Self::load_legacy(buf)
}
}
}
/// Load legacy format
pub fn load_legacy(_buf: &[u8]) -> Result<Self> {
// Implementation for loading legacy xl.meta formats
// This would handle conversion from older formats
Err(Error::other("Legacy format not yet implemented"))
}
/// Add legacy version
pub fn add_legacy(&mut self, _legacy_obj: &str) -> Result<()> {
// Implementation for adding legacy xl.meta v1 objects
Err(Error::other("Legacy version addition not yet implemented"))
Self::load(buf)
}
/// List all versions as FileInfo
@@ -1061,6 +1097,7 @@ mod test {
object: None,
delete_marker: None,
write_version: 1,
uses_legacy_checksum: false,
};
assert!(legacy_version.is_legacy(), "Should be recognized as a Legacy version");
@@ -1178,6 +1215,7 @@ mod test {
}),
delete_marker: None,
write_version: 1,
uses_legacy_checksum: false,
};
let shallow_version = FileMetaShallowVersion::try_from(version).expect("Conversion failed");
@@ -1391,6 +1429,7 @@ mod test {
object: None,
delete_marker: Some(delete_marker),
write_version: (i + 100) as u64,
uses_legacy_checksum: false,
};
let shallow_version = FileMetaShallowVersion::try_from(delete_version).unwrap();
@@ -1458,6 +1497,57 @@ mod test {
assert!(fm.validate_integrity().is_ok());
}
#[test]
fn test_add_version_clears_stale_inline_data_for_null_version() {
let mut fm = FileMeta::new();
let mut inline_fi = crate::fileinfo::FileInfo::new("test", 2, 1);
inline_fi.mod_time = Some(OffsetDateTime::now_utc());
inline_fi.data = Some(Bytes::new());
inline_fi.set_inline_data();
fm.add_version(inline_fi).unwrap();
let inline_version = fm.into_fileinfo("bucket", "test", "", true, false, true).unwrap();
assert!(inline_version.inline_data());
assert_eq!(inline_version.data, Some(Bytes::new()));
let mut disk_fi = crate::fileinfo::FileInfo::new("test", 2, 1);
disk_fi.mod_time = Some(OffsetDateTime::now_utc() + time::Duration::seconds(1));
disk_fi.size = 1024;
fm.add_version(disk_fi).unwrap();
let latest = fm.into_fileinfo("bucket", "test", "", true, false, true).unwrap();
assert!(!latest.inline_data());
assert!(latest.data.is_none());
assert!(
fm.data
.find(data_key_for_version(latest.version_id).as_str())
.unwrap()
.is_none()
);
}
#[test]
fn test_add_version_keeps_inline_data_when_version_insert_fails() {
let mut fm = FileMeta::new();
let mut inline_fi = crate::fileinfo::FileInfo::new("test", 2, 1);
inline_fi.mod_time = Some(OffsetDateTime::now_utc());
inline_fi.data = Some(Bytes::from_static(b"inline"));
inline_fi.set_inline_data();
fm.add_version(inline_fi).unwrap();
let before = fm.data.find(data_key_for_version(Some(Uuid::nil())).as_str()).unwrap();
assert_eq!(before, Some(Bytes::from_static(b"inline").to_vec()));
let invalid_disk_fi = crate::fileinfo::FileInfo::new("test", 2, 1);
let err = fm.add_version(invalid_disk_fi).unwrap_err();
assert!(err.to_string().contains("file meta version invalid"));
let after = fm.data.find(data_key_for_version(Some(Uuid::nil())).as_str()).unwrap();
assert_eq!(after, Some(Bytes::from_static(b"inline").to_vec()));
}
#[test]
fn test_version_merge_scenarios() {
// Test various version merge scenarios
+19 -5
View File
@@ -22,10 +22,26 @@ impl FileMeta {
pub fn load(buf: &[u8]) -> Result<FileMeta> {
let mut xl = FileMeta::default();
xl.unmarshal_msg(buf)?;
Ok(xl)
}
/// Read (major, minor, header_ver, meta_ver) from xl.meta without full parse.
/// Returns Err if format is not feat (e.g. legacy uses rmp_serde in meta block).
pub fn read_format_versions(buf: &[u8]) -> Result<(u16, u16, u8, u8)> {
let (buf, major, minor) = Self::check_xl2_v1(buf)?;
if buf.len() < 5 {
return Err(Error::other("insufficient data for metadata length prefix"));
}
let (mut size_buf, buf) = buf.split_at(5);
let bin_len = rmp::decode::read_bin_len(&mut size_buf)?;
if buf.len() < bin_len as usize {
return Err(Error::other("insufficient data for metadata"));
}
let (meta, _) = buf.split_at(bin_len as usize);
let (_, header_ver, meta_ver, _) = Self::decode_xl_headers(meta)?;
Ok((major, minor, header_ver, meta_ver))
}
pub fn check_xl2_v1(buf: &[u8]) -> Result<(&[u8], u16, u16)> {
if buf.len() < 8 {
return Err(Error::other("xl file header not exists"));
@@ -294,11 +310,9 @@ impl FileMeta {
let offset = wr.len();
// xl header
rmp::encode::write_uint8(&mut wr, XL_HEADER_VERSION)?;
rmp::encode::write_uint8(&mut wr, XL_META_VERSION)?;
rmp::encode::write_uint(&mut wr, XL_HEADER_VERSION as u64)?;
rmp::encode::write_uint(&mut wr, XL_META_VERSION as u64)?;
// versions
rmp::encode::write_sint(&mut wr, self.versions.len() as i64)?;
for ver in self.versions.iter() {
+7 -5
View File
@@ -40,10 +40,14 @@ impl FileMeta {
/// Count shared data directories
pub fn shared_data_dir_count(&self, version_id: Option<Uuid>, data_dir: Option<Uuid>) -> usize {
let version_id = version_id.unwrap_or_default();
let vid = version_id.unwrap_or_default();
if self.data.entries().unwrap_or_default() > 0
&& self.data.find(version_id.to_string().as_str()).unwrap_or_default().is_some()
&& self
.data
.find(super::data_key_for_version(version_id).as_str())
.unwrap_or_default()
.is_some()
{
return 0;
}
@@ -51,9 +55,7 @@ impl FileMeta {
self.versions
.iter()
.filter(|v| {
v.header.version_type == VersionType::Object
&& v.header.version_id != Some(version_id)
&& v.header.uses_data_dir()
v.header.version_type == VersionType::Object && v.header.version_id != Some(vid) && v.header.uses_data_dir()
})
.filter_map(|v| FileMetaVersion::decode_data_dir_from_meta(&v.meta).ok())
.filter(|&dir| dir.is_some() && dir == data_dir)
+212
View File
@@ -0,0 +1,212 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{Error, Result};
use rmp::Marker;
use std::io::Read;
/// Reader that prepends a single byte to the stream. Used when we've read the marker
/// and need to pass it to a decoder that expects to read the marker itself.
pub(crate) struct PrependByteReader<'a, R> {
pub(crate) byte: Option<u8>,
pub(crate) inner: &'a mut R,
}
impl<R: Read> Read for PrependByteReader<'_, R> {
#[allow(clippy::collapsible_if)]
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
if let Some(b) = self.byte.take() {
if !buf.is_empty() {
buf[0] = b;
return Ok(1);
}
self.byte = Some(b);
}
self.inner.read(buf)
}
}
/// Read MessagePack nil or array length. Returns None for nil, Some(len) for array.
pub(crate) fn read_nil_or_array_len<R: Read>(rd: &mut R) -> Result<Option<usize>> {
let mut buf = [0u8; 1];
rd.read_exact(&mut buf).map_err(Error::from)?;
let marker = buf[0];
match marker {
0xc0 => Ok(None), // nil
0x90..=0x9f => Ok(Some((marker & 0x0f) as usize)),
0xdc => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::from)?;
Ok(Some(u16::from_be_bytes(b) as usize))
}
0xdd => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::from)?;
Ok(Some(u32::from_be_bytes(b) as usize))
}
_ => Err(Error::other(format!("expected nil or array, got marker 0x{marker:02x}"))),
}
}
/// Read MessagePack nil or map length. Returns None for nil, Some(len) for map.
pub(crate) fn read_nil_or_map_len<R: Read>(rd: &mut R) -> Result<Option<usize>> {
let mut buf = [0u8; 1];
rd.read_exact(&mut buf).map_err(Error::from)?;
let marker = buf[0];
match marker {
0xc0 => Ok(None), // nil
0x80..=0x8f => Ok(Some((marker & 0x0f) as usize)),
0xde => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::from)?;
Ok(Some(u16::from_be_bytes(b) as usize))
}
0xdf => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::from)?;
Ok(Some(u32::from_be_bytes(b) as usize))
}
_ => Err(Error::other(format!("expected nil or map, got marker 0x{marker:02x}"))),
}
}
/// Skip a single MessagePack value. Used for unknown map keys.
pub(crate) fn skip_msgp_value<R: Read>(rd: &mut R) -> Result<()> {
let marker = rmp::decode::read_marker(rd).map_err(Error::from)?;
let skip_len: usize = match marker {
Marker::Null | Marker::False | Marker::True => 0,
Marker::FixPos(_) | Marker::FixNeg(_) => 0,
Marker::U8 => 1,
Marker::U16 => 2,
Marker::U32 => 4,
Marker::U64 => 8,
Marker::I8 => 1,
Marker::I16 => 2,
Marker::I32 => 4,
Marker::I64 => 8,
Marker::F32 => 4,
Marker::F64 => 8,
Marker::FixStr(n) => n as usize,
Marker::Str8 => {
let mut b = [0u8; 1];
rd.read_exact(&mut b).map_err(Error::from)?;
b[0] as usize
}
Marker::Str16 => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::from)?;
u16::from_be_bytes(b) as usize
}
Marker::Str32 => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::from)?;
u32::from_be_bytes(b) as usize
}
Marker::Bin8 => {
let mut b = [0u8; 1];
rd.read_exact(&mut b).map_err(Error::from)?;
b[0] as usize
}
Marker::Bin16 => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::from)?;
u16::from_be_bytes(b) as usize
}
Marker::Bin32 => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::from)?;
u32::from_be_bytes(b) as usize
}
Marker::FixArray(n) => {
for _ in 0..n {
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::Array16 => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::from)?;
let n = u16::from_be_bytes(b);
for _ in 0..n {
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::Array32 => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::from)?;
let n = u32::from_be_bytes(b);
for _ in 0..n {
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::FixMap(n) => {
for _ in 0..n {
skip_msgp_value(rd)?;
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::Map16 => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::from)?;
let n = u16::from_be_bytes(b);
for _ in 0..n {
skip_msgp_value(rd)?;
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::Map32 => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::from)?;
let n = u32::from_be_bytes(b);
for _ in 0..n {
skip_msgp_value(rd)?;
skip_msgp_value(rd)?;
}
return Ok(());
}
Marker::FixExt1 => 1,
Marker::FixExt2 => 2,
Marker::FixExt4 => 4,
Marker::FixExt8 => 8,
Marker::FixExt16 => 16,
Marker::Ext8 => {
let mut b = [0u8; 1];
rd.read_exact(&mut b).map_err(Error::from)?;
let len = b[0] as usize;
1 + len // type byte + data
}
Marker::Ext16 => {
let mut b = [0u8; 2];
rd.read_exact(&mut b).map_err(Error::from)?;
let len = u16::from_be_bytes(b) as usize;
2 + len // type bytes + data
}
Marker::Ext32 => {
let mut b = [0u8; 4];
rd.read_exact(&mut b).map_err(Error::from)?;
let len = u32::from_be_bytes(b) as usize;
4 + len // type bytes + data
}
Marker::Reserved => 0,
};
if skip_len > 0 {
let mut buf = vec![0u8; skip_len];
rd.read_exact(&mut buf).map_err(Error::from)?;
}
Ok(())
}
File diff suppressed because it is too large Load Diff
+52
View File
@@ -181,6 +181,58 @@ impl InlineData {
self.serialize(keys, values)
}
pub fn remove_key(&mut self, key: &str) -> Result<bool> {
let buf = self.after_version();
if buf.is_empty() {
return Ok(false);
}
let mut cur = Cursor::new(buf);
let mut fields_len = rmp::decode::read_map_len(&mut cur)? as usize;
let mut keys = Vec::with_capacity(fields_len);
let mut values = Vec::with_capacity(fields_len);
let mut found = false;
while fields_len > 0 {
fields_len -= 1;
let str_len = rmp::decode::read_str_len(&mut cur)?;
let mut field_buff = vec![0u8; str_len as usize];
cur.read_exact(&mut field_buff)?;
let find_key = String::from_utf8(field_buff)?;
let bin_len = rmp::decode::read_bin_len(&mut cur)? as usize;
let start = cur.position() as usize;
let end = start + bin_len;
cur.set_position(end as u64);
if find_key == key {
found = true;
continue;
}
keys.push(find_key);
values.push(buf[start..end].to_vec());
}
if !found {
return Ok(false);
}
if keys.is_empty() {
self.0 = Vec::new();
return Ok(true);
}
self.serialize(keys, values)?;
Ok(true)
}
pub fn remove(&mut self, remove_keys: Vec<Uuid>) -> Result<bool> {
let buf = self.after_version();
if buf.is_empty() {
+2 -2
View File
@@ -15,7 +15,7 @@
use bytes::Bytes;
use core::fmt;
use regex::Regex;
use rustfs_utils::http::RESERVED_METADATA_PREFIX_LOWER;
use rustfs_utils::http::internal_key_rustfs;
use serde::{Deserialize, Serialize};
use std::any::Any;
use std::collections::HashMap;
@@ -859,7 +859,7 @@ pub fn get_replication_state(rinfos: &ReplicatedInfos, prev_state: &ReplicationS
}
pub fn target_reset_header(arn: &str) -> String {
format!("{RESERVED_METADATA_PREFIX_LOWER}{REPLICATION_RESET}-{arn}")
internal_key_rustfs(&format!("{REPLICATION_RESET}-{arn}"))
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
+6
View File
@@ -56,6 +56,7 @@ pub fn create_real_xlmeta() -> Result<Vec<u8>> {
object: Some(object_version),
delete_marker: None,
write_version: 1,
uses_legacy_checksum: false,
};
let shallow_version = FileMetaShallowVersion::try_from(file_version)?;
@@ -74,6 +75,7 @@ pub fn create_real_xlmeta() -> Result<Vec<u8>> {
object: None,
delete_marker: Some(delete_marker),
write_version: 2,
uses_legacy_checksum: false,
};
let delete_shallow_version = FileMetaShallowVersion::try_from(delete_file_version)?;
@@ -86,6 +88,7 @@ pub fn create_real_xlmeta() -> Result<Vec<u8>> {
object: None,
delete_marker: None,
write_version: 3,
uses_legacy_checksum: false,
};
let mut legacy_shallow = FileMetaShallowVersion::try_from(legacy_version)?;
@@ -139,6 +142,7 @@ pub fn create_complex_xlmeta() -> Result<Vec<u8>> {
object: Some(object_version),
delete_marker: None,
write_version: (i + 1) as u64,
uses_legacy_checksum: false,
};
let shallow_version = FileMetaShallowVersion::try_from(file_version)?;
@@ -158,6 +162,7 @@ pub fn create_complex_xlmeta() -> Result<Vec<u8>> {
object: None,
delete_marker: Some(delete_marker),
write_version: (i + 100) as u64,
uses_legacy_checksum: false,
};
let delete_shallow_version = FileMetaShallowVersion::try_from(delete_file_version)?;
@@ -245,6 +250,7 @@ pub fn create_xlmeta_with_inline_data() -> Result<Vec<u8>> {
object: Some(object_version),
delete_marker: None,
write_version: 1,
uses_legacy_checksum: false,
};
let shallow_version = FileMetaShallowVersion::try_from(file_version)?;
+21 -49
View File
@@ -38,6 +38,11 @@ use walkdir::WalkDir;
const HEAL_FORMAT_WAIT_TIMEOUT: Duration = Duration::from_secs(25);
const HEAL_FORMAT_WAIT_INTERVAL: Duration = Duration::from_millis(250);
const NON_INLINE_TEST_DATA_SIZE: usize = 256 * 1024 + 137;
fn non_inline_test_data() -> Vec<u8> {
(0..NON_INLINE_TEST_DATA_SIZE).map(|idx| (idx % 251) as u8).collect()
}
async fn wait_for_path_exists(path: &Path, timeout: Duration, interval: Duration) -> bool {
let deadline = tokio::time::Instant::now() + timeout;
@@ -177,10 +182,10 @@ mod serial_tests {
// Create test bucket and object
let bucket_name = "test-heal-object-basic";
let object_name = "test-object.txt";
let test_data = b"Hello, this is test data for healing!";
let test_data = non_inline_test_data();
create_test_bucket(&ecstore, bucket_name).await;
upload_test_object(&ecstore, bucket_name, object_name, test_data).await;
upload_test_object(&ecstore, bucket_name, object_name, &test_data).await;
let _obj_dir = disk_paths[0].join(bucket_name).join(object_name);
// ─── 1️⃣ delete single data shard file ─────────────────────────────────────
let obj_dir = disk_paths[0].join(bucket_name).join(object_name);
@@ -198,55 +203,22 @@ mod serial_tests {
assert!(!target_part.exists());
println!("✅ Deleted shard part file: {target_part:?}");
// Create heal manager with faster interval
let cfg = HealConfig {
heal_interval: Duration::from_millis(1),
let heal_opts = HealOpts {
recreate: true,
remove: false,
update_parity: true,
..Default::default()
};
let heal_manager = HealManager::new(heal_storage.clone(), Some(cfg));
heal_manager.start().await.unwrap();
// Submit heal request for the object
let heal_request = HealRequest::new(
HealType::Object {
bucket: bucket_name.to_string(),
object: object_name.to_string(),
version_id: None,
},
HealOptions {
dry_run: false,
recursive: false,
remove_corrupted: false,
recreate_missing: true,
scan_mode: HealScanMode::Normal,
update_parity: true,
timeout: Some(Duration::from_secs(300)),
pool_index: None,
set_index: None,
},
HealPriority::Normal,
);
let task_id = heal_manager
.submit_heal_request(heal_request)
let (object_result, object_error) = heal_storage
.heal_object(bucket_name, object_name, None, &heal_opts)
.await
.expect("Failed to submit heal request");
.expect("failed to heal object");
info!("heal_object result: {:?}, error: {:?}", object_result, object_error);
assert!(object_error.is_none(), "heal_object returned error: {object_error:?}");
info!("Submitted heal request with task ID: {}", task_id);
// Wait for task completion
tokio::time::sleep(tokio::time::Duration::from_secs(8)).await;
// Attempt to fetch task status (might be removed if finished)
match heal_manager.get_task_status(&task_id).await {
Ok(status) => info!("Task status: {:?}", status),
Err(e) => info!("Task status not found (likely completed): {}", e),
}
// ─── 2️⃣ verify each part file is restored ───────
assert!(target_part.exists());
// ─── 3️⃣ verify object data integrity by actually reading it ───────
// `test_heal_format_with_data` covers on-disk shard restoration. Here we
// focus on the object-level healing contract: the object must remain
// readable with intact contents after healing.
let mut reader = ecstore
.get_object_reader(bucket_name, object_name, None, HeaderMap::new(), &ObjectOptions::default())
.await
@@ -364,10 +336,10 @@ mod serial_tests {
// Create test bucket and object
let bucket_name = "test-heal-format-with-data";
let object_name = "test-object.txt";
let test_data = b"Hello, this is test data for healing!";
let test_data = non_inline_test_data();
create_test_bucket(&ecstore, bucket_name).await;
upload_test_object(&ecstore, bucket_name, object_name, test_data).await;
upload_test_object(&ecstore, bucket_name, object_name, &test_data).await;
let obj_dir = disk_paths[0].join(bucket_name).join(object_name);
let target_part = WalkDir::new(&obj_dir)
.min_depth(2)
+58 -13
View File
@@ -32,7 +32,7 @@ use rustfs_policy::{
format::Format,
policy::{Policy, PolicyDoc, default::DEFAULT_POLICIES, iam_policy_claim_name_sa},
};
use rustfs_utils::path::path_join_buf;
use rustfs_utils::{get_env_opt_str, path::path_join_buf};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use std::sync::atomic::AtomicU8;
@@ -147,7 +147,7 @@ where
// Background ticker for synchronization
// Check if environment variable is set
let skip_background_task = std::env::var("RUSTFS_SKIP_BACKGROUND_TASK").is_ok();
let skip_background_task = get_env_opt_str("RUSTFS_SKIP_BACKGROUND_TASK").is_some();
if !skip_background_task {
// Background thread starts periodic updates or receives signal updates
@@ -595,7 +595,11 @@ where
Ok(users
.values()
.filter_map(|x| {
if !access_key.is_empty() && x.credentials.parent_user.as_str() == access_key && x.credentials.is_temp() {
if !access_key.is_empty()
&& x.credentials.parent_user.as_str() == access_key
&& x.credentials.is_temp()
&& !x.credentials.is_service_account()
{
let mut c = x.credentials.clone();
c.secret_key = String::new();
c.session_token = String::new();
@@ -1419,9 +1423,6 @@ where
}
pub async fn get_group_description(&self, name: &str) -> Result<GroupDesc> {
let (ps, updated_at) = self.policy_db_get_internal(name, true, false).await?;
let policy = ps.join(",");
let gi = self
.cache
.groups
@@ -1430,13 +1431,25 @@ where
.cloned()
.ok_or(Error::NoSuchGroup(name.to_string()))?;
Ok(GroupDesc {
name: name.to_string(),
policy,
members: gi.members,
updated_at: Some(updated_at),
status: gi.status,
})
let mapped_policy = if let Some(policy) = self.cache.group_policies.load().get(name).cloned() {
Some(policy)
} else {
let mut policies = HashMap::new();
if let Err(err) = self.api.load_mapped_policy(name, UserType::Reg, true, &mut policies).await
&& !is_err_no_such_policy(&err)
{
return Err(err);
}
if let Some(policy) = policies.get(name).cloned() {
Cache::add_or_update(&self.cache.group_policies, name, &policy, OffsetDateTime::now_utc());
Some(policy)
} else {
None
}
};
Ok(build_group_desc(name, gi, mapped_policy))
}
pub async fn list_groups(&self) -> Result<Vec<String>> {
@@ -1882,6 +1895,20 @@ fn filter_policies(cache: &Cache, policy_name: &str, bucket_name: &str) -> (Stri
(policies.join(","), Policy::merge_policies(to_merge))
}
fn build_group_desc(name: &str, group_info: GroupInfo, mapped_policy: Option<MappedPolicy>) -> GroupDesc {
let (policy, updated_at) = mapped_policy
.map(|policy| (policy.policies, Some(policy.update_at)))
.unwrap_or_else(|| (String::new(), Some(OffsetDateTime::now_utc())));
GroupDesc {
name: name.to_string(),
policy,
members: group_info.members,
updated_at,
status: group_info.status,
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -2226,4 +2253,22 @@ mod tests {
assert!(merged.statements.is_empty());
assert!(merged.is_empty());
}
#[test]
fn test_build_group_desc_preserves_policy_for_disabled_group() {
let group_info = GroupInfo {
status: STATUS_DISABLED.to_string(),
members: vec!["alice".to_string()],
..Default::default()
};
let mapped_policy = MappedPolicy::new("readonly");
let desc = build_group_desc("ops", group_info, Some(mapped_policy));
assert_eq!(desc.name, "ops");
assert_eq!(desc.status, STATUS_DISABLED);
assert_eq!(desc.policy, "readonly");
assert_eq!(desc.members, vec!["alice".to_string()]);
assert!(desc.updated_at.is_some());
}
}
+54
View File
@@ -113,7 +113,11 @@ impl UserType {
#[derive(Serialize, Deserialize, Clone)]
pub struct MappedPolicy {
pub version: i64,
/// policy, legacy: policies. Serialize as policy.
#[serde(rename = "policy", alias = "policies")]
pub policies: String,
/// updatedAt (RFC3339), legacy: update_at. Serialize as updatedAt.
#[serde(rename = "updatedAt", alias = "update_at", with = "rustfs_policy::serde_datetime")]
pub update_at: OffsetDateTime,
}
@@ -158,6 +162,13 @@ pub struct GroupInfo {
pub version: i64,
pub status: String,
pub members: Vec<String>,
/// updatedAt (RFC3339), legacy: update_at. Serialize as updatedAt.
#[serde(
rename = "updatedAt",
alias = "update_at",
default,
with = "rustfs_policy::serde_datetime::option"
)]
pub update_at: Option<OffsetDateTime>,
}
@@ -171,3 +182,46 @@ impl GroupInfo {
}
}
}
#[cfg(test)]
mod tests {
use super::{GroupInfo, MappedPolicy};
/// uses RFC3339 for updatedAt. MappedPolicy must serialize as RFC3339.
#[test]
fn test_mapped_policy_timestamps_serialize_as_rfc3339() {
let mp = MappedPolicy::new("readwrite");
let json = serde_json::to_string(&mp).expect("serialize");
assert!(json.contains('T'), "MappedPolicy updatedAt should be RFC3339; got: {}", json);
assert!(
json.contains('Z') || json.contains("+00:00"),
"MappedPolicy updatedAt should be RFC3339; got: {}",
json
);
}
/// Deserialize MappedPolicy from JSON (RFC3339 updatedAt).
#[test]
fn test_mapped_policy_deserialize_minio_style_rfc3339() {
let minio_style = r#"{"version":1,"policy":"readwrite","updatedAt":"2025-03-07T12:00:00Z"}"#;
let mp: MappedPolicy = serde_json::from_str(minio_style).expect("deserialize");
assert_eq!(mp.policies, "readwrite");
}
/// GroupInfo updatedAt: uses RFC3339.
#[test]
fn test_group_info_timestamps_serialize_as_rfc3339() {
let g = GroupInfo::new(vec!["u1".to_string()]);
let json = serde_json::to_string(&g).expect("serialize");
assert!(json.contains('T'), "GroupInfo updatedAt should be RFC3339; got: {}", json);
}
/// Deserialize GroupInfo from JSON (RFC3339 updatedAt).
#[test]
fn test_group_info_deserialize_minio_style_rfc3339() {
let minio_style = r#"{"version":1,"status":"enabled","members":["u1"],"updatedAt":"2025-03-07T12:00:00Z"}"#;
let g: GroupInfo = serde_json::from_str(minio_style).expect("deserialize");
assert_eq!(g.members, ["u1"]);
assert!(g.update_at.is_some());
}
}
+123 -3
View File
@@ -129,15 +129,55 @@ impl ObjectStore {
}
fn decrypt_data(data: &[u8]) -> Result<Vec<u8>> {
let de = rustfs_crypto::decrypt_data(get_global_action_cred().unwrap_or_default().secret_key.as_bytes(), data)?;
Ok(de)
if Self::is_plaintext_json(data) {
return Ok(data.to_vec());
}
let cred = get_global_action_cred().unwrap_or_default();
let secret_key = cred.secret_key;
let mut keys: Vec<(Vec<u8>, bool)> = vec![(secret_key.clone().into_bytes(), false)];
if !cred.access_key.is_empty() && !secret_key.is_empty() {
keys.push((format!("{}:{secret_key}", cred.access_key).into_bytes(), true));
}
const STREAM_IO_HEADER_LEN: usize = 41;
let mut last_err = None;
for (key, is_access_secret) in keys {
if is_access_secret
&& data.len() >= STREAM_IO_HEADER_LEN
&& let Ok(plain) = rustfs_crypto::decrypt_stream_io(&key, data)
{
return Ok(plain);
}
match rustfs_crypto::decrypt_data(&key, data) {
Ok(plain) => return Ok(plain),
Err(err) => last_err = Some(err),
}
}
Err(last_err.unwrap_or(rustfs_crypto::Error::ErrUnexpectedHeader).into())
}
fn encrypt_data(data: &[u8]) -> Result<Vec<u8>> {
let en = rustfs_crypto::encrypt_data(get_global_action_cred().unwrap_or_default().secret_key.as_bytes(), data)?;
let cred = get_global_action_cred().unwrap_or_default();
let password = if !cred.access_key.is_empty() && !cred.secret_key.is_empty() {
format!("{}:{}", cred.access_key, cred.secret_key).into_bytes()
} else {
cred.secret_key.clone().into_bytes()
};
let en = rustfs_crypto::encrypt_stream_io(&password, data)?;
Ok(en)
}
fn is_plaintext_json(data: &[u8]) -> bool {
std::str::from_utf8(data).is_ok() && serde_json::from_slice::<serde_json::Value>(data).is_ok()
}
#[cfg(test)]
fn encrypt_data_for_test(data: &[u8]) -> Result<Vec<u8>> {
Self::encrypt_data(data)
}
async fn load_iamconfig_bytes_with_metadata(&self, path: impl AsRef<str> + Send) -> Result<(Vec<u8>, ObjectInfo)> {
let (data, obj) = read_config_with_metadata(self.object_api.clone(), path.as_ref(), &ObjectOptions::default()).await?;
@@ -1164,3 +1204,83 @@ impl Store for ObjectStore {
// Ok(())
// }
}
#[cfg(test)]
mod tests {
use super::ObjectStore;
use rustfs_credentials::{Credentials, get_global_action_cred, init_global_action_credentials};
fn test_cred() -> Credentials {
if let Some(cred) = get_global_action_cred() {
return cred;
}
let _ = init_global_action_credentials(Some("COMPATTESTAK".to_string()), Some("COMPATTESTSK1234567890".to_string()));
get_global_action_cred().unwrap_or_default()
}
#[test]
fn test_decrypt_data_accepts_plaintext_json() {
let raw = br#"{"Version":1,"policy":"readonly"}"#;
let out = ObjectStore::decrypt_data(raw).expect("plaintext json should pass through");
assert_eq!(out, raw);
}
#[test]
fn test_decrypt_data_accepts_rustfs_legacy_secret_encryption() {
let cred = test_cred();
let plain = br#"{"accessKey":"ak","secretKey":"sk"}"#;
let encrypted = rustfs_crypto::encrypt_data(cred.secret_key.as_bytes(), plain).expect("encrypt with rustfs secret");
let out = ObjectStore::decrypt_data(&encrypted).expect("decrypt rustfs legacy encryption");
assert_eq!(out, plain);
}
#[test]
fn test_decrypt_data_accepts_access_secret_encryption() {
let cred = test_cred();
let plain = br#"{"Version":1,"updatedAt":"2025-03-07T12:00:00Z"}"#;
let root_cred = format!("{}:{}", cred.access_key, cred.secret_key);
let encrypted = rustfs_crypto::encrypt_stream_io(root_cred.as_bytes(), plain).expect("encrypt with stream_io");
let out = ObjectStore::decrypt_data(&encrypted).expect("decrypt stream_io");
assert_eq!(out, plain);
}
#[test]
fn test_decrypt_data_corrupt_stream_io_fails() {
let cred = test_cred();
let plain = br#"{"Version":1}"#;
let root_cred = format!("{}:{}", cred.access_key, cred.secret_key);
let mut encrypted = rustfs_crypto::encrypt_stream_io(root_cred.as_bytes(), plain).expect("encrypt with stream_io");
if encrypted.len() > 50 {
encrypted[50] ^= 0xFF; // corrupt one byte
}
let result = ObjectStore::decrypt_data(&encrypted);
assert!(result.is_err(), "corrupt stream_io data should fail decrypt");
}
#[test]
fn test_decrypt_data_short_data_fails() {
let short = &[0x00u8; 40]; // less than 41-byte stream_io header, not valid JSON
let result = ObjectStore::decrypt_data(short);
assert!(result.is_err(), "short non-JSON data should fail decrypt");
}
#[test]
fn test_encrypt_data_produces_stream_io_format() {
let _ = test_cred();
let plain = br#"{"Version":1,"policy":"readonly"}"#;
let encrypted = ObjectStore::encrypt_data_for_test(plain).expect("encrypt should succeed");
// stream_io header: salt(32) + alg_id(1) + nonce_prefix(8) = 41 bytes
const STREAM_IO_HEADER_LEN: usize = 41;
assert!(
encrypted.len() >= STREAM_IO_HEADER_LEN,
"encrypted should have at least 41-byte stream_io header"
);
assert!(
encrypted[32] == 0x00 || encrypted[32] == 0x01 || encrypted[32] == 0x02,
"alg_id should be 0x00, 0x01, or 0x02"
);
// Round-trip: encrypt then decrypt
let decrypted = ObjectStore::decrypt_data(&encrypted).expect("decrypt should succeed");
assert_eq!(plain, decrypted.as_slice());
}
}
+1 -4
View File
@@ -361,10 +361,6 @@ impl<T: Store> IamSys<T> {
return Err(IamError::IAMActionNotAllowed);
}
if opts.expiration.is_none() {
return Err(IamError::InvalidExpiration);
}
// TODO: check allow_site_replicator_account
let policy_buf = if let Some(policy) = opts.session_policy {
@@ -619,6 +615,7 @@ impl<T: Store> IamSys<T> {
}
let updated_at = self.store.add_user(access_key, args).await?;
self.load_user(access_key, UserType::Reg).await?;
self.notify_for_user(access_key, false).await;
+2
View File
@@ -36,6 +36,7 @@ time = { workspace = true }
moka = { workspace = true }
rustfs-credentials = { workspace = true }
rustfs-policy = { workspace = true }
rustfs-utils = { workspace = true }
# Middleware dependencies
tower = { workspace = true }
http = { workspace = true }
@@ -50,6 +51,7 @@ tokio = { workspace = true, features = ["test-util"] }
tower = { workspace = true, features = ["util"] }
hyper = { workspace = true, features = ["server"] }
serde_json = { workspace = true }
temp-env = { workspace = true }
[[test]]
name = "integration"
+57 -39
View File
@@ -13,6 +13,7 @@
// limitations under the License.
use crate::{KeystoneError, KeystoneVersion, Result};
use rustfs_utils::{get_env_bool, get_env_opt_str, get_env_str, get_env_u64};
use serde::{Deserialize, Serialize};
use std::time::Duration;
@@ -80,59 +81,35 @@ pub struct RoleMapping {
impl KeystoneConfig {
/// Load configuration from environment variables
pub fn from_env() -> Result<Self> {
let enable = std::env::var("RUSTFS_KEYSTONE_ENABLE")
.unwrap_or_else(|_| "false".to_string())
.parse()
.unwrap_or(false);
let enable = get_env_bool("RUSTFS_KEYSTONE_ENABLE", false);
if !enable {
return Ok(Self::default());
}
let auth_url = std::env::var("RUSTFS_KEYSTONE_AUTH_URL")
.map_err(|_| KeystoneError::ConfigError("RUSTFS_KEYSTONE_AUTH_URL not set".to_string()))?;
let auth_url = get_env_opt_str("RUSTFS_KEYSTONE_AUTH_URL")
.ok_or_else(|| KeystoneError::ConfigError("RUSTFS_KEYSTONE_AUTH_URL not set".to_string()))?;
let version = std::env::var("RUSTFS_KEYSTONE_VERSION").unwrap_or_else(|_| "v3".to_string());
let version = get_env_str("RUSTFS_KEYSTONE_VERSION", "v3");
let admin_user = std::env::var("RUSTFS_KEYSTONE_ADMIN_USER").ok();
let admin_password = std::env::var("RUSTFS_KEYSTONE_ADMIN_PASSWORD").ok();
let admin_project = std::env::var("RUSTFS_KEYSTONE_ADMIN_PROJECT").ok();
let admin_domain = std::env::var("RUSTFS_KEYSTONE_ADMIN_DOMAIN").ok();
let admin_user = get_env_opt_str("RUSTFS_KEYSTONE_ADMIN_USER");
let admin_password = get_env_opt_str("RUSTFS_KEYSTONE_ADMIN_PASSWORD");
let admin_project = get_env_opt_str("RUSTFS_KEYSTONE_ADMIN_PROJECT");
let admin_domain = get_env_opt_str("RUSTFS_KEYSTONE_ADMIN_DOMAIN");
let verify_ssl = std::env::var("RUSTFS_KEYSTONE_VERIFY_SSL")
.unwrap_or_else(|_| "true".to_string())
.parse()
.unwrap_or(true);
let verify_ssl = get_env_bool("RUSTFS_KEYSTONE_VERIFY_SSL", true);
let enable_cache = std::env::var("RUSTFS_KEYSTONE_ENABLE_CACHE")
.unwrap_or_else(|_| "true".to_string())
.parse()
.unwrap_or(true);
let enable_cache = get_env_bool("RUSTFS_KEYSTONE_ENABLE_CACHE", true);
let cache_size = std::env::var("RUSTFS_KEYSTONE_CACHE_SIZE")
.unwrap_or_else(|_| "10000".to_string())
.parse()
.unwrap_or(10000);
let cache_size = get_env_u64("RUSTFS_KEYSTONE_CACHE_SIZE", 10000);
let cache_ttl_seconds = std::env::var("RUSTFS_KEYSTONE_CACHE_TTL")
.unwrap_or_else(|_| "300".to_string())
.parse()
.unwrap_or(300);
let cache_ttl_seconds = get_env_u64("RUSTFS_KEYSTONE_CACHE_TTL", 300);
let enable_tenant_prefix = std::env::var("RUSTFS_KEYSTONE_TENANT_PREFIX")
.unwrap_or_else(|_| "true".to_string())
.parse()
.unwrap_or(true);
let enable_tenant_prefix = get_env_bool("RUSTFS_KEYSTONE_TENANT_PREFIX", true);
let implicit_tenants = std::env::var("RUSTFS_KEYSTONE_IMPLICIT_TENANTS")
.unwrap_or_else(|_| "true".to_string())
.parse()
.unwrap_or(true);
let implicit_tenants = get_env_bool("RUSTFS_KEYSTONE_IMPLICIT_TENANTS", true);
let timeout_seconds = std::env::var("RUSTFS_KEYSTONE_TIMEOUT")
.unwrap_or_else(|_| "30".to_string())
.parse()
.unwrap_or(30);
let timeout_seconds = get_env_u64("RUSTFS_KEYSTONE_TIMEOUT", 30);
Ok(Self {
enable,
@@ -224,6 +201,7 @@ impl Default for KeystoneConfig {
#[cfg(test)]
mod tests {
use super::*;
use temp_env::with_vars;
#[test]
fn test_default_config() {
@@ -248,4 +226,44 @@ mod tests {
config.version = "invalid".to_string();
assert!(config.get_version().is_err());
}
#[test]
fn test_from_env_reads_configuration() {
with_vars(
vec![
("RUSTFS_KEYSTONE_ENABLE", Some("true")),
("RUSTFS_KEYSTONE_AUTH_URL", Some("https://keystone.example.com")),
("RUSTFS_KEYSTONE_VERSION", Some("v2.0")),
("RUSTFS_KEYSTONE_ADMIN_USER", Some("admin")),
("RUSTFS_KEYSTONE_ADMIN_PASSWORD", Some("secret")),
("RUSTFS_KEYSTONE_ADMIN_PROJECT", Some("service")),
("RUSTFS_KEYSTONE_ADMIN_DOMAIN", Some("Default")),
("RUSTFS_KEYSTONE_VERIFY_SSL", Some("false")),
("RUSTFS_KEYSTONE_ENABLE_CACHE", Some("false")),
("RUSTFS_KEYSTONE_CACHE_SIZE", Some("2048")),
("RUSTFS_KEYSTONE_CACHE_TTL", Some("900")),
("RUSTFS_KEYSTONE_TENANT_PREFIX", Some("false")),
("RUSTFS_KEYSTONE_IMPLICIT_TENANTS", Some("false")),
("RUSTFS_KEYSTONE_TIMEOUT", Some("99")),
],
|| {
let config = KeystoneConfig::from_env().expect("keystone config should load from env");
assert!(config.enable);
assert_eq!(config.auth_url, "https://keystone.example.com");
assert_eq!(config.version, "v2.0");
assert_eq!(config.admin_user.as_deref(), Some("admin"));
assert_eq!(config.admin_password.as_deref(), Some("secret"));
assert_eq!(config.admin_project.as_deref(), Some("service"));
assert_eq!(config.admin_domain.as_deref(), Some("Default"));
assert!(!config.verify_ssl);
assert!(!config.enable_cache);
assert_eq!(config.cache_size, 2048);
assert_eq!(config.cache_ttl_seconds, 900);
assert!(!config.enable_tenant_prefix);
assert!(!config.implicit_tenants);
assert_eq!(config.timeout_seconds, 99);
},
);
}
}
+2
View File
@@ -56,6 +56,7 @@ moka = { workspace = true, features = ["future"] }
# Additional dependencies
md5 = { workspace = true }
arc-swap = { workspace = true }
rustfs-utils = { workspace = true }
# HTTP client for Vault
reqwest = { workspace = true }
@@ -63,6 +64,7 @@ vaultrs = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
temp-env = { workspace = true }
[features]
default = []
+50 -16
View File
@@ -15,6 +15,7 @@
//! KMS configuration management
use crate::error::{KmsError, Result};
use rustfs_utils::{get_env_bool, get_env_opt_str, get_env_str};
use serde::{Deserialize, Serialize};
use std::path::PathBuf;
use std::time::Duration;
@@ -304,7 +305,7 @@ impl KmsConfig {
let mut config = Self::default();
// Backend type
if let Ok(backend_type) = std::env::var("RUSTFS_KMS_BACKEND") {
if let Some(backend_type) = get_env_opt_str("RUSTFS_KMS_BACKEND") {
config.backend = match backend_type.to_lowercase().as_str() {
"local" => KmsBackend::Local,
"vault" => KmsBackend::Vault,
@@ -313,12 +314,12 @@ impl KmsConfig {
}
// Default key ID
if let Ok(key_id) = std::env::var("RUSTFS_KMS_DEFAULT_KEY_ID") {
if let Some(key_id) = get_env_opt_str("RUSTFS_KMS_DEFAULT_KEY_ID") {
config.default_key_id = Some(key_id);
}
// Timeout
if let Ok(timeout_str) = std::env::var("RUSTFS_KMS_TIMEOUT_SECS") {
if let Some(timeout_str) = get_env_opt_str("RUSTFS_KMS_TIMEOUT_SECS") {
let timeout_secs = timeout_str
.parse::<u64>()
.map_err(|_| KmsError::configuration_error("Invalid timeout value"))?;
@@ -326,22 +327,20 @@ impl KmsConfig {
}
// Retry attempts
if let Ok(retries_str) = std::env::var("RUSTFS_KMS_RETRY_ATTEMPTS") {
if let Some(retries_str) = get_env_opt_str("RUSTFS_KMS_RETRY_ATTEMPTS") {
config.retry_attempts = retries_str
.parse()
.map_err(|_| KmsError::configuration_error("Invalid retry attempts value"))?;
}
// Enable cache
if let Ok(cache_str) = std::env::var("RUSTFS_KMS_ENABLE_CACHE") {
config.enable_cache = cache_str.parse().unwrap_or(true);
}
config.enable_cache = get_env_bool("RUSTFS_KMS_ENABLE_CACHE", config.enable_cache);
// Backend-specific configuration
match config.backend {
KmsBackend::Local => {
let key_dir = std::env::var("RUSTFS_KMS_LOCAL_KEY_DIR").unwrap_or_else(|_| "./kms_keys".to_string());
let master_key = std::env::var("RUSTFS_KMS_LOCAL_MASTER_KEY").ok();
let key_dir = get_env_str("RUSTFS_KMS_LOCAL_KEY_DIR", "./kms_keys");
let master_key = get_env_opt_str("RUSTFS_KMS_LOCAL_MASTER_KEY");
config.backend_config = BackendConfig::Local(LocalConfig {
key_dir: PathBuf::from(key_dir),
@@ -350,17 +349,16 @@ impl KmsConfig {
});
}
KmsBackend::Vault => {
let address = std::env::var("RUSTFS_KMS_VAULT_ADDRESS").unwrap_or_else(|_| "http://localhost:8200".to_string());
let token = std::env::var("RUSTFS_KMS_VAULT_TOKEN").unwrap_or_else(|_| "dev-token".to_string());
let address = get_env_str("RUSTFS_KMS_VAULT_ADDRESS", "http://localhost:8200");
let token = get_env_str("RUSTFS_KMS_VAULT_TOKEN", "dev-token");
config.backend_config = BackendConfig::Vault(Box::new(VaultConfig {
address,
auth_method: VaultAuthMethod::Token { token },
namespace: std::env::var("RUSTFS_KMS_VAULT_NAMESPACE").ok(),
mount_path: std::env::var("RUSTFS_KMS_VAULT_MOUNT_PATH").unwrap_or_else(|_| "transit".to_string()),
kv_mount: std::env::var("RUSTFS_KMS_VAULT_KV_MOUNT").unwrap_or_else(|_| "secret".to_string()),
key_path_prefix: std::env::var("RUSTFS_KMS_VAULT_KEY_PREFIX")
.unwrap_or_else(|_| "rustfs/kms/keys".to_string()),
namespace: get_env_opt_str("RUSTFS_KMS_VAULT_NAMESPACE"),
mount_path: get_env_str("RUSTFS_KMS_VAULT_MOUNT_PATH", "transit"),
kv_mount: get_env_str("RUSTFS_KMS_VAULT_KV_MOUNT", "secret"),
key_path_prefix: get_env_str("RUSTFS_KMS_VAULT_KEY_PREFIX", "rustfs/kms/keys"),
tls: None,
}));
}
@@ -374,6 +372,7 @@ impl KmsConfig {
#[cfg(test)]
mod tests {
use super::*;
use temp_env::with_vars;
use tempfile::TempDir;
#[test]
@@ -423,4 +422,39 @@ mod tests {
config.retry_attempts = 0;
assert!(config.validate().is_err());
}
#[test]
fn test_from_env_reads_vault_settings() {
with_vars(
vec![
("RUSTFS_KMS_BACKEND", Some("vault")),
("RUSTFS_KMS_DEFAULT_KEY_ID", Some("tenant-key")),
("RUSTFS_KMS_TIMEOUT_SECS", Some("42")),
("RUSTFS_KMS_RETRY_ATTEMPTS", Some("7")),
("RUSTFS_KMS_ENABLE_CACHE", Some("false")),
("RUSTFS_KMS_VAULT_ADDRESS", Some("https://vault.example.com")),
("RUSTFS_KMS_VAULT_TOKEN", Some("vault-token")),
("RUSTFS_KMS_VAULT_NAMESPACE", Some("tenant-a")),
("RUSTFS_KMS_VAULT_MOUNT_PATH", Some("transit-alt")),
("RUSTFS_KMS_VAULT_KV_MOUNT", Some("secret-alt")),
("RUSTFS_KMS_VAULT_KEY_PREFIX", Some("tenant/keys")),
],
|| {
let config = KmsConfig::from_env().expect("kms config should load from env");
assert_eq!(config.backend, KmsBackend::Vault);
assert_eq!(config.default_key_id.as_deref(), Some("tenant-key"));
assert_eq!(config.timeout, Duration::from_secs(42));
assert_eq!(config.retry_attempts, 7);
assert!(!config.enable_cache);
let vault = config.vault_config().expect("vault backend config");
assert_eq!(vault.address, "https://vault.example.com");
assert_eq!(vault.namespace.as_deref(), Some("tenant-a"));
assert_eq!(vault.mount_path, "transit-alt");
assert_eq!(vault.kv_mount, "secret-alt");
assert_eq!(vault.key_path_prefix, "tenant/keys");
},
);
}
}
+51 -2
View File
@@ -13,10 +13,11 @@
// limitations under the License.
use serde::Deserialize;
use serde::Deserializer;
use serde::Serialize;
use time::OffsetDateTime;
#[derive(Debug, Serialize, Deserialize, Default)]
#[derive(Debug, Serialize, Default, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum GroupStatus {
#[default]
@@ -24,6 +25,20 @@ pub enum GroupStatus {
Disabled,
}
impl<'de> Deserialize<'de> for GroupStatus {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
let value = String::deserialize(deserializer)?;
match value.as_str() {
"" | "enabled" => Ok(Self::Enabled),
"disabled" => Ok(Self::Disabled),
_ => Err(serde::de::Error::unknown_variant(&value, &["enabled", "disabled"])),
}
}
}
#[derive(Debug, Serialize, Deserialize, Default)]
pub struct GroupAddRemove {
pub group: String,
@@ -40,6 +55,40 @@ pub struct GroupDesc {
pub status: String,
pub members: Vec<String>,
pub policy: String,
#[serde(rename = "updatedAt", skip_serializing_if = "Option::is_none")]
#[serde(
rename = "updatedAt",
skip_serializing_if = "Option::is_none",
with = "time::serde::rfc3339::option"
)]
pub updated_at: Option<OffsetDateTime>,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn group_desc_updated_at_serializes_as_rfc3339() {
let now = OffsetDateTime::now_utc().replace_nanosecond(0).unwrap();
let group = GroupDesc {
name: "group-a".to_string(),
status: "enabled".to_string(),
members: vec!["user-a".to_string()],
policy: "readwrite".to_string(),
updated_at: Some(now),
};
let json = serde_json::to_string(&group).unwrap();
let decoded: GroupDesc = serde_json::from_str(&json).unwrap();
assert!(json.contains("\"updatedAt\":\""));
assert!(json.contains('T'));
assert_eq!(decoded.updated_at, Some(now));
}
#[test]
fn group_status_accepts_empty_string_as_enabled() {
let status: GroupStatus = serde_json::from_str(r#""""#).unwrap();
assert_eq!(status, GroupStatus::Enabled);
}
}
+325 -71
View File
@@ -13,6 +13,7 @@
// limitations under the License.
use serde::{Deserialize, Deserializer, Serialize, Serializer};
use serde_json::Value;
use serde_json::value::RawValue;
use std::collections::HashMap;
use time::OffsetDateTime;
@@ -88,7 +89,7 @@ pub struct UserInfo {
#[serde(rename = "memberOf", skip_serializing_if = "Option::is_none")]
pub member_of: Option<Vec<String>>,
#[serde(rename = "updatedAt")]
#[serde(rename = "updatedAt", with = "time::serde::rfc3339::option")]
pub updated_at: Option<OffsetDateTime>,
}
@@ -134,47 +135,61 @@ pub struct ListServiceAccountsResp {
pub accounts: Vec<ServiceAccountInfo>,
}
#[derive(Debug, Serialize, Deserialize, Default)]
pub struct ListAccessKeysResp {
#[serde(rename = "serviceAccounts", default)]
pub service_accounts: Vec<ServiceAccountInfo>,
#[serde(rename = "stsKeys", default)]
pub sts_keys: Vec<ServiceAccountInfo>,
}
pub const ACCESS_KEY_LIST_USERS_ONLY: &str = "users-only";
pub const ACCESS_KEY_LIST_STS_ONLY: &str = "sts-only";
pub const ACCESS_KEY_LIST_SVCACC_ONLY: &str = "svcacc-only";
pub const ACCESS_KEY_LIST_ALL: &str = "all";
#[derive(Debug, Serialize, Deserialize)]
pub struct AddServiceAccountReq {
#[serde(rename = "policy", skip_serializing_if = "Option::is_none")]
pub policy: Option<String>,
#[serde(
rename = "policy",
skip_serializing_if = "Option::is_none",
default,
deserialize_with = "deserialize_optional_policy_value"
)]
pub policy: Option<Value>,
#[serde(rename = "targetUser", skip_serializing_if = "Option::is_none")]
pub target_user: Option<String>,
#[serde(rename = "accessKey")]
#[serde(rename = "accessKey", default)]
pub access_key: String,
#[serde(rename = "secretKey")]
#[serde(rename = "secretKey", default)]
pub secret_key: String,
#[serde(rename = "name")]
#[serde(rename = "name", skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
#[serde(rename = "description", skip_serializing_if = "Option::is_none")]
pub description: Option<String>,
#[serde(rename = "expiration", with = "time::serde::rfc3339::option")]
#[serde(
rename = "expiration",
skip_serializing_if = "Option::is_none",
default,
with = "time::serde::rfc3339::option"
)]
pub expiration: Option<OffsetDateTime>,
#[serde(rename = "comment", skip_serializing_if = "Option::is_none")]
pub comment: Option<String>,
}
impl AddServiceAccountReq {
pub fn validate(&self) -> Result<(), String> {
if self.access_key.is_empty() {
return Err("accessKey is empty".to_string());
}
if self.secret_key.is_empty() {
return Err("secretKey is empty".to_string());
}
if self.name.is_none() {
return Err("name is empty".to_string());
}
// TODO: validate
Ok(())
validate_service_account_name(self.name.as_deref())?;
validate_service_account_description(self.description.as_deref().or(self.comment.as_deref()))?;
validate_service_account_expiration(self.expiration)
}
}
@@ -195,7 +210,7 @@ pub struct AddServiceAccountResp<'a> {
pub credentials: Credentials<'a>,
}
#[derive(Serialize)]
#[derive(Debug, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
pub struct InfoServiceAccountResp {
pub parent_user: String,
@@ -213,10 +228,70 @@ pub struct InfoServiceAccountResp {
pub expiration: Option<OffsetDateTime>,
}
pub type TemporaryAccountInfoResp = InfoServiceAccountResp;
#[derive(Debug, Serialize, Deserialize, Default, PartialEq, Eq)]
pub struct LDAPSpecificAccessKeyInfo {
#[serde(rename = "username", skip_serializing_if = "Option::is_none")]
pub username: Option<String>,
}
impl LDAPSpecificAccessKeyInfo {
pub fn is_empty(&self) -> bool {
self.username.is_none()
}
}
#[derive(Debug, Serialize, Deserialize, Default, PartialEq, Eq)]
pub struct OpenIDSpecificAccessKeyInfo {
#[serde(rename = "configName", skip_serializing_if = "Option::is_none")]
pub config_name: Option<String>,
#[serde(rename = "userID", skip_serializing_if = "Option::is_none")]
pub user_id: Option<String>,
#[serde(rename = "userIDClaim", skip_serializing_if = "Option::is_none")]
pub user_id_claim: Option<String>,
#[serde(rename = "displayName", skip_serializing_if = "Option::is_none")]
pub display_name: Option<String>,
#[serde(rename = "displayNameClaim", skip_serializing_if = "Option::is_none")]
pub display_name_claim: Option<String>,
}
impl OpenIDSpecificAccessKeyInfo {
pub fn is_empty(&self) -> bool {
self.config_name.is_none()
&& self.user_id.is_none()
&& self.user_id_claim.is_none()
&& self.display_name.is_none()
&& self.display_name_claim.is_none()
}
}
#[derive(Debug, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "camelCase")]
pub struct InfoAccessKeyResp {
pub access_key: String,
#[serde(flatten)]
pub info: InfoServiceAccountResp,
pub user_type: String,
pub user_provider: String,
#[serde(rename = "ldapSpecificInfo", skip_serializing_if = "LDAPSpecificAccessKeyInfo::is_empty")]
pub ldap_specific_info: LDAPSpecificAccessKeyInfo,
#[serde(
rename = "openIDSpecificInfo",
skip_serializing_if = "OpenIDSpecificAccessKeyInfo::is_empty"
)]
pub open_id_specific_info: OpenIDSpecificAccessKeyInfo,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct UpdateServiceAccountReq {
#[serde(rename = "newPolicy", skip_serializing_if = "Option::is_none")]
pub new_policy: Option<String>,
#[serde(
rename = "newPolicy",
skip_serializing_if = "Option::is_none",
default,
deserialize_with = "deserialize_optional_policy_value"
)]
pub new_policy: Option<Value>,
#[serde(rename = "newSecretKey", skip_serializing_if = "Option::is_none")]
pub new_secret_key: Option<String>,
@@ -230,18 +305,95 @@ pub struct UpdateServiceAccountReq {
#[serde(rename = "newDescription", skip_serializing_if = "Option::is_none")]
pub new_description: Option<String>,
#[serde(rename = "newExpiration", skip_serializing_if = "Option::is_none")]
#[serde(rename = "newExpiration", skip_serializing_if = "Option::is_none", default)]
#[serde(with = "time::serde::rfc3339::option")]
pub new_expiration: Option<OffsetDateTime>,
}
impl UpdateServiceAccountReq {
pub fn validate(&self) -> Result<(), String> {
// TODO: validate
Ok(())
validate_service_account_name(self.new_name.as_deref())?;
validate_service_account_description(self.new_description.as_deref())?;
validate_service_account_expiration(self.new_expiration)
}
}
fn deserialize_optional_policy_value<'de, D>(deserializer: D) -> Result<Option<Value>, D::Error>
where
D: Deserializer<'de>,
{
let value = Option::<Value>::deserialize(deserializer)?;
Ok(value.map(normalize_policy_value))
}
fn normalize_policy_value(value: Value) -> Value {
match value {
Value::String(policy) => serde_json::from_str(&policy).unwrap_or(Value::String(policy)),
other => other,
}
}
fn validate_service_account_name(name: Option<&str>) -> Result<(), String> {
let Some(name) = name else {
return Ok(());
};
if name.is_empty() {
return Ok(());
}
if name.len() > 32 {
return Err("name must not be longer than 32 characters".to_string());
}
let mut chars = name.chars();
let Some(first) = chars.next() else {
return Ok(());
};
if !first.is_ascii_alphabetic() {
return Err(
"name must contain only ASCII letters, digits, underscores and hyphens and must start with a letter".to_string(),
);
}
if chars.any(|c| !c.is_ascii_alphanumeric() && c != '_' && c != '-') {
return Err(
"name must contain only ASCII letters, digits, underscores and hyphens and must start with a letter".to_string(),
);
}
Ok(())
}
fn validate_service_account_description(description: Option<&str>) -> Result<(), String> {
let Some(description) = description else {
return Ok(());
};
if description.len() > 256 {
return Err("description must be at most 256 bytes long".to_string());
}
Ok(())
}
fn validate_service_account_expiration(expiration: Option<OffsetDateTime>) -> Result<(), String> {
let Some(expiration) = expiration else {
return Ok(());
};
if expiration.unix_timestamp() == 0 {
return Ok(());
}
if expiration < OffsetDateTime::now_utc() {
return Err("the expiration time should be in the future".to_string());
}
Ok(())
}
#[derive(Debug, Serialize, Deserialize, Default)]
pub struct AccountInfo {
pub account_name: String,
@@ -423,6 +575,54 @@ pub struct IAMEntities {
pub sts_policies: Vec<HashMap<String, Vec<String>>>,
}
/// PolicyEntitiesResult - contains response to a policy entities query.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PolicyEntitiesResult {
#[serde(rename = "timestamp", with = "time::serde::rfc3339")]
pub timestamp: time::OffsetDateTime,
#[serde(rename = "userMappings", skip_serializing_if = "Vec::is_empty")]
pub user_mappings: Vec<UserPolicyEntities>,
#[serde(rename = "groupMappings", skip_serializing_if = "Vec::is_empty")]
pub group_mappings: Vec<GroupPolicyEntities>,
#[serde(rename = "policyMappings", skip_serializing_if = "Vec::is_empty")]
pub policy_mappings: Vec<PolicyEntities>,
}
impl Default for PolicyEntitiesResult {
fn default() -> Self {
Self {
timestamp: time::OffsetDateTime::UNIX_EPOCH,
user_mappings: Vec::new(),
group_mappings: Vec::new(),
policy_mappings: Vec::new(),
}
}
}
/// UserPolicyEntities - user -> policies mapping
#[derive(Default, Debug, Clone, Serialize, Deserialize)]
pub struct UserPolicyEntities {
pub user: String,
pub policies: Vec<String>,
#[serde(rename = "memberOfMappings", skip_serializing_if = "Vec::is_empty")]
pub member_of_mappings: Vec<GroupPolicyEntities>,
}
/// GroupPolicyEntities - group -> policies mapping
#[derive(Default, Debug, Clone, Serialize, Deserialize)]
pub struct GroupPolicyEntities {
pub group: String,
pub policies: Vec<String>,
}
/// PolicyEntities - policy -> user+group mapping
#[derive(Default, Debug, Clone, Serialize, Deserialize)]
pub struct PolicyEntities {
pub policy: String,
pub users: Vec<String>,
pub groups: Vec<String>,
}
/// IAMErrEntities - represents errored out IAM entries while import with error
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct IAMErrEntities {
@@ -659,13 +859,14 @@ mod tests {
#[test]
fn test_add_service_account_req_validate_success() {
let req = AddServiceAccountReq {
policy: Some("ReadOnlyAccess".to_string()),
policy: Some(serde_json::json!({"Version": "2012-10-17"})),
target_user: Some("testuser".to_string()),
access_key: "AKIAIOSFODNN7EXAMPLE".to_string(),
secret_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY".to_string(),
name: Some("test-service".to_string()),
description: Some("Test service account".to_string()),
expiration: None,
comment: None,
};
let result = req.validate();
@@ -673,54 +874,82 @@ mod tests {
}
#[test]
fn test_add_service_account_req_validate_empty_access_key() {
fn test_add_service_account_req_validate_allows_generated_credentials() {
let req = AddServiceAccountReq {
policy: None,
target_user: None,
access_key: "".to_string(),
secret_key: "secret".to_string(),
name: Some("test".to_string()),
description: None,
expiration: None,
};
let result = req.validate();
assert!(result.is_err());
assert!(result.unwrap_err().contains("accessKey is empty"));
}
#[test]
fn test_add_service_account_req_validate_empty_secret_key() {
let req = AddServiceAccountReq {
policy: None,
target_user: None,
access_key: "AKIAIOSFODNN7EXAMPLE".to_string(),
secret_key: "".to_string(),
name: Some("test".to_string()),
description: None,
expiration: None,
};
let result = req.validate();
assert!(result.is_err());
assert!(result.unwrap_err().contains("secretKey is empty"));
}
#[test]
fn test_add_service_account_req_validate_empty_name() {
let req = AddServiceAccountReq {
policy: None,
target_user: None,
access_key: "AKIAIOSFODNN7EXAMPLE".to_string(),
secret_key: "secret".to_string(),
name: None,
description: None,
expiration: None,
comment: None,
};
assert!(req.validate().is_ok());
}
#[test]
fn test_add_service_account_req_deserializes_stringified_policy_json() {
let req: AddServiceAccountReq = serde_json::from_str(
r#"{
"policy":"{\"Version\":\"2012-10-17\",\"Statement\":[]}",
"accessKey":"AKIAIOSFODNN7EXAMPLE",
"secretKey":"secret"
}"#,
)
.unwrap();
assert_eq!(req.policy, Some(serde_json::json!({"Version":"2012-10-17","Statement":[]})));
}
#[test]
fn test_add_service_account_req_allows_missing_policy_field() {
let req: AddServiceAccountReq = serde_json::from_str(
r#"{
"accessKey":"AKIAIOSFODNN7EXAMPLE",
"secretKey":"secret"
}"#,
)
.unwrap();
assert_eq!(req.policy, None);
}
#[test]
fn test_add_service_account_req_validate_invalid_name() {
let req = AddServiceAccountReq {
policy: None,
target_user: None,
access_key: "AKIAIOSFODNN7EXAMPLE".to_string(),
secret_key: "secret".to_string(),
name: Some("1invalid".to_string()),
description: None,
expiration: None,
comment: None,
};
let result = req.validate();
assert!(result.is_err());
assert!(result.unwrap_err().contains("name is empty"));
assert!(result.unwrap_err().contains("must start with a letter"));
}
#[test]
fn test_add_service_account_req_validate_rejects_long_description() {
let req = AddServiceAccountReq {
policy: None,
target_user: None,
access_key: "AKIAIOSFODNN7EXAMPLE".to_string(),
secret_key: "secret".to_string(),
name: Some("test".to_string()),
description: Some("a".repeat(257)),
expiration: None,
comment: None,
};
let result = req.validate();
assert!(result.is_err());
assert!(result.unwrap_err().contains("at most 256 bytes"));
}
#[test]
@@ -793,7 +1022,7 @@ mod tests {
#[test]
fn test_update_service_account_req_validate() {
let req = UpdateServiceAccountReq {
new_policy: Some("FullAccess".to_string()),
new_policy: Some(serde_json::json!({"Version": "2012-10-17"})),
new_secret_key: Some("newsecret".to_string()),
new_status: Some("enabled".to_string()),
new_name: Some("updated-service".to_string()),
@@ -805,6 +1034,25 @@ mod tests {
assert!(result.is_ok());
}
#[test]
fn test_update_service_account_req_deserializes_stringified_policy_json() {
let req: UpdateServiceAccountReq = serde_json::from_str(
r#"{
"newPolicy":"{\"Version\":\"2012-10-17\",\"Statement\":[]}"
}"#,
)
.unwrap();
assert_eq!(req.new_policy, Some(serde_json::json!({"Version":"2012-10-17","Statement":[]})));
}
#[test]
fn test_update_service_account_req_allows_missing_policy_field() {
let req: UpdateServiceAccountReq = serde_json::from_str(r#"{}"#).unwrap();
assert_eq!(req.new_policy, None);
}
#[test]
fn test_account_info_creation() {
use crate::BackendInfo;
@@ -904,6 +1152,7 @@ mod tests {
#[test]
fn test_serialization_deserialization_roundtrip() {
let now = OffsetDateTime::now_utc().replace_nanosecond(0).unwrap();
let user_info = UserInfo {
auth_info: Some(UserAuthInfo {
auth_type: UserAuthType::Ldap,
@@ -914,16 +1163,19 @@ mod tests {
policy_name: Some("ReadOnlyAccess".to_string()),
status: AccountStatus::Enabled,
member_of: Some(vec!["group1".to_string()]),
updated_at: None,
updated_at: Some(now),
};
let json = serde_json::to_string(&user_info).unwrap();
let deserialized: UserInfo = serde_json::from_str(&json).unwrap();
assert!(json.contains("\"updatedAt\":\""));
assert!(json.contains('T'));
assert_eq!(deserialized.secret_key.unwrap(), "secret123");
assert_eq!(deserialized.policy_name.unwrap(), "ReadOnlyAccess");
assert_eq!(deserialized.status, AccountStatus::Enabled);
assert_eq!(deserialized.member_of.unwrap().len(), 1);
assert_eq!(deserialized.updated_at, Some(now));
}
#[test]
@@ -962,13 +1214,14 @@ mod tests {
fn test_edge_cases() {
// Test empty strings and edge cases
let req = AddServiceAccountReq {
policy: Some("".to_string()),
policy: Some(serde_json::Value::Null),
target_user: Some("".to_string()),
access_key: "valid_key".to_string(),
secret_key: "valid_secret".to_string(),
name: Some("valid_name".to_string()),
description: Some("".to_string()),
expiration: None,
comment: None,
};
// Should still validate successfully with empty optional strings
@@ -977,13 +1230,14 @@ mod tests {
// Test very long strings
let long_string = "a".repeat(1000);
let long_req = AddServiceAccountReq {
policy: Some(long_string.clone()),
policy: Some(serde_json::json!({"Statement": [long_string.clone()]})),
target_user: Some(long_string.clone()),
access_key: long_string.clone(),
secret_key: long_string.clone(),
name: Some(long_string.clone()),
description: Some(long_string),
name: Some("valid_name".to_string()),
description: Some("valid description".to_string()),
expiration: None,
comment: None,
};
assert!(long_req.validate().is_ok());
+7 -1
View File
@@ -37,7 +37,6 @@ chrono = { workspace = true, features = ["serde"] }
futures = { workspace = true }
form_urlencoded = { workspace = true }
hashbrown = { workspace = true }
quick-xml = { workspace = true, features = ["serialize", "async-tokio"] }
rayon = { workspace = true }
rumqttc = { workspace = true }
rustc-hash = { workspace = true }
@@ -49,6 +48,13 @@ tracing = { workspace = true }
url = { workspace = true }
wildmatch = { workspace = true, features = ["serde"] }
# quick-xml dependencies for custom S3KeyFilter XML deserialization
# Custom deserializer implemented for S3KeyFilter to handle both XML structures:
# - Direct FilterRule elements under S3Key (AWS S3 format)
# - FilterRuleList wrapper format
# This makes quick-xml 0.39.2 work correctly with Filter elements
quick-xml = { workspace = true, features = ["serialize", "serde-types", "encoding"] }
[dev-dependencies]
tokio = { workspace = true, features = ["test-util"] }
tracing-subscriber = { workspace = true, features = ["env-filter"] }
+5 -9
View File
@@ -111,15 +111,11 @@ pub mod notifier_global {
suffix: &str,
target_ids: &[TargetID],
) -> Result<(), NotificationError> {
// Construct pattern, simple splicing of prefixes and suffixes
let mut pattern = String::new();
if !prefix.is_empty() {
pattern.push_str(prefix);
}
pattern.push('*');
if !suffix.is_empty() {
pattern.push_str(suffix);
}
// Construct pattern using proper pattern function
let pattern = crate::rules::pattern::new_pattern(
if prefix.is_empty() { None } else { Some(prefix) },
if suffix.is_empty() { None } else { Some(suffix) },
);
// Create BucketNotificationConfig
let mut bucket_config = BucketNotificationConfig::new(region);
+1 -1
View File
@@ -117,7 +117,7 @@ impl BucketNotificationConfig {
// The ARN in queue_conf should now have its region set if it was originally empty.
// Ensure TargetID can be cloned or extracted correctly.
let target_id = queue_conf.arn.target_id.clone();
let pattern_str = queue_conf.filter.filter_rule_list.pattern();
let pattern_str = queue_conf.filter.pattern();
rules_map.add_rule_config(&queue_conf.events, pattern_str, target_id);
}
+440
View File
@@ -0,0 +1,440 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Integration tests for BucketNotificationConfig
//!
//! This module contains tests that simulate the exact flow from XML configuration
//! to event matching, including filter rules with prefix and suffix.
use super::*;
use rustfs_s3_common::EventName;
use rustfs_targets::arn::{ARN, TargetID};
use std::io::Cursor;
#[cfg(test)]
mod integration_tests {
use super::*;
fn create_test_arn() -> ARN {
ARN {
partition: "rustfs".to_string(),
service: "sqs".to_string(),
region: "".to_string(),
target_id: TargetID::new("primary".to_string(), "webhook".to_string()),
}
}
#[test]
fn test_create_arn() {
let arn = create_test_arn();
assert_eq!(arn.partition, "rustfs");
assert_eq!(arn.service, "sqs");
assert_eq!(arn.region, "");
assert_eq!(arn.target_id, TargetID::new("primary".to_string(), "webhook".to_string()));
}
/// Test exact bug scenario from the bug report
#[test]
fn test_bug_report_exact_scenario_xml() {
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<NotificationConfiguration>
<QueueConfiguration>
<Id>test-queue</Id>
<Queue>arn:rustfs:sqs:ap-northeast-1:primary:webhook</Queue>
<Event>s3:ObjectCreated:*</Event>
<Filter>
<S3Key>
<FilterRule>
<Name>prefix</Name>
<Value>uploads/</Value>
</FilterRule>
<FilterRule>
<Name>suffix</Name>
<Value>.csv</Value>
</FilterRule>
</S3Key>
</Filter>
</QueueConfiguration>
</NotificationConfiguration>"#;
let current_region = "ap-northeast-1";
let arn_list = vec!["arn:rustfs:sqs:ap-northeast-1:primary:webhook".to_string()];
let config = BucketNotificationConfig::from_xml(Cursor::new(xml.as_bytes()), current_region, &arn_list).unwrap();
// Verify pattern is "uploads/*.csv"
let rules_map = config.get_rules_map();
// Debug: Print the patterns stored
println!("\nPatterns stored in rules_map:");
for (event_name, pattern_rules) in rules_map.inner().iter() {
for pattern in pattern_rules.inner().keys() {
println!(" Event: {:?}, Pattern: '{}'", event_name, pattern);
}
}
// The event should be registered for ObjectCreatedAll and all its expansions
assert!(rules_map.has_subscriber(&EventName::ObjectCreatedAll));
assert!(rules_map.has_subscriber(&EventName::ObjectCreatedPut));
// Test matching "uploads/test.csv" with ObjectCreatedPut
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/test.csv");
println!("Matching 'uploads/test.csv' with ObjectCreatedPut: {} targets", targets.len());
for target in &targets {
println!(" Target: {:?}", target);
}
// Should find the target
assert!(!targets.is_empty(), "Should find at least one target");
// Test matching "uploads/subdir/file.csv" - should also match
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/subdir/file.csv");
assert!(!targets.is_empty(), "Nested CSV files should match");
// Test matching "uploads/test.txt" - should NOT match
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/test.txt");
assert!(targets.is_empty(), "TXT files should not match");
// Test matching "files/test.csv" - should NOT match (wrong prefix)
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "files/test.csv");
assert!(targets.is_empty(), "Files in wrong prefix should not match");
}
/// Test prefix only filter
#[test]
fn test_prefix_only_filter_xml() {
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<NotificationConfiguration>
<QueueConfiguration>
<Id>test-queue</Id>
<Queue>arn:rustfs:sqs:ap-northeast-1:primary:webhook</Queue>
<Event>s3:ObjectCreated:*</Event>
<Filter>
<S3Key>
<FilterRuleList>
<FilterRule>
<Name>prefix</Name>
<Value>images/</Value>
</FilterRule>
</FilterRuleList>
</S3Key>
</Filter>
</QueueConfiguration>
</NotificationConfiguration>"#;
let current_region = "ap-northeast-1";
let arn_list = vec!["arn:rustfs:sqs:ap-northeast-1:primary:webhook".to_string()];
let config = BucketNotificationConfig::from_xml(Cursor::new(xml.as_bytes()), current_region, &arn_list).unwrap();
let rules_map = config.get_rules_map();
// Debug: Print the patterns stored
println!("\nPatterns stored in rules_map:");
for (event_name, pattern_rules) in rules_map.inner().iter() {
for pattern in pattern_rules.inner().keys() {
println!(" Event: {:?}, Pattern: '{}'", event_name, pattern);
}
}
// Test matching "images/photo.jpg"
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "images/photo.jpg");
println!("Matching 'images/photo.jpg': {} targets", targets.len());
assert!(!targets.is_empty(), "Files in images/ should match");
// Test matching "uploads/photo.jpg" - should NOT match
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/photo.jpg");
println!("Matching 'uploads/photo.jpg': {} targets", targets.len());
assert!(targets.is_empty(), "Files not in images/ should not match");
}
/// Test suffix only filter
#[test]
fn test_suffix_only_filter_xml() {
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<NotificationConfiguration>
<QueueConfiguration>
<Id>test-queue</Id>
<Queue>arn:rustfs:sqs:ap-northeast-1:primary:webhook</Queue>
<Event>s3:ObjectCreated:*</Event>
<Filter>
<S3Key>
<FilterRuleList>
<FilterRule>
<Name>suffix</Name>
<Value>.pdf</Value>
</FilterRule>
</FilterRuleList>
</S3Key>
</Filter>
</QueueConfiguration>
</NotificationConfiguration>"#;
let current_region = "ap-northeast-1";
let arn_list = vec!["arn:rustfs:sqs:ap-northeast-1:primary:webhook".to_string()];
let config = BucketNotificationConfig::from_xml(Cursor::new(xml.as_bytes()), current_region, &arn_list).unwrap();
let rules_map = config.get_rules_map();
// Test matching "document.pdf" - should match
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "document.pdf");
assert!(!targets.is_empty(), "PDF files should match");
// Test matching "document.txt" - should NOT match
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "document.txt");
assert!(targets.is_empty(), "Non-PDF files should not match");
}
/// Test no filter (match all)
#[test]
fn test_no_filter_xml() {
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<NotificationConfiguration>
<QueueConfiguration>
<Id>test-queue</Id>
<Queue>arn:rustfs:sqs:ap-northeast-1:primary:webhook</Queue>
<Event>s3:ObjectCreated:*</Event>
</QueueConfiguration>
</NotificationConfiguration>"#;
let current_region = "ap-northeast-1";
let arn_list = vec!["arn:rustfs:sqs:ap-northeast-1:primary:webhook".to_string()];
let config = BucketNotificationConfig::from_xml(Cursor::new(xml.as_bytes()), current_region, &arn_list).unwrap();
let rules_map = config.get_rules_map();
// All files should match
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "anything.csv");
assert!(!targets.is_empty(), "All files should match");
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/test.txt");
assert!(!targets.is_empty(), "All files should match");
}
/// Test specific event type instead of compound
#[test]
fn test_specific_event_type_xml() {
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<NotificationConfiguration>
<QueueConfiguration>
<Id>test-queue</Id>
<Queue>arn:rustfs:sqs:ap-northeast-1:primary:webhook</Queue>
<Event>s3:ObjectCreated:Put</Event>
<Filter>
<S3Key>
<FilterRule>
<Name>prefix</Name>
<Value>uploads/</Value>
</FilterRule>
<FilterRule>
<Name>suffix</Name>
<Value>.csv</Value>
</FilterRule>
</S3Key>
</Filter>
</QueueConfiguration>
</NotificationConfiguration>"#;
let current_region = "ap-northeast-1";
let arn_list = vec!["arn:rustfs:sqs:ap-northeast-1:primary:webhook".to_string()];
let config = BucketNotificationConfig::from_xml(Cursor::new(xml.as_bytes()), current_region, &arn_list).unwrap();
let rules_map = config.get_rules_map();
// Debug: Print all events that have subscribers
println!("\nAll events with subscribers:");
for (event_name, _pattern_rules) in rules_map.inner().iter() {
println!(" Event: {:?}", event_name);
}
// Note: has_subscriber uses bitmask logic, so has_subscriber(&ObjectCreatedAll) returns true
// if any ObjectCreated* event is registered. This is intentional for efficiency.
// Only ObjectCreatedPut was explicitly registered in the XML.
// Should have subscriber for ObjectCreatedPut
assert!(rules_map.has_subscriber(&EventName::ObjectCreatedPut));
// Should NOT have subscriber for ObjectCreatedPost (only Put was configured)
assert!(!rules_map.has_subscriber(&EventName::ObjectCreatedPost));
// Test matching with Put event
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/test.csv");
assert!(!targets.is_empty(), "Put event should match");
}
/// Test URL-encoded object keys
#[test]
fn test_url_encoded_keys() {
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<NotificationConfiguration>
<QueueConfiguration>
<Id>test-queue</Id>
<Queue>arn:rustfs:sqs:ap-northeast-1:primary:webhook</Queue>
<Event>s3:ObjectCreated:*</Event>
<Filter>
<S3Key>
<FilterRule>
<Name>prefix</Name>
<Value>uploads/</Value>
</FilterRule>
<FilterRule>
<Name>suffix</Name>
<Value>.csv</Value>
</FilterRule>
</S3Key>
</Filter>
</QueueConfiguration>
</NotificationConfiguration>"#;
let current_region = "ap-northeast-1";
let arn_list = vec!["arn:rustfs:sqs:ap-northeast-1:primary:webhook".to_string()];
let config = BucketNotificationConfig::from_xml(Cursor::new(xml.as_bytes()), current_region, &arn_list).unwrap();
let rules_map = config.get_rules_map();
// Test with URL-encoded space: "uploads/my%20file.csv"
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/my%20file.csv");
// This may or may not match depending on encoding
println!("URL-encoded key 'uploads/my%20file.csv' matched: {}", !targets.is_empty());
// Test with unencoded version
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/my file.csv");
assert!(!targets.is_empty(), "Unencoded key should match");
}
/// Test multiple queue configurations
#[test]
fn test_multiple_queue_configs_xml() {
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<NotificationConfiguration>
<QueueConfiguration>
<Id>csv-queue</Id>
<Queue>arn:rustfs:sqs::primary:webhook-csv</Queue>
<Event>s3:ObjectCreated:*</Event>
<Filter>
<S3Key>
<FilterRule>
<Name>prefix</Name>
<Value>uploads/</Value>
</FilterRule>
<FilterRule>
<Name>suffix</Name>
<Value>.csv</Value>
</FilterRule>
</S3Key>
</Filter>
</QueueConfiguration>
<QueueConfiguration>
<Id>jpg-queue</Id>
<Queue>arn:rustfs:sqs::primary:webhook-jpg</Queue>
<Event>s3:ObjectCreated:*</Event>
<Filter>
<S3Key>
<FilterRuleList>
<FilterRule>
<Name>prefix</Name>
<Value>images/</Value>
</FilterRule>
<FilterRule>
<Name>suffix</Name>
<Value>.jpg</Value>
</FilterRule>
</FilterRuleList>
</S3Key>
</Filter>
</QueueConfiguration>
</NotificationConfiguration>"#;
let current_region = "ap-northeast-1";
let arn_list = vec![
"arn:rustfs:sqs:ap-northeast-1:primary:webhook-csv".to_string(),
"arn:rustfs:sqs:ap-northeast-1:primary:webhook-jpg".to_string(),
];
let config = BucketNotificationConfig::from_xml(Cursor::new(xml.as_bytes()), current_region, &arn_list).unwrap();
let rules_map = config.get_rules_map();
// Test CSV files
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/test.csv");
println!("Targets for 'uploads/test.csv': {:?}", targets);
assert!(!targets.is_empty(), "CSV files should match");
// Should have csv target (ARN: arn:rustfs:sqs::primary:webhook-csv)
let csv_target = TargetID::new("primary".to_string(), "webhook-csv".to_string());
assert!(targets.contains(&csv_target), "Should find CSV webhook, got: {:?}", targets);
// Test JPG files
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "images/photo.jpg");
assert!(!targets.is_empty(), "JPG files should match");
// Should have jpg target (ARN: arn:rustfs:sqs:primary:webhook-jpg)
let jpg_target = TargetID::new("primary".to_string(), "webhook-jpg".to_string());
assert!(targets.contains(&jpg_target), "Should find JPG webhook");
}
/// Test compound event type expansion
#[test]
fn test_compound_event_expansion_integration() {
let xml = r#"<?xml version="1.0" encoding="UTF-8"?>
<NotificationConfiguration>
<QueueConfiguration>
<Id>test-queue</Id>
<Queue>arn:rustfs:sqs:ap-northeast-1:primary:webhook</Queue>
<Event>s3:ObjectCreated:*</Event>
<Filter>
<S3Key>
<FilterRuleList>
<FilterRule>
<Name>prefix</Name>
<Value>data/</Value>
</FilterRule>
</FilterRuleList>
</S3Key>
</Filter>
</QueueConfiguration>
</NotificationConfiguration>"#;
let current_region = "ap-northeast-1";
let arn_list = vec!["arn:rustfs:sqs:ap-northeast-1:primary:webhook".to_string()];
let config = BucketNotificationConfig::from_xml(Cursor::new(xml.as_bytes()), current_region, &arn_list).unwrap();
let rules_map = config.get_rules_map();
// ObjectCreated:* should be expanded to all ObjectCreated events
let event_types = [
EventName::ObjectCreatedPut,
EventName::ObjectCreatedPost,
EventName::ObjectCreatedCopy,
EventName::ObjectCreatedCompleteMultipartUpload,
EventName::ObjectCreatedPutRetention,
EventName::ObjectCreatedPutLegalHold,
EventName::ObjectCreatedPutTagging,
EventName::ObjectCreatedDeleteTagging,
];
for event_type in event_types {
assert!(rules_map.has_subscriber(&event_type), "Event {:?} should have subscribers", event_type);
// All should match "data/file.csv"
let targets = rules_map.match_rules(event_type, "data/file.csv");
assert!(!targets.is_empty(), "Event {:?} should match", event_type);
}
}
}
+6
View File
@@ -13,8 +13,14 @@
// limitations under the License.
mod config;
#[cfg(test)]
mod config_test;
pub mod pattern;
mod pattern_rules;
#[cfg(test)]
mod pattern_rules_test;
#[cfg(test)]
mod pattern_test;
mod rules_map;
mod subscriber_index;
mod subscriber_snapshot;
@@ -0,0 +1,481 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Tests for PatternRules and RulesMap
//!
//! This module contains comprehensive tests for the rule matching logic
//! used in bucket notifications, specifically testing the flow from
//! configuration to event matching.
use super::*;
use rustfs_s3_common::EventName;
use rustfs_targets::arn::TargetID;
#[cfg(test)]
mod pattern_rules_tests {
use super::*;
/// Test basic pattern rules matching
#[test]
fn test_pattern_rules_basic_matching() {
let mut rules = PatternRules::new();
let target_id = TargetID::new("test-id".to_string(), "webhook".to_string());
// Add a rule for *.csv files
rules.add("*.csv".to_string(), target_id.clone());
// Test matching
assert!(rules.match_simple("test.csv"));
assert!(rules.match_simple("uploads/test.csv"));
assert!(!rules.match_simple("test.txt"));
// Test match_targets
let targets = rules.match_targets("test.csv");
assert!(targets.contains(&target_id));
let targets = rules.match_targets("test.txt");
assert!(!targets.contains(&target_id));
}
/// Test multiple patterns with different targets
#[test]
fn test_multiple_patterns() {
let mut rules = PatternRules::new();
let csv_target = TargetID::new("csv-target".to_string(), "webhook".to_string());
let jpg_target = TargetID::new("jpg-target".to_string(), "webhook".to_string());
rules.add("*.csv".to_string(), csv_target.clone());
rules.add("*.jpg".to_string(), jpg_target.clone());
// CSV files match csv_target
let targets = rules.match_targets("test.csv");
assert!(targets.contains(&csv_target));
assert!(!targets.contains(&jpg_target));
// JPG files match jpg_target
let targets = rules.match_targets("test.jpg");
assert!(!targets.contains(&csv_target));
assert!(targets.contains(&jpg_target));
// Both patterns match for nested paths
let targets = rules.match_targets("uploads/test.csv");
assert!(targets.contains(&csv_target));
let targets = rules.match_targets("images/photo.jpg");
assert!(targets.contains(&jpg_target));
}
/// Test prefix patterns
#[test]
fn test_prefix_patterns() {
let mut rules = PatternRules::new();
let target_id = TargetID::new("prefix-target".to_string(), "webhook".to_string());
rules.add("uploads/*".to_string(), target_id.clone());
rules.add("images/*".to_string(), target_id.clone());
assert!(rules.match_simple("uploads/test.csv"));
assert!(rules.match_simple("uploads/subdir/test.csv"));
assert!(rules.match_simple("images/photo.jpg"));
assert!(!rules.match_simple("documents/test.txt"));
}
/// Test prefix + suffix pattern (bug scenario)
#[test]
fn test_prefix_suffix_pattern_bug_scenario() {
let mut rules = PatternRules::new();
let target_id = TargetID::new("webhook-target".to_string(), "webhook".to_string());
// This is the pattern generated from prefix="uploads/", suffix=".csv"
rules.add("uploads/*.csv".to_string(), target_id.clone());
// These should match
assert!(rules.match_simple("uploads/test.csv"));
assert!(rules.match_simple("uploads/file1.csv"));
assert!(rules.match_simple("uploads/nested/path/file.csv"));
// These should not match
assert!(!rules.match_simple("uploads/test.txt"));
assert!(!rules.match_simple("files/test.csv"));
// Verify match_targets returns correct targets
let targets = rules.match_targets("uploads/test.csv");
assert_eq!(targets.len(), 1);
assert!(targets.contains(&target_id));
}
/// Test match_all pattern
#[test]
fn test_match_all_pattern() {
let mut rules = PatternRules::new();
let target_id = TargetID::new("all-target".to_string(), "webhook".to_string());
rules.add("*".to_string(), target_id.clone());
assert!(rules.match_simple("anything"));
assert!(rules.match_simple("uploads/test.csv"));
assert!(rules.match_simple(""));
let targets = rules.match_targets("any_key");
assert!(targets.contains(&target_id));
}
/// Test empty rules
#[test]
fn test_empty_rules() {
let rules = PatternRules::new();
assert!(rules.is_empty());
assert!(!rules.match_simple("anything"));
let targets = rules.match_targets("test.csv");
assert!(targets.is_empty());
}
/// Test union operation
#[test]
fn test_union() {
let mut rules1 = PatternRules::new();
let mut rules2 = PatternRules::new();
let target1 = TargetID::new("target1".to_string(), "webhook".to_string());
let target2 = TargetID::new("target2".to_string(), "webhook".to_string());
rules1.add("*.csv".to_string(), target1.clone());
rules2.add("*.jpg".to_string(), target2.clone());
let combined = rules1.union(&rules2);
assert!(combined.match_simple("test.csv"));
assert!(combined.match_simple("test.jpg"));
assert!(!combined.match_simple("test.png"));
let targets = combined.match_targets("test.csv");
assert!(targets.contains(&target1));
}
/// Test difference operation
#[test]
fn test_difference() {
let mut rules1 = PatternRules::new();
let mut rules2 = PatternRules::new();
let target1 = TargetID::new("target1".to_string(), "webhook".to_string());
let target2 = TargetID::new("target2".to_string(), "webhook".to_string());
// Add same target to multiple patterns in rules1
rules1.add("*.csv".to_string(), target1.clone());
rules1.add("*.jpg".to_string(), target1.clone());
rules1.add("*.txt".to_string(), target1.clone());
// Add different target to .jpg pattern in rules2
rules2.add("*.jpg".to_string(), target2.clone());
let diff = rules1.difference(&rules2);
// After difference:
// *.csv: target1 remains (pattern doesn't exist in rules2)
// *.jpg: target1 remains (target1 != target2, so it's not removed)
// *.txt: target1 remains (pattern doesn't exist in rules2)
assert!(diff.match_simple("test.csv"));
assert!(diff.match_simple("test.jpg"));
assert!(diff.match_simple("test.txt"));
}
/// Test difference operation removing same target
#[test]
fn test_difference_same_target() {
let mut rules1 = PatternRules::new();
let mut rules2 = PatternRules::new();
let target1 = TargetID::new("target1".to_string(), "webhook".to_string());
// Add same target to multiple patterns in rules1
rules1.add("*.csv".to_string(), target1.clone());
rules1.add("*.jpg".to_string(), target1.clone());
rules1.add("*.txt".to_string(), target1.clone());
// Add same target to .jpg pattern in rules2
rules2.add("*.jpg".to_string(), target1.clone());
let diff = rules1.difference(&rules2);
// After difference:
// *.csv: target1 remains (pattern doesn't exist in rules2)
// *.jpg: target1 is removed (same target in both)
// *.txt: target1 remains (pattern doesn't exist in rules2)
assert!(diff.match_simple("test.csv"));
assert!(!diff.match_simple("test.jpg"));
assert!(diff.match_simple("test.txt"));
}
/// Test remove_pattern
#[test]
fn test_remove_pattern() {
let mut rules = PatternRules::new();
let target_id = TargetID::new("test-target".to_string(), "webhook".to_string());
rules.add("*.csv".to_string(), target_id.clone());
rules.add("*.jpg".to_string(), target_id.clone());
assert!(rules.match_simple("test.csv"));
assert!(rules.match_simple("test.jpg"));
// Remove .csv pattern
assert!(rules.remove_pattern("*.csv"));
assert!(!rules.match_simple("test.csv"));
assert!(rules.match_simple("test.jpg"));
// Remove non-existent pattern
assert!(!rules.remove_pattern("*.png"));
}
}
#[cfg(test)]
mod rules_map_tests {
use super::*;
/// Test basic rule addition and matching
#[test]
fn test_rules_map_basic() {
let mut rules_map = RulesMap::new();
let target_id = TargetID::new("test-target".to_string(), "webhook".to_string());
// Add rule for ObjectCreatedPut with pattern *
rules_map.add_rule_config(&[EventName::ObjectCreatedPut], "*".to_string(), target_id.clone());
// Check has_subscriber
assert!(rules_map.has_subscriber(&EventName::ObjectCreatedPut));
assert!(!rules_map.has_subscriber(&EventName::ObjectRemovedDelete));
// Check match_rules
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "any_key");
assert!(targets.contains(&target_id));
// Check event with different name doesn't match
let targets = rules_map.match_rules(EventName::ObjectRemovedDelete, "any_key");
assert!(!targets.contains(&target_id));
}
/// Test compound event expansion
#[test]
fn test_compound_event_expansion() {
let mut rules_map = RulesMap::new();
let target_id = TargetID::new("test-target".to_string(), "webhook".to_string());
// Add rule for ObjectCreatedAll (compound event)
rules_map.add_rule_config(&[EventName::ObjectCreatedAll], "*.csv".to_string(), target_id.clone());
// ObjectCreatedAll should be expanded to all ObjectCreated events
assert!(rules_map.has_subscriber(&EventName::ObjectCreatedAll));
assert!(rules_map.has_subscriber(&EventName::ObjectCreatedPut));
assert!(rules_map.has_subscriber(&EventName::ObjectCreatedPost));
assert!(rules_map.has_subscriber(&EventName::ObjectCreatedCopy));
// All should match .csv files
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "test.csv");
assert!(targets.contains(&target_id));
let targets = rules_map.match_rules(EventName::ObjectCreatedPost, "test.csv");
assert!(targets.contains(&target_id));
}
/// Test exact bug scenario
#[test]
fn test_bug_scenario_exact_flow() {
let mut rules_map = RulesMap::new();
let target_id = TargetID::new("webhook-primary".to_string(), "webhook".to_string());
// Configuration: Events=["s3:ObjectCreated:*"], prefix="uploads/", suffix=".csv"
// This generates pattern "uploads/*.csv"
rules_map.add_rule_config(&[EventName::ObjectCreatedAll], "uploads/*.csv".to_string(), target_id.clone());
// When ObjectCreatedPut event occurs with key "uploads/test.csv"
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/test.csv");
// Should match
assert!(!targets.is_empty(), "Targets should not be empty for matching event");
assert!(targets.contains(&target_id), "Should find webhook-primary target");
// Test non-matching key
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/test.txt");
assert!(targets.is_empty(), "Targets should be empty for non-matching key");
// Test matching different suffix
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/subdir/file.csv");
assert!(!targets.is_empty(), "Nested CSV files should also match");
assert!(targets.contains(&target_id));
}
/// Test multiple patterns for same event
#[test]
fn test_multiple_patterns_same_event() {
let mut rules_map = RulesMap::new();
let csv_target = TargetID::new("csv-target".to_string(), "webhook".to_string());
let jpg_target = TargetID::new("jpg-target".to_string(), "webhook".to_string());
// Add different patterns for same event
rules_map.add_rule_config(&[EventName::ObjectCreatedPut], "*.csv".to_string(), csv_target.clone());
rules_map.add_rule_config(&[EventName::ObjectCreatedPut], "*.jpg".to_string(), jpg_target.clone());
// Both patterns should work
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "test.csv");
assert!(targets.contains(&csv_target));
assert!(!targets.contains(&jpg_target));
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "test.jpg");
assert!(!targets.contains(&csv_target));
assert!(targets.contains(&jpg_target));
}
/// Test prefix only filter
#[test]
fn test_prefix_only_filter() {
let mut rules_map = RulesMap::new();
let target_id = TargetID::new("test-target".to_string(), "webhook".to_string());
// Configuration: prefix="images/", no suffix
// Pattern should be "images/*"
rules_map.add_rule_config(&[EventName::ObjectCreatedAll], "images/*".to_string(), target_id.clone());
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "images/photo.jpg");
assert!(targets.contains(&target_id));
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/photo.jpg");
assert!(!targets.contains(&target_id));
}
/// Test suffix only filter
#[test]
fn test_suffix_only_filter() {
let mut rules_map = RulesMap::new();
let target_id = TargetID::new("test-target".to_string(), "webhook".to_string());
// Configuration: no prefix, suffix=".pdf"
// Pattern should be "*.pdf"
rules_map.add_rule_config(&[EventName::ObjectCreatedAll], "*.pdf".to_string(), target_id.clone());
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "document.pdf");
assert!(targets.contains(&target_id));
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "document.txt");
assert!(!targets.contains(&target_id));
}
/// Test empty pattern (no filter)
#[test]
fn test_no_filter() {
let mut rules_map = RulesMap::new();
let target_id = TargetID::new("test-target".to_string(), "webhook".to_string());
// Configuration: no prefix, no suffix
// Pattern should default to "*" (match all)
rules_map.add_rule_config(&[EventName::ObjectCreatedAll], "".to_string(), target_id.clone());
// All keys should match
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "anything.csv");
assert!(targets.contains(&target_id));
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "uploads/test.txt");
assert!(targets.contains(&target_id));
}
/// Test different event types
#[test]
fn test_different_event_types() {
let mut rules_map = RulesMap::new();
let put_target = TargetID::new("put-target".to_string(), "webhook".to_string());
let delete_target = TargetID::new("delete-target".to_string(), "webhook".to_string());
// Add rules for different event types
rules_map.add_rule_config(&[EventName::ObjectCreatedPut], "*.csv".to_string(), put_target.clone());
rules_map.add_rule_config(&[EventName::ObjectRemovedDelete], "*.csv".to_string(), delete_target.clone());
// Put event should match put_target
let targets = rules_map.match_rules(EventName::ObjectCreatedPut, "test.csv");
assert!(targets.contains(&put_target));
assert!(!targets.contains(&delete_target));
// Delete event should match delete_target
let targets = rules_map.match_rules(EventName::ObjectRemovedDelete, "test.csv");
assert!(!targets.contains(&put_target));
assert!(targets.contains(&delete_target));
}
/// Test remove_map
#[test]
fn test_remove_map() {
let mut rules_map1 = RulesMap::new();
let mut rules_map2 = RulesMap::new();
let target1 = TargetID::new("target1".to_string(), "webhook".to_string());
let target2 = TargetID::new("target2".to_string(), "webhook".to_string());
rules_map1.add_rule_config(&[EventName::ObjectCreatedPut], "*.csv".to_string(), target1.clone());
rules_map1.add_rule_config(&[EventName::ObjectCreatedPost], "*.jpg".to_string(), target1.clone());
rules_map2.add_rule_config(&[EventName::ObjectCreatedPut], "*.csv".to_string(), target2.clone());
// Remove rules_map2 from rules_map1
rules_map1.remove_map(&rules_map2);
// ObjectCreatedPut rule should still exist with target1 (different from target2)
let targets = rules_map1.match_rules(EventName::ObjectCreatedPut, "test.csv");
assert!(targets.contains(&target1));
assert!(!targets.contains(&target2));
// ObjectCreatedPost rule should still exist
let targets = rules_map1.match_rules(EventName::ObjectCreatedPost, "test.jpg");
assert!(targets.contains(&target1));
}
/// Test remove_map removing same target
#[test]
fn test_remove_map_same_target() {
let mut rules_map1 = RulesMap::new();
let mut rules_map2 = RulesMap::new();
let target1 = TargetID::new("target1".to_string(), "webhook".to_string());
rules_map1.add_rule_config(&[EventName::ObjectCreatedPut], "*.csv".to_string(), target1.clone());
rules_map1.add_rule_config(&[EventName::ObjectCreatedPost], "*.jpg".to_string(), target1.clone());
rules_map2.add_rule_config(&[EventName::ObjectCreatedPut], "*.csv".to_string(), target1.clone());
// Remove rules_map2 from rules_map1
rules_map1.remove_map(&rules_map2);
// ObjectCreatedPut rule should be removed (same target and pattern)
let targets = rules_map1.match_rules(EventName::ObjectCreatedPut, "test.csv");
assert!(targets.is_empty());
// ObjectCreatedPost rule should still exist
let targets = rules_map1.match_rules(EventName::ObjectCreatedPost, "test.jpg");
assert!(targets.contains(&target1));
}
/// Test contains_target_id
#[test]
fn test_contains_target_id() {
let mut rules_map = RulesMap::new();
let target_id = TargetID::new("test-target".to_string(), "webhook".to_string());
assert!(!rules_map.contains_target_id(&target_id));
rules_map.add_rule_config(&[EventName::ObjectCreatedPut], "*.csv".to_string(), target_id.clone());
assert!(rules_map.contains_target_id(&target_id));
}
}
+166
View File
@@ -0,0 +1,166 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Pattern matching tests for notification filter rules
//!
//! This module contains tests for the pattern matching logic used in
//! bucket notification filters, specifically testing prefix and suffix
//! filter rules as used in S3 bucket notifications.
use super::pattern;
#[cfg(test)]
mod bug_reproduction_tests {
use super::*;
/// Test case from bug report:
/// - Prefix: "uploads/"
/// - Suffix: ".csv"
/// - Object key: "uploads/test.csv"
#[test]
fn test_uploads_csv_pattern() {
let pattern = pattern::new_pattern(Some("uploads/"), Some(".csv"));
assert_eq!(pattern, "uploads/*.csv");
assert!(pattern::match_simple(&pattern, "uploads/test.csv"));
assert!(pattern::match_simple(&pattern, "uploads/subdir/file.csv"));
assert!(!pattern::match_simple(&pattern, "uploads/test.txt"));
}
/// Test prefix only
#[test]
fn test_prefix_only() {
let pattern = pattern::new_pattern(Some("images/"), None);
assert_eq!(pattern, "images/*");
assert!(pattern::match_simple(&pattern, "images/photo.jpg"));
assert!(pattern::match_simple(&pattern, "images/subdir/photo.png"));
assert!(!pattern::match_simple(&pattern, "documents/photo.jpg"));
}
/// Test suffix only
#[test]
fn test_suffix_only() {
let pattern = pattern::new_pattern(None, Some(".jpg"));
assert_eq!(pattern, "*.jpg");
assert!(pattern::match_simple(&pattern, "photo.jpg"));
assert!(pattern::match_simple(&pattern, "uploads/photo.jpg"));
assert!(pattern::match_simple(&pattern, "a/b/photo.jpg"));
assert!(!pattern::match_simple(&pattern, "photo.png"));
}
/// Test empty pattern (no filter)
#[test]
fn test_empty_pattern() {
let pattern = pattern::new_pattern(None, None);
assert_eq!(pattern, "");
// Empty pattern should not match anything
assert!(!pattern::match_simple(&pattern, "anything"));
}
/// Test complex patterns
#[test]
fn test_complex_patterns() {
// Pattern with both prefix and suffix
let pattern = pattern::new_pattern(Some("data/uploads/"), Some(".csv"));
assert_eq!(pattern, "data/uploads/*.csv");
assert!(pattern::match_simple(&pattern, "data/uploads/test.csv"));
assert!(!pattern::match_simple(&pattern, "data/test.csv"));
// Pattern with prefix containing special characters
let pattern = pattern::new_pattern(Some("special-chars_123/"), Some(".txt"));
assert_eq!(pattern, "special-chars_123/*.txt");
assert!(pattern::match_simple(&pattern, "special-chars_123/file.txt"));
}
/// Test edge cases
#[test]
fn test_edge_cases() {
// Empty prefix and non-empty suffix
let pattern = pattern::new_pattern(Some(""), Some(".csv"));
assert_eq!(pattern, "*.csv");
// Non-empty prefix and empty suffix
let pattern = pattern::new_pattern(Some("uploads/"), Some(""));
assert_eq!(pattern, "uploads/*");
// Pattern ending with star in prefix
let pattern = pattern::new_pattern(Some("uploads*"), Some(".csv"));
assert_eq!(pattern, "uploads*.csv");
// Pattern starting with star in suffix
let pattern = pattern::new_pattern(Some("uploads/"), Some("*csv"));
assert_eq!(pattern, "uploads/*csv");
}
/// Test the exact scenario from bug report
#[test]
fn test_bug_report_exact_scenario() {
// Configuration: prefix="uploads/", suffix=".csv"
let prefix = Some("uploads/");
let suffix = Some(".csv");
let object_key = "uploads/test.csv";
// Generate pattern from filter rules
let pattern = pattern::new_pattern(prefix, suffix);
println!("Generated pattern: '{}'", pattern);
// Verify pattern is correct
assert_eq!(pattern, "uploads/*.csv");
// Test matching
let matches = pattern::match_simple(&pattern, object_key);
println!("Pattern '{}' matches key '{}': {}", pattern, object_key, matches);
assert!(matches, "Pattern '{}' should match object key '{}'", pattern, object_key);
// Test other keys
assert!(pattern::match_simple(&pattern, "uploads/file1.csv"));
assert!(pattern::match_simple(&pattern, "uploads/nested/file.csv"));
assert!(!pattern::match_simple(&pattern, "uploads/file.txt"));
assert!(!pattern::match_simple(&pattern, "files/test.csv"));
}
/// Test with multiple slashes in path
#[test]
fn test_multiple_slashes() {
let pattern = pattern::new_pattern(Some("a/b/c/"), Some(".csv"));
assert_eq!(pattern, "a/b/c/*.csv");
assert!(pattern::match_simple(&pattern, "a/b/c/test.csv"));
assert!(!pattern::match_simple(&pattern, "a/b/test.csv"));
}
/// Test wildcard behavior
#[test]
fn test_wildcard_behavior() {
// Test that * matches any characters including slashes
let pattern = "uploads/*.csv";
assert!(pattern::match_simple(pattern, "uploads/test.csv"));
assert!(pattern::match_simple(pattern, "uploads/subdir/test.csv"));
assert!(pattern::match_simple(pattern, "uploads/a/b/c/test.csv"));
// Test that multiple wildcards work
let pattern = "a*b*c.csv";
assert!(pattern::match_simple(pattern, "a123b456c.csv"));
assert!(pattern::match_simple(pattern, "abc.csv"));
}
/// Test match_all pattern
#[test]
fn test_match_all_pattern() {
// S3 uses "*" to match all objects
let pattern = "*";
assert!(pattern::match_simple(pattern, "anything"));
assert!(pattern::match_simple(pattern, "uploads/test.csv"));
assert!(pattern::match_simple(pattern, ""));
}
}
+101 -13
View File
@@ -83,17 +83,20 @@ impl FilterRule {
}
}
#[derive(Debug, Serialize, Deserialize, Clone, Default, PartialEq, Eq)]
pub struct FilterRuleList {
#[derive(Debug, Serialize, Clone, Default, PartialEq, Eq)]
pub struct S3KeyFilter {
#[serde(rename = "FilterRule", default, skip_serializing_if = "Vec::is_empty")]
pub rules: Vec<FilterRule>,
pub filter_rule_list: Vec<FilterRule>,
}
impl FilterRuleList {
impl S3KeyFilter {
/// Validate filter rules for duplicates.
/// According to AWS S3 documentation, there can be at most one prefix
/// and one suffix filter rule per queue configuration.
pub fn validate(&self) -> Result<(), ParseConfigError> {
let mut has_prefix = false;
let mut has_suffix = false;
for rule in &self.rules {
for rule in &self.filter_rule_list {
rule.validate()?;
if rule.name == "prefix" {
if has_prefix {
@@ -110,11 +113,19 @@ impl FilterRuleList {
Ok(())
}
/// Check if filter rule list is empty.
pub fn is_empty(&self) -> bool {
self.filter_rule_list.is_empty()
}
/// Generate pattern string from filter rules.
/// This method extracts prefix and suffix values from filter rules
/// and generates a wildcard pattern string for matching object keys.
pub fn pattern(&self) -> String {
let mut prefix_val: Option<&str> = None;
let mut suffix_val: Option<&str> = None;
for rule in &self.rules {
for rule in &self.filter_rule_list {
if rule.name == "prefix" {
prefix_val = Some(&rule.value);
} else if rule.name == "suffix" {
@@ -123,16 +134,92 @@ impl FilterRuleList {
}
pattern::new_pattern(prefix_val, suffix_val)
}
}
pub fn is_empty(&self) -> bool {
self.rules.is_empty()
#[derive(Debug, serde::Deserialize)]
struct S3KeyContent {
#[serde(rename = "FilterRule", default)]
filter_rule_list: Vec<FilterRule>,
#[serde(rename = "FilterRuleList", default)]
filter_rule_list_wrapper: Option<S3KeyFilterRuleList>,
}
#[derive(Debug, serde::Deserialize)]
struct S3KeyFilterRuleList {
#[serde(rename = "FilterRule", default)]
filter_rule_list: Vec<FilterRule>,
}
impl S3KeyContent {
/// Get all filter rules from this S3Key content, handling both direct FilterRule
/// and FilterRuleList wrapper structures
fn get_filter_rules(&self) -> Vec<FilterRule> {
// If we have a FilterRuleList wrapper, use that
if let Some(wrapper) = &self.filter_rule_list_wrapper
&& !wrapper.filter_rule_list.is_empty()
{
return wrapper.filter_rule_list.clone();
}
// Otherwise use direct FilterRule list
self.filter_rule_list.clone()
}
}
#[derive(Debug, Serialize, Deserialize, Clone, Default, PartialEq, Eq)]
pub struct S3KeyFilter {
#[serde(rename = "FilterRuleList", default, skip_serializing_if = "FilterRuleList::is_empty")]
pub filter_rule_list: FilterRuleList,
/// Custom deserializer for S3KeyFilter to handle Filter element correctly.
/// AWS S3 XML structure: <Filter><S3Key><FilterRule>...</FilterRule></S3Key></Filter>
impl<'de> Deserialize<'de> for S3KeyFilter {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de>,
{
struct S3KeyFilterVisitor {
filter_rules: Vec<FilterRule>,
}
impl S3KeyFilterVisitor {
fn new() -> Self {
Self {
filter_rules: Vec::new(),
}
}
}
impl<'de> serde::de::Visitor<'de> for S3KeyFilterVisitor {
type Value = S3KeyFilter;
fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
write!(
formatter,
"an S3Key filter map with an `S3Key` element containing one or more \
`FilterRule` children (e.g. <Filter><S3Key><FilterRule>...</FilterRule></S3Key></Filter>)"
)
}
fn visit_map<V>(mut self, mut map: V) -> Result<Self::Value, V::Error>
where
V: serde::de::MapAccess<'de>,
{
while let Some(key) = map.next_key::<String>()? {
match key.as_str() {
"S3Key" => {
// Parse S3Key content which contains FilterRule(s)
let s3key_content: S3KeyContent = map.next_value()?;
self.filter_rules = s3key_content.get_filter_rules();
}
_ => {
map.next_value::<serde::de::IgnoredAny>()?;
}
}
}
Ok(S3KeyFilter {
filter_rule_list: self.filter_rules,
})
}
}
deserializer.deserialize_map(S3KeyFilterVisitor::new())
}
}
#[derive(Debug, Serialize, Deserialize, Clone, PartialEq, Eq)]
@@ -163,7 +250,7 @@ impl QueueConfig {
return Err(ParseConfigError::DuplicateEventName(event.to_string()));
}
}
self.filter.filter_rule_list.validate()?;
self.filter.validate()?;
// Validate ARN (similar to Go's Queue.Validate)
// The Go code checks targetList.Exists(q.ARN.TargetID)
@@ -241,6 +328,7 @@ pub struct NotificationConfiguration {
skip_serializing_if = "Vec::is_empty"
)]
pub lambda_list: Vec<LambdaConfigDetail>, // Modify: Use a new structure
#[serde(
rename = "TopicConfiguration", // Tags for each topic configuration item in XML
default,
+2 -4
View File
@@ -18,6 +18,7 @@ use rustfs_targets::{
store::{Key, Store},
target::EntityTarget,
};
use rustfs_utils::get_env_usize;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::sync::{Semaphore, mpsc};
@@ -179,10 +180,7 @@ pub async fn stream_events_with_batching(
// Configuration parameters
const DEFAULT_BATCH_SIZE: usize = 1;
let batch_size = std::env::var("RUSTFS_EVENT_BATCH_SIZE")
.ok()
.and_then(|s| s.parse::<usize>().ok())
.unwrap_or(DEFAULT_BATCH_SIZE);
let batch_size = get_env_usize("RUSTFS_EVENT_BATCH_SIZE", DEFAULT_BATCH_SIZE);
const BATCH_TIMEOUT: Duration = Duration::from_secs(5);
const MAX_RETRIES: usize = 5;
const BASE_RETRY_DELAY: Duration = Duration::from_secs(2);
+6
View File
@@ -40,6 +40,10 @@ flate2 = { workspace = true }
glob = { workspace = true }
jiff = { workspace = true }
metrics = { workspace = true }
crossbeam-channel = { workspace = true }
crossbeam-deque = { workspace = true }
crossbeam-utils = { workspace = true }
num_cpus = { workspace = true }
nvml-wrapper = { workspace = true, optional = true }
opentelemetry = { workspace = true }
opentelemetry-appender-tracing = { workspace = true, features = ["experimental_use_tracing_span_context", "experimental_metadata_attributes"] }
@@ -56,6 +60,7 @@ tracing-subscriber = { workspace = true, features = ["registry", "std", "fmt", "
tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] }
sysinfo = { workspace = true }
thiserror = { workspace = true }
zstd = { workspace = true, features = ["zstdmt"] }
[target.'cfg(unix)'.dependencies]
pyroscope = { workspace = true, features = ["backend-pprof-rs"] }
@@ -65,3 +70,4 @@ pyroscope = { workspace = true, features = ["backend-pprof-rs"] }
tokio = { workspace = true, features = ["full"] }
tempfile = { workspace = true }
temp-env = { workspace = true }
serde_json = { workspace = true }
+177 -5
View File
@@ -14,7 +14,7 @@ logging, distributed tracing, metrics via OpenTelemetry, and continuous profilin
| **Distributed tracing** | OTLP/HTTP export to Jaeger, Tempo, or any OTel collector |
| **Metrics** | OTLP/HTTP export, bridged from the `metrics` crate facade |
| **Continuous Profiling** | CPU/Memory profiling export to Pyroscope |
| **Log cleanup** | Background task: size limits, gzip compression, retention policies |
| **Log cleanup** | Background task: size limits, zstd/gzip compression, retention policies |
| **GPU metrics** *(optional)* | Enable with the `gpu` feature flag |
---
@@ -148,9 +148,15 @@ All configuration is read from environment variables at startup.
|-------------------------------------------------|--------------|-------------------------------------------------------------|
| `RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES` | `2147483648` | Hard cap on total log directory size (2 GiB) |
| `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` | `0` | Per-file size cap; `0` = unlimited |
| `RUSTFS_OBS_LOG_COMPRESS_OLD_FILES` | `true` | Gzip-compress files before deleting |
| `RUSTFS_OBS_LOG_COMPRESS_OLD_FILES` | `true` | Compress files before deleting |
| `RUSTFS_OBS_LOG_GZIP_COMPRESSION_LEVEL` | `6` | Gzip level `1` (fastest) `9` (best) |
| `RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS` | `30` | Delete `.gz` archives older than N days; `0` = keep forever |
| `RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM` | `zstd` | Compression codec: `zstd` or `gzip` |
| `RUSTFS_OBS_LOG_PARALLEL_COMPRESS` | `true` | Enable work-stealing parallel compression |
| `RUSTFS_OBS_LOG_PARALLEL_WORKERS` | `6` | Number of cleaner worker threads |
| `RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL` | `8` | Zstd level `1` (fastest) `21` (best ratio) |
| `RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP` | `true` | Fallback to gzip when zstd compression fails |
| `RUSTFS_OBS_LOG_ZSTD_WORKERS` | `1` | zstdmt worker threads per compression task |
| `RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS` | `30` | Delete `.gz` / `.zst` archives older than N days; `0` = keep forever |
| `RUSTFS_OBS_LOG_EXCLUDE_PATTERNS` | _(empty)_ | Comma-separated glob patterns to never clean up |
| `RUSTFS_OBS_LOG_DELETE_EMPTY_FILES` | `true` | Remove zero-byte files |
| `RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS` | `3600` | Minimum file age (seconds) before cleanup |
@@ -159,6 +165,159 @@ All configuration is read from environment variables at startup.
---
## Cleaner & Rotation Metrics
The log rotation and cleanup pipeline emits these metrics (via the `metrics` facade):
| Metric | Type | Description |
|---|---|---|
| `rustfs.log_cleaner.deleted_files_total` | counter | Number of files deleted per cleanup pass |
| `rustfs.log_cleaner.freed_bytes_total` | counter | Bytes reclaimed by deletion |
| `rustfs.log_cleaner.compress_duration_seconds` | histogram | Compression stage duration |
| `rustfs.log_cleaner.steal_success_rate` | gauge | Work-stealing success ratio in parallel mode |
| `rustfs.log_cleaner.runs_total` | counter | Successful cleanup loop runs |
| `rustfs.log_cleaner.run_failures_total` | counter | Failed or panicked cleanup loop runs |
| `rustfs.log_cleaner.rotation_total` | counter | Successful file rotations |
| `rustfs.log_cleaner.rotation_failures_total` | counter | Failed file rotations |
| `rustfs.log_cleaner.rotation_duration_seconds` | histogram | Rotation latency |
| `rustfs.log_cleaner.active_file_size_bytes` | gauge | Current active log file size |
These metrics cover compression, cleanup, and file rotation end-to-end.
### Metric Semantics
- `deleted_files_total` and `freed_bytes_total` are emitted after each cleanup pass and include both normal log cleanup and expired compressed archive cleanup.
- `compress_duration_seconds` measures compression stage wall-clock time for both serial and parallel modes.
- `steal_success_rate` is updated by the parallel work-stealing path and remains at the last computed value.
- `rotation_*` metrics are emitted by `RollingAppender` and include retries; a failed final rotation increments `rotation_failures_total`.
- `active_file_size_bytes` is sampled on writes and after successful roll, so dashboards can track current active file growth.
### Grafana Dashboard JSON Draft (Ready to Import)
> Save this as `rustfs-log-cleaner-dashboard.json`, then import from Grafana UI.
> For Prometheus datasources, metric names are usually normalized to underscores,
> so `rustfs.log_cleaner.deleted_files_total` becomes `rustfs_log_cleaner_deleted_files_total`.
>
> The same panels are now checked in at:
> `.docker/observability/grafana/dashboards/rustfs.json`
> (row title: `Log Cleaner`).
```json
{
"uid": "rustfs-log-cleaner",
"title": "RustFS Log Cleaner",
"timezone": "browser",
"schemaVersion": 39,
"version": 1,
"refresh": "10s",
"tags": ["rustfs", "observability", "log-cleaner"],
"time": {
"from": "now-6h",
"to": "now"
},
"panels": [
{
"id": 1,
"title": "Cleanup Runs / Failures",
"type": "timeseries",
"targets": [
{ "refId": "A", "expr": "sum(rate(rustfs_log_cleaner_runs_total[5m]))", "legendFormat": "runs/s" },
{ "refId": "B", "expr": "sum(rate(rustfs_log_cleaner_run_failures_total[5m]))", "legendFormat": "failures/s" }
],
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }
},
{
"id": 2,
"title": "Freed Bytes / Deleted Files",
"type": "timeseries",
"targets": [
{ "refId": "A", "expr": "sum(rate(rustfs_log_cleaner_freed_bytes_total[15m]))", "legendFormat": "bytes/s" },
{ "refId": "B", "expr": "sum(rate(rustfs_log_cleaner_deleted_files_total[15m]))", "legendFormat": "files/s" }
],
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }
},
{
"id": 3,
"title": "Compression P95 Latency",
"type": "timeseries",
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.95, sum(rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])) by (le))",
"legendFormat": "p95"
}
],
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 }
},
{
"id": 4,
"title": "Rotation Success / Failure",
"type": "timeseries",
"targets": [
{ "refId": "A", "expr": "sum(rate(rustfs_log_cleaner_rotation_total[5m]))", "legendFormat": "rotation/s" },
{ "refId": "B", "expr": "sum(rate(rustfs_log_cleaner_rotation_failures_total[5m]))", "legendFormat": "rotation_failures/s" }
],
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 }
},
{
"id": 5,
"title": "Steal Success Rate",
"type": "timeseries",
"targets": [
{ "refId": "A", "expr": "max(rustfs_log_cleaner_steal_success_rate)", "legendFormat": "ratio" }
],
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 }
},
{
"id": 6,
"title": "Active File Size",
"type": "timeseries",
"targets": [
{ "refId": "A", "expr": "max(rustfs_log_cleaner_active_file_size_bytes)", "legendFormat": "bytes" }
],
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 }
}
]
}
```
### PromQL Templates
Use these templates directly in Grafana panels/alerts.
- **Cleanup run rate**
- `sum(rate(rustfs_log_cleaner_runs_total[$__rate_interval]))`
- **Cleanup failure rate**
- `sum(rate(rustfs_log_cleaner_run_failures_total[$__rate_interval]))`
- **Cleanup failure ratio**
- `sum(rate(rustfs_log_cleaner_run_failures_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_log_cleaner_runs_total[$__rate_interval])), 1e-9)`
- **Freed bytes throughput**
- `sum(rate(rustfs_log_cleaner_freed_bytes_total[$__rate_interval]))`
- **Deleted files throughput**
- `sum(rate(rustfs_log_cleaner_deleted_files_total[$__rate_interval]))`
- **Compression p95 latency**
- `histogram_quantile(0.95, sum(rate(rustfs_log_cleaner_compress_duration_seconds_bucket[$__rate_interval])) by (le))`
- **Rotation failure ratio**
- `sum(rate(rustfs_log_cleaner_rotation_failures_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_log_cleaner_rotation_total[$__rate_interval])), 1e-9)`
- **Work-stealing efficiency (latest)**
- `max(rustfs_log_cleaner_steal_success_rate)`
- **Active file size (latest)**
- `max(rustfs_log_cleaner_active_file_size_bytes)`
### Suggested Alerts
- **CleanupFailureRatioHigh**: failure ratio > 0.05 for 10m.
- **CompressionLatencyP95High**: p95 above your baseline SLO for 15m.
- **RotationFailuresDetected**: rotation failure rate > 0 for 3 consecutive windows.
- **NoCleanupActivity**: runs rate == 0 for expected active environments.
### Metrics Compatibility
The project is currently in active development. Metric names and labels are updated directly when architecture evolves, and no backward-compatibility shim is maintained for old names.
Use the metric names documented in this README as the current source of truth.
---
## Examples
### Stdout-only (development default)
@@ -207,6 +366,19 @@ export RUSTFS_OBS_LOG_DRY_RUN=true
# Observe log output — no files will actually be deleted.
```
### Parallel zstd cleanup (recommended production profile)
```bash
export RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs
export RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM=zstd
export RUSTFS_OBS_LOG_PARALLEL_COMPRESS=true
export RUSTFS_OBS_LOG_PARALLEL_WORKERS=6
export RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL=8
export RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP=true
export RUSTFS_OBS_LOG_ZSTD_WORKERS=1
./rustfs
```
---
## Module Structure
@@ -229,9 +401,9 @@ rustfs-obs/src/
├── cleaner/ # Background log-file cleanup subsystem
│ ├── mod.rs # LogCleaner public API + tests
│ ├── types.rs # FileInfo shared type
│ ├── types.rs # Shared cleaner types (match mode, compression codec, FileInfo)
│ ├── scanner.rs # Filesystem discovery
│ ├── compress.rs # Gzip compression helper
│ ├── compress.rs # Gzip/Zstd compression helper
│ └── core.rs # Selection, compression, deletion logic
└── system/ # Host metrics (CPU, memory, disk, GPU)
+102 -43
View File
@@ -1,71 +1,130 @@
# Log Cleaner Subsystem
The `cleaner` module provides a robust, background log-file lifecycle manager for RustFS. It is designed to run periodically to enforce retention policies, compress old logs, and prevent disk exhaustion.
The `cleaner` module is a production-focused background lifecycle manager for RustFS log archives.
It periodically discovers rolled files, applies retention constraints, compresses candidates, and then deletes sources safely.
## Architecture
The subsystem is designed to be conservative by default:
The cleaner operates as a pipeline:
- it never touches the currently active log file;
- it refuses symlink deletion during the destructive phase;
- it keeps compression and source deletion as separate steps;
- it supports a full dry-run mode for policy verification.
1. **Discovery (`scanner.rs`)**: Scans the configured log directory for eligible files.
* **Non-recursive**: Only scans the top-level directory for safety.
* **Filtering**: Ignores the currently active log file, files matching exclude patterns, and files that do not match the configured prefix/suffix pattern.
* **Performance**: Uses `std::fs::read_dir` directly to minimize overhead and syscalls.
## Execution Pipeline
2. **Selection (`core.rs`)**: Applies retention policies to select files for deletion.
* **Keep Count**: Ensures at least `N` recent files are kept.
* **Total Size**: Deletes oldest files if the total size exceeds the limit.
* **Single File Size**: Deletes individual files that exceed a size limit (e.g., runaway logs).
1. **Discovery (`scanner.rs`)**
- Performs a shallow `read_dir` scan (no recursion) for predictable latency.
- Excludes the active log file, exclusion-pattern matches, and files younger than the age threshold.
- Classifies regular logs and compressed archives (`.gz` / `.zst`) in one pass.
3. **Action (`core.rs` / `compress.rs`)**:
* **Compression**: Optionally compresses selected files using Gzip (level 1-9) before deletion.
* **Deletion**: Removes the original file (and eventually the compressed archive based on retention days).
2. **Selection (`core.rs`)**
- Enforces keep-count first.
- Applies total-size and single-file-size constraints to oldest files.
- Produces an ordered list of files to process.
## Configuration
3. **Compression + Deletion (`core.rs` + `compress.rs`)**
- Supports `zstd` and `gzip` codecs.
- Uses parallel work stealing when enabled (`Injector + Worker::new_fifo + Stealer`).
- Always deletes source files in a serial pass after compression to minimize file-lock race issues.
The cleaner is configured via `LogCleanerBuilder`. When initialized via `rustfs-obs::init_obs`, it reads from environment variables.
4. **Archive Expiry (`core.rs`)**
- Applies a separate retention window to already-compressed files.
- Keeps archive expiration independent from plain-log keep-count logic.
| Parameter | Env Var | Description |
|-----------|---------|-------------|
| `log_dir` | `RUSTFS_OBS_LOG_DIRECTORY` | The directory to scan. |
| `file_pattern` | `RUSTFS_OBS_LOG_FILENAME` | The base filename pattern (e.g., `rustfs.log`). |
| `active_filename` | (Derived) | The exact name of the currently active log file, excluded from cleanup. |
| `match_mode` | `RUSTFS_OBS_LOG_MATCH_MODE` | `prefix` or `suffix`. Determines how `file_pattern` is matched against filenames. |
| `keep_files` | `RUSTFS_OBS_LOG_KEEP_FILES` | Minimum number of rolling log files to keep. |
| `max_total_size_bytes` | `RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES` | Maximum aggregate size of all log files. Oldest files are deleted to satisfy this. |
| `compress_old_files` | `RUSTFS_OBS_LOG_COMPRESS_OLD_FILES` | If `true`, files selected for removal are first gzipped. |
| `compressed_file_retention_days` | `RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS` | Age in days after which `.gz` files are deleted. |
## Compression Modes
## Timestamp Format & Rotation
- **Primary codec**: `zstd` (default) for better ratio and faster decompression.
- **Fallback codec**: `gzip` when zstd fallback is enabled.
- **Dry-run**: reports planned compression/deletion operations without touching filesystem state.
The cleaner works in tandem with the `RollingAppender` in `telemetry/rolling.rs`.
## Safety Model
* **Rotation**: Logs are rotated based on time (Daily/Hourly/Minutely) or Size.
* **Naming**: Archived logs use a high-precision timestamp format: `YYYYMMDDHHMMSS.uuuuuu` (microseconds), plus a unique counter to prevent collisions.
* **Suffix Mode**: `<timestamp>-<counter>.<filename>` (e.g., `20231027103001.123456-0.rustfs.log`)
* **Prefix Mode**: `<filename>.<timestamp>-<counter>` (e.g., `rustfs.log.20231027103001.123456-0`)
- **No recursive traversal**: the scanner only inspects the immediate log directory.
- **No symlink following**: filesystem metadata is collected with `symlink_metadata`.
- **Idempotent archives**: an existing `*.gz` or `*.zst` target means the file is treated as already compressed.
- **Best-effort cleanup**: individual file failures are logged and do not abort the whole maintenance pass.
This high-precision naming ensures that files sort chronologically by name, and collisions are virtually impossible even under high load.
## Work-Stealing Strategy
## Usage Example
The parallel path in `core.rs` uses this fixed lookup sequence per worker:
1. `local_worker.pop()`
2. `injector.steal_batch_and_pop(&local_worker)`
3. randomized victim polling via `Steal::from_iter(...)`
This strategy keeps local cache affinity while still balancing stragglers.
## Metrics and Tracing
The cleaner emits tracing events and runtime metrics:
- `rustfs.log_cleaner.deleted_files_total` (counter)
- `rustfs.log_cleaner.freed_bytes_total` (counter)
- `rustfs.log_cleaner.compress_duration_seconds` (histogram)
- `rustfs.log_cleaner.steal_success_rate` (gauge)
- `rustfs.log_cleaner.rotation_total` (counter)
- `rustfs.log_cleaner.rotation_failures_total` (counter)
- `rustfs.log_cleaner.rotation_duration_seconds` (histogram)
- `rustfs.log_cleaner.active_file_size_bytes` (gauge)
These values can be wired into dashboards and alert rules for cleanup health.
## Retention Decision Order
For regular logs, the cleaner evaluates candidates in this order:
1. keep at least `keep_files` newest matching generations;
2. remove older files if total retained size still exceeds `max_total_size_bytes`;
3. remove any file whose individual size exceeds `max_single_file_size_bytes`;
4. if compression is enabled, archive before deletion;
5. delete the original file only after successful compression.
## Key Environment Variables
| Env Var | Meaning |
|---|---|
| `RUSTFS_OBS_LOG_COMPRESSION_ALGORITHM` | `zstd` or `gzip` |
| `RUSTFS_OBS_LOG_PARALLEL_COMPRESS` | Enable work-stealing compression |
| `RUSTFS_OBS_LOG_PARALLEL_WORKERS` | Worker count for parallel compressor |
| `RUSTFS_OBS_LOG_ZSTD_COMPRESSION_LEVEL` | Zstd level (1-21) |
| `RUSTFS_OBS_LOG_ZSTD_FALLBACK_TO_GZIP` | Fallback switch on zstd failure |
| `RUSTFS_OBS_LOG_ZSTD_WORKERS` | zstdmt worker threads per compression task |
| `RUSTFS_OBS_LOG_DRY_RUN` | Dry-run mode |
| `RUSTFS_OBS_LOG_COMPRESSED_FILE_RETENTION_DAYS` | Retention window for `*.gz` / `*.zst` archives |
| `RUSTFS_OBS_LOG_DELETE_EMPTY_FILES` | Remove zero-byte regular log files during scanning |
| `RUSTFS_OBS_LOG_MIN_FILE_AGE_SECONDS` | Minimum age for regular log eligibility |
## Builder Example
```rust
use rustfs_obs::LogCleaner;
use rustfs_obs::types::FileMatchMode;
use rustfs_obs::types::{CompressionAlgorithm, FileMatchMode};
use std::path::PathBuf;
let cleaner = LogCleaner::builder(
PathBuf::from("/var/log/rustfs"),
"rustfs.log.".to_string(),
"rustfs.log".to_string(),
"rustfs.log".to_string(),
)
.match_mode(FileMatchMode::Prefix)
.keep_files(10)
.max_total_size_bytes(1024 * 1024 * 100) // 100 MB
.match_mode(FileMatchMode::Suffix)
.keep_files(30)
.max_total_size_bytes(2 * 1024 * 1024 * 1024)
.compress_old_files(true)
.compression_algorithm(CompressionAlgorithm::Zstd)
.parallel_compress(true)
.parallel_workers(6)
.zstd_compression_level(8)
.zstd_fallback_to_gzip(true)
.zstd_workers(1)
.dry_run(false)
.build();
// Run cleanup (blocking operation, spawn in a background task)
if let Ok((deleted, freed)) = cleaner.cleanup() {
println!("Cleaned up {} files, freed {} bytes", deleted, freed);
}
let _ = cleaner.cleanup();
```
## Operational Notes
- Prefer `FileMatchMode::Suffix` when rotations prepend timestamps to the filename.
- Prefer `FileMatchMode::Prefix` when rotations append counters or timestamps after a stable base name.
- Keep `parallel_workers` modest when `zstd_workers` is greater than `1`, because each compression task may already use internal codec threads.
+178 -39
View File
@@ -12,66 +12,205 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Gzip compression helper for old log files.
//! Compression helpers for old log files.
//!
//! Files are compressed in place: `<name>` → `<name>.gz`. The original file
//! is **not** deleted here — deletion is handled by the caller after
//! compression succeeds.
//! This module performs compression only. Source-file deletion is intentionally
//! handled by the caller in a separate step to avoid platform-specific file
//! locking issues (especially on Windows).
//!
//! The separation is important for operational safety: a failed archive write
//! must never result in premature source deletion, and an already-existing
//! archive should make repeated cleanup passes idempotent.
use super::types::CompressionAlgorithm;
use flate2::Compression;
use flate2::write::GzEncoder;
use rustfs_config::observability::DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION;
use std::ffi::OsString;
use std::fs::File;
use std::io::{BufReader, BufWriter, Write};
use std::path::Path;
use tracing::{debug, info};
use std::path::{Path, PathBuf};
use tracing::{debug, info, warn};
/// Compress `path` to `<path>.gz` using gzip.
/// Compression options shared by serial and parallel cleaner paths.
///
/// If a `.gz` file for the given path already exists the function returns
/// `Ok(())` immediately without overwriting the existing archive.
///
/// # Arguments
/// * `path` - Path to the uncompressed log file.
/// * `level` - Gzip compression level (`1``9`); clamped automatically.
/// * `dry_run` - When `true`, log what would be done without writing anything.
///
/// # Errors
/// Propagates any I/O error encountered while opening, reading, writing, or
/// flushing files.
pub(super) fn compress_file(path: &Path, level: u32, dry_run: bool) -> Result<(), std::io::Error> {
let gz_path = path.with_extension(DEFAULT_OBS_LOG_GZIP_COMPRESSION_EXTENSION);
/// The core cleaner prepares this immutable bundle once per cleanup pass and
/// then hands it to each worker so all files in that run use the same policy.
#[derive(Debug, Clone)]
pub(super) struct CompressionOptions {
/// Preferred compression codec for the current cleanup pass.
pub algorithm: CompressionAlgorithm,
/// Gzip level (1..=9).
pub gzip_level: u32,
/// Zstd level (1..=21).
pub zstd_level: i32,
/// Internal zstd worker threads used by zstdmt.
pub zstd_workers: usize,
/// If true, fallback to gzip when zstd encoding fails.
pub zstd_fallback_to_gzip: bool,
/// Dry-run mode reports planned actions without writing files.
pub dry_run: bool,
}
if gz_path.exists() {
debug!("Compressed file already exists, skipping: {:?}", gz_path);
return Ok(());
/// Compression output metadata used for metrics and deletion gating.
///
/// Callers inspect the output size and archive path before deciding whether it
/// is safe to remove the source log file.
#[derive(Debug, Clone)]
pub(super) struct CompressionOutput {
/// Final path of the compressed archive file.
pub archive_path: PathBuf,
/// Codec actually used to produce the output.
pub algorithm_used: CompressionAlgorithm,
/// Input bytes before compression.
pub input_bytes: u64,
/// Compressed output bytes on disk.
pub output_bytes: u64,
}
/// Compress a single source file with the requested codec and fallback policy.
///
/// This function centralizes the fallback behavior so the orchestration layer
/// does not need per-codec branching.
pub(super) fn compress_file(path: &Path, options: &CompressionOptions) -> Result<CompressionOutput, std::io::Error> {
match options.algorithm {
CompressionAlgorithm::Gzip => compress_gzip(path, options.gzip_level, options.dry_run),
CompressionAlgorithm::Zstd => match compress_zstd(path, options.zstd_level, options.zstd_workers, options.dry_run) {
Ok(output) => Ok(output),
Err(err) if options.zstd_fallback_to_gzip => {
warn!(
file = ?path,
error = %err,
"zstd compression failed, fallback to gzip"
);
compress_gzip(path, options.gzip_level, options.dry_run)
}
Err(err) => Err(err),
},
}
}
/// Compress a file to `*.gz` using the configured gzip level.
fn compress_gzip(path: &Path, level: u32, dry_run: bool) -> Result<CompressionOutput, std::io::Error> {
let archive_path = archive_path(path, CompressionAlgorithm::Gzip);
compress_with_writer(path, &archive_path, dry_run, CompressionAlgorithm::Gzip, |reader, writer| {
let mut encoder = GzEncoder::new(writer, Compression::new(level.clamp(1, 9)));
let written = std::io::copy(reader, &mut encoder)?;
let mut out = encoder.finish()?;
out.flush()?;
Ok(written)
})
}
/// Compress a file to `*.zst` using multi-threaded zstd when available.
fn compress_zstd(path: &Path, level: i32, workers: usize, dry_run: bool) -> Result<CompressionOutput, std::io::Error> {
let archive_path = archive_path(path, CompressionAlgorithm::Zstd);
compress_with_writer(path, &archive_path, dry_run, CompressionAlgorithm::Zstd, |reader, writer| {
let mut encoder = zstd::stream::Encoder::new(writer, level.clamp(1, 21))?;
encoder.multithread(workers.max(1) as u32)?;
let written = std::io::copy(reader, &mut encoder)?;
let mut out = encoder.finish()?;
out.flush()?;
Ok(written)
})
}
fn compress_with_writer<F>(
path: &Path,
archive_path: &Path,
dry_run: bool,
algorithm_used: CompressionAlgorithm,
mut writer_fn: F,
) -> Result<CompressionOutput, std::io::Error>
where
F: FnMut(&mut BufReader<File>, BufWriter<File>) -> Result<u64, std::io::Error>,
{
// Keep idempotent behavior: existing archive means this file has already
// been handled in a previous cleanup pass.
if archive_path.exists() {
debug!(file = ?archive_path, "compressed archive already exists, skipping");
let input_bytes = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0);
let output_bytes = std::fs::metadata(archive_path).map(|m| m.len()).unwrap_or(0);
return Ok(CompressionOutput {
archive_path: archive_path.to_path_buf(),
algorithm_used,
input_bytes,
output_bytes,
});
}
let input_bytes = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0);
if dry_run {
info!("[DRY RUN] Would compress file: {:?} -> {:?}", path, gz_path);
return Ok(());
info!("[DRY RUN] Would compress file: {:?} -> {:?}", path, archive_path);
return Ok(CompressionOutput {
archive_path: archive_path.to_path_buf(),
algorithm_used,
input_bytes,
output_bytes: 0,
});
}
// Create the output archive only after the dry-run short-circuit so this
// helper remains side-effect free when the caller is evaluating policy.
let input = File::open(path)?;
let output = File::create(&gz_path)?;
// Write to a temporary file first and then atomically rename it into place
// so callers never observe a partially written archive at `archive_path`.
let mut tmp_name = archive_path.as_os_str().to_owned();
tmp_name.push(".tmp");
let tmp_archive_path = std::path::PathBuf::from(tmp_name);
let output = File::create(&tmp_archive_path)?;
let mut reader = BufReader::new(input);
let mut writer = BufWriter::new(output);
let writer = BufWriter::new(output);
let mut encoder = GzEncoder::new(Vec::new(), Compression::new(level.clamp(1, 9)));
std::io::copy(&mut reader, &mut encoder)?;
let compressed = encoder.finish()?;
if let Err(e) = writer_fn(&mut reader, writer) {
// Best-effort cleanup of the incomplete temporary archive.
let _ = std::fs::remove_file(&tmp_archive_path);
return Err(e);
}
writer.write_all(&compressed)?;
writer.flush()?;
// Preserve Unix mode bits so rotated archives keep the same access policy.
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
if let Ok(src_meta) = std::fs::metadata(path) {
let mode = src_meta.permissions().mode();
let _ = std::fs::set_permissions(&tmp_archive_path, std::fs::Permissions::from_mode(mode));
}
}
// Atomically move the fully written temp file into its final location.
std::fs::rename(&tmp_archive_path, archive_path)?;
let output_bytes = std::fs::metadata(archive_path).map(|m| m.len()).unwrap_or(0);
debug!(
"Compressed {:?} -> {:?} ({} bytes -> {} bytes)",
path,
gz_path,
std::fs::metadata(path).map(|m| m.len()).unwrap_or(0),
compressed.len()
file = ?path,
archive = ?archive_path,
input_bytes,
output_bytes,
algorithm = %algorithm_used,
"compression finished"
);
Ok(())
Ok(CompressionOutput {
archive_path: archive_path.to_path_buf(),
algorithm_used,
input_bytes,
output_bytes,
})
}
/// Build `<filename>.<ext>` without replacing an existing extension.
///
/// Rotated log filenames often already encode a generation number or suffix
/// (for example `server.log.3`). Appending the archive extension preserves that
/// information in the final artifact name (`server.log.3.gz`).
fn archive_path(path: &Path, algorithm: CompressionAlgorithm) -> PathBuf {
let ext = algorithm.extension();
let mut file_name: OsString = path
.file_name()
.map_or_else(|| OsString::from("archive"), |n| n.to_os_string());
file_name.push(format!(".{ext}"));
path.with_file_name(file_name)
}

Some files were not shown because too many files have changed in this diff Show More