Compare commits

..

1 Commits

Author SHA1 Message Date
overtrue c568a54797 fix(replication): honor disabled version deletes 2026-07-30 13:37:01 +08:00
277 changed files with 5558 additions and 33152 deletions
+16 -80
View File
@@ -1,21 +1,19 @@
---
name: rustfs-release-publish
description: "End-to-end RustFS release pipeline: first publish any merged-but-unreleased rustfs/console changes and wait for its latest Release asset, then bump RustFS version files on main directly to the final target, publish a visible GitHub prerelease from a preview tag without updating latest channels, validate it, and publish the final tag on the SAME commit. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
---
# RustFS Release Publish (preview-validated pipeline)
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Pipeline shape:
```
check console main against its latest Release
-> if ahead: publish console -> wait for Release asset + latest API
-> bump RustFS version files to <target> (final version, ONE commit) -> merge
bump version files to <target> (final version, ONE commit) -> merge
-> tag <preview-tag> at that commit -> CI green
-> verify preview Release assets -> run binary locally + console checks
-> verify release artifacts -> run binary locally + console checks
-> validate with latest rc client
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
```
@@ -25,7 +23,7 @@ On validation failure: fix lands on main via normal PR (version files are alread
## Required inputs
- Final target version, for example `1.0.0-beta.10`.
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` after `git fetch --tags`).
If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
@@ -47,18 +45,14 @@ Rules:
## Preview tag naming
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N``build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
@@ -69,61 +63,6 @@ Rules:
- `gh auth status` works; confirm you can view `gh release list -L 3`.
- Confirm the exact final target version with the user if not explicit.
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
1. Read the latest published Console tag and compare it with Console `main`:
```bash
CONSOLE_REPO="rustfs/console"
CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
--jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
```
- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
```bash
CONSOLE_SCRATCH=$(mktemp -d)
gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
```
If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
```bash
git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
```
Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
3. Find the exact tag run and wait for completion:
```bash
gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
```
4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
```bash
gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
--jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
```
Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
## Phase 1 — Version bump to the final target (once)
- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
@@ -146,17 +85,16 @@ git push origin "<preview-tag>"
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
## Phase 3 — CI and preview Release verification
## Phase 3 — CI and artifact verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
- `isPrerelease` must be `true`.
- Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
## Phase 4 — Run the artifact locally, verify the console
@@ -219,15 +157,13 @@ git push origin "<target>"
```
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Output contract
Always report:
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
- Any deviation from this pipeline and why the user approved it.
@@ -18,7 +18,7 @@ Validated baseline: release pattern used in PR `#2957`.
If target version is missing or ambiguous, stop and ask before editing.
Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
## Read before editing
-10
View File
@@ -60,16 +60,6 @@ The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-con
| `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
| `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
The file `prometheus-rules/rustfs-kms-alerts.yml` contains alerting rules for the KMS backend operation metrics. Thresholds are conservative defaults pending staging baseline calibration; response procedures live in `docs/operations/kms-observability-runbook.md`, and the matching dashboard is `deploy/observability/grafana/rustfs-kms-observability.json`.
| Alert | Severity | Condition |
|-------|----------|-----------|
| `KmsBackendFatalErrors` | Critical | Fatal (non-retryable) attempt failures > 0 for 5m |
| `KmsBackendHighErrorRate` | Critical | Non-success operation ratio > 5% for 10m (with traffic guard) |
| `KmsBackendP99LatencyHigh` | Warning | Operation p99 duration (incl. retries) > 2s for 10m |
| `KmsBackendAttemptFailureSpike` | Warning | Attempt failure rate > 0.5/s for 10m |
| `KmsBackendRetryBudgetExhausted` | Warning | budget_exhausted / deadline_exceeded outcomes > 0.05/s for 10m |
### Enabling Alert Rules
Add the alert rules file to your Prometheus configuration:
-10
View File
@@ -60,16 +60,6 @@
| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线,持续 10 分钟 |
| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%,持续 5 分钟 |
文件 `prometheus-rules/rustfs-kms-alerts.yml` 包含 KMS 后端操作指标的告警规则。阈值为保守默认值,待 staging 基线校准;响应流程见 `docs/operations/kms-observability-runbook.md`,配套仪表盘为 `deploy/observability/grafana/rustfs-kms-observability.json`
| 告警 | 级别 | 条件 |
|------|------|------|
| `KmsBackendFatalErrors` | 严重 | fatal(不可重试)尝试失败 > 0,持续 5 分钟 |
| `KmsBackendHighErrorRate` | 严重 | 非 success 操作占比 > 5%,持续 10 分钟(含流量下限保护) |
| `KmsBackendP99LatencyHigh` | 警告 | 操作 p99 耗时(含重试)> 2s,持续 10 分钟 |
| `KmsBackendAttemptFailureSpike` | 警告 | 尝试失败率 > 0.5/s,持续 10 分钟 |
| `KmsBackendRetryBudgetExhausted` | 警告 | budget_exhausted / deadline_exceeded 结果 > 0.05/s,持续 10 分钟 |
### 启用告警规则
在 Prometheus 配置中添加告警规则文件:
@@ -1,188 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =============================================================================
# RustFS KMS backend — Prometheus alerting rules
# =============================================================================
#
# Metric source: the KMS operation-policy choke point in
# crates/kms/src/policy.rs. All label values are static enum strings
# (operation, op_class, outcome, error_class); key identifiers, key material,
# and tokens never appear in labels.
#
# Response procedures: docs/operations/kms-observability-runbook.md
#
# IMPORTANT — threshold status: every numeric threshold below is a
# conservative default chosen without a production baseline. Calibrate against
# a staging baseline before relying on these alerts for paging, and prefer
# loosening over tightening until the baseline exists. Formal SLO targets are
# deliberately not encoded here (see rustfs/backlog#1584).
#
# NOTE: prometheus.yml loads /etc/prometheus/rules/*.yml — keep the .yml
# extension or the file is silently ignored by the docker-compose stack.
#
# Validate: promtool check rules rustfs-kms-alerts.yml
# =============================================================================
groups:
# ==========================================================================
# Critical alerts — immediate action required
# ==========================================================================
- name: rustfs-kms-critical
interval: 30s
rules:
# ------------------------------------------------------------------
# 1. KmsBackendFatalErrors
# Any attempt failure classified as fatal (non-retryable): auth
# or permission errors, malformed requests, missing keys. The
# policy never retries these, so even a low rate means real
# operations are failing right now.
# ------------------------------------------------------------------
- alert: KmsBackendFatalErrors
expr: |
sum by (operation) (rate(rustfs_kms_backend_attempt_failures_total{error_class="fatal"}[5m])) > 0
for: 5m
labels:
severity: critical
component: kms
annotations:
summary: "KMS backend fatal errors on operation {{ $labels.operation }}"
description: >-
Attempt failures classified as fatal are occurring at
{{ $value | printf "%.3f" }}/s on operation
{{ $labels.operation }}. Fatal failures are not retried:
each one is a KMS backend call that failed permanently
(authentication, permissions, malformed request, or a
missing key/version).
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendfatalerrors"
# ------------------------------------------------------------------
# 2. KmsBackendHighErrorRate
# Sustained share of operations terminating without success
# (fatal, budget_exhausted, deadline_exceeded). The cancelled
# outcome is excluded because shutdowns legitimately produce it.
# The traffic guard keeps a single failure on a near-idle
# cluster from firing the alert.
# Threshold: 5% for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendHighErrorRate
expr: |
(
sum(rate(rustfs_kms_backend_operations_total{outcome!~"success|cancelled"}[5m]))
/
clamp_min(sum(rate(rustfs_kms_backend_operations_total[5m])), 1e-9)
) > 0.05
and
sum(rate(rustfs_kms_backend_operations_total[5m])) > 0.02
for: 10m
labels:
severity: critical
component: kms
annotations:
summary: "KMS backend non-success ratio above 5% for 10m"
description: >-
{{ $value | humanizePercentage }} of KMS backend operations
are terminating in fatal, budget_exhausted, or
deadline_exceeded. Object encryption and decryption paths
depending on the KMS are degraded or failing.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
# ==========================================================================
# Warning alerts — investigation needed
# ==========================================================================
- name: rustfs-kms-warning
interval: 30s
rules:
# ------------------------------------------------------------------
# 3. KmsBackendP99LatencyHigh
# p99 wall-clock duration of whole operations (attempts plus
# backoff) is sustained above 2 seconds. Because the histogram
# includes retries, a high p99 usually means the retry policy
# is absorbing backend failures, not that every call is slow.
# Threshold: 2s for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendP99LatencyHigh
expr: |
histogram_quantile(0.99,
sum by (le) (rate(rustfs_kms_backend_operation_duration_seconds_bucket[5m]))
) > 2
for: 10m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend operation p99 latency above 2s for 10m"
description: >-
The 99th-percentile KMS backend operation duration is
{{ $value | humanizeDuration }}, including retries and
backoff. Encryption and decryption latency is leaking into
S3 request latency.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendp99latencyhigh"
# ------------------------------------------------------------------
# 4. KmsBackendAttemptFailureSpike
# Aggregate attempt-failure rate (all error classes) sustained
# above an absolute floor. An absolute threshold is used instead
# of an offset-1d baseline ratio because fresh deployments have
# no baseline and an empty offset vector would keep a ratio
# alert from ever firing; switch to a baseline-relative form
# (see rustfs-get-optimization-alerts.yaml for the pattern)
# once a stable staging baseline exists.
# Threshold: 0.5/s for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendAttemptFailureSpike
expr: |
sum(rate(rustfs_kms_backend_attempt_failures_total[5m])) > 0.5
for: 10m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend attempt failures above 0.5/s for 10m"
description: >-
KMS backend attempts are failing at
{{ $value | printf "%.2f" }}/s across all error classes.
The retry policy may still be masking these from callers —
check the error-class breakdown before it stops absorbing
them.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendattemptfailurespike"
# ------------------------------------------------------------------
# 5. KmsBackendRetryBudgetExhausted
# Operations are running out of retry budget (budget_exhausted)
# or operation deadline (deadline_exceeded). These surface to
# callers as failed KMS operations even though every individual
# failure was retryable — the backend is unhealthy for longer
# than the policy can bridge.
# Threshold: 0.05/s for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendRetryBudgetExhausted
expr: |
sum by (outcome) (rate(rustfs_kms_backend_operations_total{outcome=~"budget_exhausted|deadline_exceeded"}[5m])) > 0.05
for: 10m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend operations exhausting retry budget ({{ $labels.outcome }})"
description: >-
KMS backend operations are terminating as
{{ $labels.outcome }} at {{ $value | printf "%.3f" }}/s.
Retryable failures are outlasting the retry budget, so
callers are seeing hard failures.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
+11 -31
View File
@@ -25,13 +25,9 @@ inputs:
required: false
default: "rustfs-deps"
cache-save-if:
description: >-
Whether to save the cache. The fail-safe default is 'false': a caller that
wants to populate a cache must opt in explicitly, so a forgotten input
costs a cold cache (minutes) rather than silently consuming the
repository-wide 10GB Actions cache quota and evicting other lanes.
description: "Condition for saving cache"
required: false
default: "false"
default: "true"
install-cross-tools:
description: "Install cross-compilation tools"
required: false
@@ -40,43 +36,28 @@ inputs:
description: "Target architecture to add"
required: false
default: ""
install-build-packaging-tools:
description: >-
Install musl-tools/zip/unzip, needed for musl linking and release
packaging. Off for CI test lanes, which use none of them.
github-token:
description: "GitHub token for API access"
required: false
default: "true"
install-test-tools:
description: >-
Install cargo-nextest and the rustfmt/clippy components. Off for release
and audit lanes, which run no tests and no lints.
required: false
default: "true"
default: ""
runs:
using: "composite"
steps:
# protobuf-compiler is deliberately absent: the setup-protoc step below
# installs 34.1 into the tool cache and prepends it to PATH, so the apt
# build (older, and never version-matched) was shadowed on every run and
# simply never used.
- name: Install system dependencies (Ubuntu)
if: runner.os == 'Linux'
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y \
musl-tools \
build-essential \
pkg-config \
libssl-dev \
ripgrep
# musl-gcc is needed by the native musl release leg, and zip/unzip by the
# release packaging steps. No CI test lane touches any of them.
- name: Install packaging and cross-linking dependencies (Ubuntu)
if: runner.os == 'Linux' && inputs.install-build-packaging-tools == 'true'
shell: bash
run: sudo apt-get install -y musl-tools zip unzip
ripgrep \
unzip \
zip \
protobuf-compiler
- name: Install protoc
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
@@ -94,7 +75,7 @@ runs:
with:
toolchain: ${{ inputs.rust-version }}
targets: ${{ inputs.target }}
components: ${{ inputs.install-test-tools == 'true' && 'rustfmt, clippy' || '' }}
components: rustfmt, clippy
- name: Install Zig
if: inputs.install-cross-tools == 'true'
@@ -105,7 +86,6 @@ runs:
uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
- name: Install cargo-nextest
if: inputs.install-test-tools == 'true'
uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
- name: Setup Rust cache
@@ -37,7 +37,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -46,11 +45,8 @@ jobs:
name: Architecture Migration Rules
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
run: |
+4 -75
View File
@@ -23,8 +23,6 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
pull_request:
types: [ opened, synchronize, reopened, closed ]
@@ -35,16 +33,9 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
schedule:
# Daily, not weekly. This schedule exists to catch RustSec advisories
# published against an unchanged dependency tree; at weekly cadence a new
# advisory could sit unnoticed for seven days. The check list is unchanged —
# splitting it into a light daily advisories-only run and a weekly full run
# would create runs where sources/bans/licenses go unverified.
- cron: '0 3 * * *' # Daily 03:00 UTC (staggered after the midnight ci/build crons)
- cron: '0 3 * * 0' # Weekly on Sunday 03:00 UTC (staggered after the midnight ci/build crons)
workflow_dispatch:
permissions:
@@ -64,7 +55,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -80,30 +70,11 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# cargo-deny compiles nothing, so the full setup composite (apt packages,
# protoc, flatc, nextest, rustfmt/clippy) was pure overhead here. It does
# still need a real cargo: `cargo deny check` runs `cargo metadata`, and
# Cargo.toml pins datafusion and s3s as git dependencies, which must be
# materialised into ~/.cargo/git — a cold clone is hundreds of MB, so the
# cache stays.
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
# Was relying on the composite's default, which used to be "true": every
# PR touching Cargo.toml/Cargo.lock saved a second, PR-scoped copy of this
# cache and pushed the main-scoped lanes out of the 10GB quota. The
# default is now "false", but state it explicitly — see
# scripts/security/check_cache_save_if.sh.
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-all-crates: true
cache-on-failure: true
shared-key: rustfs-cargo-deny
save-if: ${{ github.ref == 'refs/heads/main' }}
cache-shared-key: rustfs-cargo-deny
- name: Install cargo-deny
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
@@ -121,28 +92,13 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
- name: Check setup cache-save-if is explicit
run: ./scripts/security/check_cache_save_if.sh
- name: Check every job declares a timeout
run: ./scripts/security/check_job_timeouts.sh
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
timeout-minutes: 30
if: github.event_name == 'pull_request' && github.event.action != 'closed'
permissions:
contents: read
@@ -150,8 +106,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Dependency Review
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5
@@ -164,28 +118,3 @@ jobs:
# conscious re-review of the license/provenance claim (backlog#1181).
allow-dependencies-licenses: pkg:cargo/rustfs-uring@0.1.0
comment-summary-in-pr: always
alert-on-failure:
name: Alert on scheduled failure
# dependency-review is deliberately excluded: it only runs on pull_request,
# so it can never contribute a failure to a scheduled run.
needs: [cargo-deny, workflow-pin-report]
# A scheduled cargo-deny failure usually means the dependency tree just
# matched a newly published advisory — the single most important signal this
# workflow produces, and until now it was only visible to whoever happened to
# open the Actions tab. Same ci-8 mechanism coverage.yml and
# e2e-replication-nightly.yml already use.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
+85 -89
View File
@@ -50,18 +50,12 @@ on:
- "**/*.svg"
- ".gitignore"
- ".dockerignore"
- "flake.lock"
schedule:
- cron: "0 1 * * 0" # Weekly on Sunday 01:00 UTC (staggered after the ci.yml midnight cron)
workflow_dispatch:
inputs:
build_docker:
# Advisory only. docker.yml triggers on workflow_run and its job-level
# condition requires the triggering event to be a tag push, so a manual
# dispatch of this workflow never produces images regardless of this
# value. Kept because the summary step reports it; wiring it up would
# mean teaching docker.yml's version parser a second event shape.
description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
description: "Build and push Docker images after binary build"
required: false
default: true
type: boolean
@@ -89,7 +83,6 @@ jobs:
build-check:
name: Build Strategy Check
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
build_type: ${{ steps.check.outputs.build_type }}
@@ -99,8 +92,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Determine build strategy
id: check
@@ -116,21 +107,13 @@ jobs:
# Determine build type based on trigger
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
# Tag push - preview, release, or prerelease
# Tag push - release or prerelease
should_build=true
tag_name="${GITHUB_REF#refs/tags/}"
version="${tag_name}"
# Preview tags publish a GitHub prerelease for validation, but
# must not update any latest channel.
if [[ "$tag_name" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Preview build detected: $tag_name"
elif [[ "$tag_name" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $tag_name (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
# Check if this is a prerelease
if [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
echo "🚀 Prerelease build detected: $tag_name"
@@ -173,7 +156,6 @@ jobs:
name: Prepare Platform Matrix
needs: build-check
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
matrix: ${{ steps.select.outputs.matrix }}
selected: ${{ steps.select.outputs.selected }}
@@ -181,14 +163,10 @@ jobs:
- name: Select target platforms
id: select
shell: bash
env:
# via env, not interpolation: a dispatch input is free-form text and
# would otherwise be pasted into the script for bash to evaluate.
RAW_PLATFORMS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}
run: |
set -euo pipefail
selected="$RAW_PLATFORMS"
selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
@@ -259,7 +237,6 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Setup Rust environment
@@ -268,17 +245,9 @@ jobs:
rust-version: stable
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}
# main only. A cache saved on refs/tags/X is scoped to that tag: no
# other tag, no main run and no PR can restore it, so every release
# cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
# six legs each) that nobody could read, evicting the hot lanes from
# the repo-wide 10GB quota. Tag builds still restore the main-scoped
# cache, since default-branch caches are readable from every ref.
# The one real cost: re-running a failed leg of the same tag no longer
# finds that tag's own warm cache and falls back to main's.
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
install-cross-tools: ${{ matrix.cross }}
install-test-tools: 'false'
- name: Download static console assets
shell: bash
@@ -725,14 +694,9 @@ jobs:
needs: [ build-check, build-rustfs ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Build completion summary
shell: bash
env:
# dispatch input via env: free-form text must not be pasted into the
# script for bash to evaluate.
INPUT_BUILD_DOCKER: ${{ github.event.inputs.build_docker }}
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
@@ -750,10 +714,6 @@ jobs:
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview artifacts are published in a GitHub prerelease"
echo "⏭️ Preview releases do not update latest channels"
;;
"development")
echo "🛠️ Development build artifacts have been uploaded to OSS dev directory"
echo "⚠️ This is a development build - not suitable for production use"
@@ -772,9 +732,7 @@ jobs:
echo ""
echo "🐳 Docker Images:"
if [[ "$BUILD_TYPE" == "preview" ]]; then
echo "⏭️ Preview tags do not publish Docker images"
elif [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
if [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
echo "⏭️ Docker image build was skipped (binary only build)"
elif [[ "$BUILD_STATUS" == "success" ]]; then
echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
@@ -782,13 +740,12 @@ jobs:
echo "❌ Docker image build will be skipped due to build failure"
fi
# Create GitHub Release for every valid release tag, including previews
# Create GitHub Release (only for tag pushes)
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
outputs:
@@ -798,7 +755,6 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Create GitHub Release
@@ -811,12 +767,9 @@ jobs:
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
# Determine release type for title
if [[ "$BUILD_TYPE" == "preview" ]]; then
RELEASE_TYPE="preview"
elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
@@ -830,34 +783,61 @@ jobs:
RELEASE_TYPE="release"
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
# Check if release already exists
if gh release view "$TAG" >/dev/null 2>&1; then
echo "Release $TAG already exists"
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
else
TITLE="RustFS $VERSION"
# Get release notes from tag message
RELEASE_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
RELEASE_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
RELEASE_NOTES="Release ${VERSION}"
fi
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
TITLE="RustFS $VERSION"
fi
# Create the release
PRERELEASE_FLAG=""
if [[ "$IS_PRERELEASE" == "true" ]]; then
PRERELEASE_FLAG="--prerelease"
fi
gh release create "$TAG" \
--title "$TITLE" \
--notes "$RELEASE_NOTES" \
$PRERELEASE_FLAG \
--draft
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
fi
./scripts/release/create_or_update_release.sh \
"$TAG" \
"$TARGET_COMMITISH" \
"$TITLE" \
"$IS_PRERELEASE"
echo "release_id=$RELEASE_ID" >> "$GITHUB_OUTPUT"
echo "release_url=$RELEASE_URL" >> "$GITHUB_OUTPUT"
echo "Created release: $RELEASE_URL"
# Prepare and upload release assets
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -940,10 +920,9 @@ jobs:
# the pointed-to version is a prerelease.
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
needs: [ build-check, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Update latest.json
env:
@@ -1001,34 +980,51 @@ jobs:
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Publish release
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Update release notes and publish
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
# Publish the release and correct its channel state on retries.
# Only a stable final release may become GitHub Latest.
if [[ "$BUILD_TYPE" == "release" ]]; then
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=false \
-f make_latest=true >/dev/null
# Determine release type
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=true \
-f make_latest=false >/dev/null
RELEASE_TYPE="release"
fi
# Get original release notes from tag
ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
ORIGINAL_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
ORIGINAL_NOTES="Release ${VERSION}"
fi
fi
# Publish the release (remove draft status)
gh release edit "$TAG" --draft=false
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
-231
View File
@@ -1,231 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Sole writer of the Rust dependency caches that ci.yml restores.
#
# Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
# concurrency group cancels in-progress runs on main pushes, and merges land far
# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
# 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
# Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
# so the writer lanes were saving nothing and every PR paid a cold restore —
# 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
#
# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
# superseded runs (which is correct — nobody needs test results for a commit
# that is already three merges behind) while the caches still get written.
#
# The group below deliberately does NOT cancel in progress. GitHub keeps at most
# one running plus one pending run per group, so a burst of merges collapses
# into "current run finishes, newest queued run follows" rather than a pile-up.
# That also bounds this workflow to one self-hosted runner at a time.
#
# Each job below owns exactly one shared-key and is the only place that sets
# cache-save-if to anything but 'false' for it; every lane in ci.yml reads.
# scripts/security/check_cache_save_if.sh keeps the declarations explicit.
#
# The builds are supersets of what the reading lanes compile, because a reader
# restores only what the writer saved. Feature resolution matters here: a lane
# built with e2e-test-hooks resolves dependency features differently, which
# changes -Cmetadata, so the plain build does not cover it. See
# rustfs/backlog#1600.
name: Cache Warm
on:
push:
branches: [ main ]
# Mirrors ci.yml's push paths-ignore: if a commit cannot change what ci.yml
# compiles, it cannot change what ci.yml needs restored either.
paths-ignore:
- "**.md"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "scripts/probe.sh"
- "LICENSE*"
- ".gitignore"
- ".dockerignore"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
workflow_dispatch:
inputs:
emit_timings:
description: >-
Also emit cargo --timings for the ci-dev build and upload it. Used to
decide whether sccache is worth adopting (rustfs/backlog#1601 gate).
required: false
default: false
type: boolean
permissions:
contents: read
concurrency:
group: cache-warm
cancel-in-progress: false
env:
CARGO_TERM_COLOR: always
jobs:
# Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'true'
install-build-packaging-tools: 'false'
# --all-targets covers the test binaries nextest builds, including
# e2e_test, which test-and-lint's own run excludes. The second build adds
# the e2e-test-hooks feature resolution that build-rustfs-debug-binary uses
# and that no lint lane enables.
# rustfs/backlog#1601 gate. sccache can only cache compilation units whose
# --emit includes link, so it covers workspace rlibs and nothing else:
# clippy is metadata-only, and the ~100 test binaries, the rustfs bin and
# every build script invoke the system linker. Before spending a bucket,
# credentials and a supply-chain boundary on it, measure how much of the
# build is actually rlib codegen.
#
# Read from the report: workspace lib codegen as a share of the build, and
# s3select-query's own rlib as a share. The plan adopts sccache only above
# 50% and 25% respectively; if linking dominates instead, the answer is
# mold/lld plus split-debuginfo, which is exactly the part sccache cannot
# touch. Off by default — this doubles the ci-dev build.
- name: Build ci-dev superset (with --timings)
if: inputs.emit_timings
env:
CARGO_BUILD_JOBS: "2"
run: cargo build --workspace --all-targets --timings
- name: Upload cargo timings report
if: inputs.emit_timings
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: cargo-timings-ci-dev
path: target/cargo-timings/
retention-days: 30
if-no-files-found: error
- name: Build ci-dev superset
env:
# Same limit ci.yml puts on its nextest step: this builds the same
# ~100 workspace test binaries, and three concurrent links saturate the
# self-hosted runner's overlay I/O and can wedge Cargo (#5394).
CARGO_BUILD_JOBS: "2"
run: |
cargo build --workspace --all-targets
cargo build -p rustfs --bins --features e2e-test-hooks
# Readers: test-and-lint-rio-v2, build-rustfs-debug-binary-rio-v2.
warm-ci-feat-rio:
name: Warm ci-feat-rio
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-rio
cache-save-if: 'true'
install-build-packaging-tools: 'false'
- name: Build ci-feat-rio superset
run: |
cargo build -p rustfs -p rustfs-ecstore --all-targets --features rio-v2
cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
# Readers: the swift and sftp legs of test-and-lint-protocols. Built in
# sequence rather than as `--features swift,sftp`, which is a combination no
# lane actually compiles; running both leaves the union in target/.
warm-ci-feat-proto:
name: Warm ci-feat-proto
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-proto
cache-save-if: 'true'
install-build-packaging-tools: 'false'
- name: Build ci-feat-proto superset
run: |
cargo build -p rustfs -p rustfs-protocols --all-targets --features swift
cargo build -p rustfs -p rustfs-protocols --all-targets --features sftp
# Reader: uring-integration. Runs on ubuntu-latest to match it: rust-cache's
# key covers runner.os and arch but not the runner label or image, so a cache
# written on sm-standard-4 would be restored by the hosted runner as if it
# belonged to it.
warm-ci-uring:
name: Warm ci-uring
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-uring
cache-save-if: 'true'
install-build-packaging-tools: 'false'
- name: Install build dependencies
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
- name: Build ci-uring superset
run: cargo build -p rustfs-ecstore --all-targets
+8 -79
View File
@@ -12,24 +12,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Companion to ci.yml for required status checks.
# Companion to ci.yml for the required "Test and Lint" status check.
#
# ci.yml skips docs-only pull requests via paths-ignore, but the branch ruleset
# requires a check named "Test and Lint" — without this workflow a docs-only PR
# would wait on it forever. This workflow triggers on exactly the paths ci.yml
# ignores and reports success under the same job name. Mixed PRs trigger both
# workflows and the real check still gates: a required check with any failing
# run blocks the merge.
# ci.yml skips docs-only pull requests via paths-ignore, but the branch
# ruleset requires a check named "Test and Lint" — without this workflow a
# docs-only PR would wait on that check forever. This workflow triggers on
# exactly the paths ci.yml ignores and reports an instant success under the
# same job name. Mixed PRs trigger both workflows and the real check still
# gates: a required check with any failing run blocks the merge.
# https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks
#
# "Quick Checks" is mirrored here ahead of the ruleset change that will make it
# required too (rustfs/backlog#1599). Until that change lands this job is
# inert; mirroring it first is what lets the ruleset change happen without
# stranding docs-only PRs on a check nobody reports.
#
# Keep the paths list below in sync with the pull_request paths-ignore list
# in ci.yml, and keep the quick-checks steps below byte-identical to the
# quick-checks job in ci.yml.
# in ci.yml.
name: Continuous Integration (docs only)
@@ -53,82 +47,17 @@ on:
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- "flake.lock"
permissions:
contents: read
jobs:
# Deliberately NOT a bare `echo`. Once "Quick Checks" becomes a required
# check, ci.yml gates every expensive job behind it, so a mixed PR reports
# two check runs with this name: the real one (45-51s) and this companion.
# GitHub has no written contract for how it picks between same-named
# required check runs ("latest wins" vs "any failure blocks"), so instead of
# relying on ordering we make both runs execute the same commands against
# the same merge ref — their conclusions are then necessarily identical and
# the choice does not matter. Keep these steps byte-identical to the
# quick-checks job in ci.yml (a guard script that asserts this, and the paths
# sync below, is tracked in rustfs/backlog#1603).
#
# For a genuinely docs-only PR this adds no strictness (no code changed, so
# fmt and the guards always pass) and costs ~50s of ubuntu-latest.
quick-checks:
name: Quick Checks
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
run: sudo apt-get update && sudo apt-get install -y ripgrep
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
components: rustfmt
- name: Check code formatting
run: cargo fmt --all --check
- name: Check unsafe code allowances
run: ./scripts/check_unsafe_code_allowances.sh
- name: Check layered dependencies
run: ./scripts/check_layer_dependencies.sh
- name: Check architecture migration rules
run: ./scripts/check_architecture_migration_rules.sh
- name: Check tokio io-uring feature guard
run: ./scripts/check_no_tokio_io_uring.sh
- name: Check extension schema boundaries
run: ./scripts/check_extension_schema_boundaries.sh
- name: Check body-cache whitelist guard
run: ./scripts/check_body_cache_whitelist.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
- name: Check CI paths stay in sync
run: ./scripts/check_ci_paths_sync.sh
- name: Check io_uring lane --lib precondition
run: ./scripts/check_uring_lane_lib_only.sh
test-and-lint:
name: Test and Lint
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Docs-only PRs skip the full code CI, but they are exactly where a
# planning-type document could be slipped in (git add -f bypasses
+60 -181
View File
@@ -33,7 +33,6 @@ on:
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- "flake.lock"
pull_request:
types: [ opened, synchronize, reopened, closed ]
branches: [ main ]
@@ -55,7 +54,6 @@ on:
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- "flake.lock"
merge_group:
types: [ checks_requested ]
schedule:
@@ -83,7 +81,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -92,20 +89,13 @@ jobs:
name: Typos
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Typos check with custom config file
uses: crate-ci/typos@37bb98842b0d8c4ffebdb75301a13db0267cef89 # master
# Fast, compile-free checks that fail early so contributors get feedback in
# ~1 minute instead of waiting for the full test job.
#
# These steps are mirrored byte-for-byte in ci-docs-only.yml so that a mixed
# PR, which reports two check runs named "Quick Checks", cannot get one red
# and one green. Edit both jobs together.
quick-checks:
name: Quick Checks
if: github.event_name != 'pull_request' || github.event.action != 'closed'
@@ -114,8 +104,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
run: sudo apt-get update && sudo apt-get install -y ripgrep
@@ -149,48 +137,24 @@ jobs:
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
- name: Check CI paths stay in sync
run: ./scripts/check_ci_paths_sync.sh
- name: Check io_uring lane --lib precondition
run: ./scripts/check_uring_lane_lib_only.sh
test-and-lint:
name: Test and Lint
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 90
# Both lines are required. Job-level `permissions` replaces the workflow
# block rather than merging with it, so declaring only `actions: write`
# would drop `contents: read` and break this job's checkout and the
# repo-token the setup action hands to setup-protoc.
permissions:
contents: read
actions: write
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
# This job's token can cancel runs and delete Actions caches. Checkout
# otherwise writes it into .git/config, where a PR's own build.rs or
# proc-macro could read it back out.
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
# Every lane in this workflow reads its cache and none writes it.
# cache-warm.yml is the sole writer for all four keys: this workflow
# cancels superseded runs on main, and a cancelled run never reaches
# rust-cache's post step, so writing from here saved nothing (12 of 15
# consecutive main-push runs were cancelled). See rustfs/backlog#1600.
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-test
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Prepare test evidence
run: |
@@ -205,14 +169,8 @@ jobs:
# Clippy runs before the test pass: lint failures are the most common
# CI-only breakage and should surface in minutes, not after 20+ minutes
# of tests.
# Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
# experiment, since --all-targets is check-only for workspace members and
# never links the ~100 test binaries the limit exists to throttle.
- name: Run clippy lints
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo clippy --all-targets -- -D warnings
run: cargo clippy --all-targets -- -D warnings
- name: Run nextest tests
env:
@@ -221,8 +179,33 @@ jobs:
CARGO_BUILD_JOBS: "2"
run: |
mkdir -p artifacts/test-and-lint
./scripts/ci/resource_sampler.sh start nextest
trap './scripts/ci/resource_sampler.sh stop' EXIT
# Evidence sampler for issue #5394: the post-mortem pgrep below runs
# only after `timeout` has already TERM'd the whole cargo process
# group, so it cannot name a wedged process. Sample system and
# process state every 60s instead; the last samples before the
# timeout show what was stuck (rustc, linker, build script, memory
# pressure, ...). The log rides along in the existing artifact.
(
while true; do
{
echo "=== $(date --utc --iso-8601=seconds)"
echo "--- load"; cat /proc/loadavg
echo "--- psi"; grep -H . /proc/pressure/* 2>/dev/null || true
echo "--- mem"; free -m
echo "--- disk"; df -h / /home/runner 2>/dev/null || df -h /
echo "--- top-rss"
ps -eo pid,ppid,stat,etime,rss,pcpu,args --sort=-rss | head -15
echo "--- build/test processes"
ps -eo pid,ppid,stat,etime,rss,pcpu,args | grep -E '[c]argo|[r]ustc|[n]extest|[c]ollect2|rust-ll[d]|[b]uild-script|deps[/]' || true
echo "--- d-state (uninterruptible IO)"
ps -eo pid,stat,etime,args | awk 'NR > 1 && $2 ~ /D/' || true
echo
} >> artifacts/test-and-lint/sampler.log 2>&1 || true
sleep 60
done
) &
sampler_pid=$!
trap 'kill "${sampler_pid}" 2>/dev/null || true' EXIT
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 75m \
cargo nextest run --profile ci --all --exclude e2e_test \
@@ -294,48 +277,6 @@ jobs:
- name: Run rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh
# Early stop. Once this job has failed the PR cannot merge, so the sibling
# lanes are burning runners on a result nobody can act on: on run
# 30674613104 three lanes had already failed while Test and Lint and the
# rio-v2 variant kept going past 70 minutes.
#
# Only this job may cancel. The lanes that are NOT required checks
# (protocols, ILM, e2e, s3-tests) must never hold that power: a flake in
# one of them would turn the required "Test and Lint" into `cancelled`,
# which blocks the merge. Today a maintainer can merge with sftp red, and
# that has to stay true.
#
# These steps run last so the `if: always()` artifact upload above still
# captures logs and diagnostics before the run goes away.
- name: Annotate early-stop reason
if: failure() && github.event_name == 'pull_request'
run: |
echo "## CI early-stop" >> "$GITHUB_STEP_SUMMARY"
echo "Job \`${GITHUB_JOB}\` (Test and Lint) failed; cancelling run ${GITHUB_RUN_ID} to free runners." >> "$GITHUB_STEP_SUMMARY"
echo "Sibling jobs showing **cancelled** were stopped by this job, not by their own failure." >> "$GITHUB_STEP_SUMMARY"
# curl rather than `gh`: every existing `gh` call in this repo runs on
# ubuntu-latest, and the sm-standard-* images are custom and trimmed (they
# ship no C toolchain, see the e2e job below), so `gh` is not known to
# exist here.
#
# Fork PRs are excluded explicitly instead of relying on the error path:
# their GITHUB_TOKEN is forced read-only and job-level permissions cannot
# raise it, so the call would always 403. Skipping keeps their logs clean.
- name: Cancel run on failure (same-repo PR only)
if: >-
failure() && github.event_name == 'pull_request'
&& github.event.pull_request.head.repo.full_name == github.repository
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
curl -fsS -X POST \
-H "Authorization: Bearer ${GH_TOKEN}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/cancel" || true
# Dedicated serial lane for the ILM / lifecycle integration tests. These tests
# drive the object layer through process-global singletons (the GLOBAL_ENV
# ECStore, the global tier-config manager, background-expiry workers) and bind
@@ -349,7 +290,6 @@ jobs:
test-ilm-integration-serial:
name: ILM Integration (serial)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 45
env:
@@ -357,16 +297,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-ilm-serial
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
@@ -389,7 +327,6 @@ jobs:
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 60
env:
@@ -397,16 +334,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-rio
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-test-rio-v2
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run rio-v2 clippy lints
run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
@@ -419,17 +354,10 @@ jobs:
test-and-lint-protocols:
name: "Test and Lint (${{ matrix.features.name }})"
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 60
strategy:
# On a PR, one failing protocol leg is enough to know the PR is not ready,
# so stop the sibling leg instead of paying another ~40 minutes for it.
# Everywhere else (main pushes, the merge queue, the weekly schedule) keep
# the full signal: there we want to know whether swift AND sftp are broken,
# not just whichever failed first. This is the only part of the early-stop
# work that also covers fork PRs, since it needs no token.
fail-fast: ${{ github.event_name == 'pull_request' }}
fail-fast: false
matrix:
features:
- name: swift
@@ -441,16 +369,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-proto
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-test-${{ matrix.features.name }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run clippy with ${{ matrix.features.name }}
run: |
@@ -463,7 +389,6 @@ jobs:
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 30
env:
@@ -471,16 +396,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-rustfs-debug-binary
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build debug binary
run: cargo build -p rustfs --bins --features e2e-test-hooks
@@ -496,7 +419,6 @@ jobs:
build-rustfs-debug-binary-rio-v2:
name: Build RustFS Debug Binary (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 30
env:
@@ -504,16 +426,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-rio
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-rustfs-debug-binary-rio-v2
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build debug binary with rio-v2
run: cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
@@ -528,14 +448,6 @@ jobs:
uring-integration:
name: io_uring Integration (real)
# The pull_request trigger includes `closed` purely so the concurrency
# group cancels in-flight runs of a closed PR; every other job opts out of
# that run with this guard (or is skipped through its `needs` chain). This
# job had neither, so each closed/merged PR really ran the whole io_uring
# suite (measured 4m17s / 7m19s / 7m31s on runs 30678272341 / 30678117601 /
# 30662728539) and kept the cancellation run in progress for minutes.
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
# GitHub-hosted ubuntu-latest runs a recent kernel with io_uring and, unlike
# a container, applies no seccomp filter that would block io_uring_setup — so
# the probe succeeds and the tests exercise the real UringBackend/FdCache/
@@ -546,24 +458,17 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
# Keeps its own key rather than joining ci-dev. rust-cache's key is
# built from runner.os/arch plus rustc and lockfile fingerprints — it
# does NOT include the runner label or image. ubuntu-latest and
# sm-standard-4 are therefore indistinguishable to it, so sharing a key
# would let two different system images overwrite each other's
# artifacts, and would make a 2-core hosted runner unpack ci-dev's ~3GB
# instead of this lane's ~1.3GB. cache-warm.yml warms this key on
# ubuntu-latest for the same reason.
cache-shared-key: ci-uring
cache-save-if: 'false'
install-build-packaging-tools: 'false'
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install build dependencies
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
# ext4 supports O_DIRECT; the runner's default TMPDIR may sit on tmpfs or
# overlayfs, where open(O_DIRECT) returns EINVAL/EOPNOTSUPP and the native
@@ -590,17 +495,7 @@ jobs:
RUSTFS_IO_URING_READ_ENABLE: "true"
RUSTFS_URING_TESTS_MUST_RUN: "1"
TMPDIR: /mnt/rustfs-odirect
# --lib narrows what gets compiled, not what gets run: every selected
# test lives in the lib target. The 7 integration binaries under
# crates/ecstore/tests/ each reported "running 0 tests" here, so they
# were compiled and linked for nothing.
#
# The `uring_` filter must stay exactly as it is. libtest matches on
# substring, so it also selects names containing `during_` — 6 of the 18
# selected tests are such incidental matches. Narrowing the filter to
# `io_uring` would silently drop them, which is a coverage change.
# scripts/check_uring_lane_lib_only.sh guards the --lib precondition.
run: cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture
run: cargo test -p rustfs-ecstore uring_ -- --test-threads=1 --nocapture
e2e-tests:
name: End-to-End Tests
@@ -610,8 +505,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Full setup with dependency caching: the smoke-suite step below
# compiles the e2e_test crate, which pulls in most of the workspace.
@@ -620,9 +513,9 @@ jobs:
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-e2e
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
@@ -696,16 +589,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-e2e
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
@@ -741,8 +632,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Clean up previous test run
run: |
@@ -797,8 +686,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -859,20 +746,12 @@ jobs:
# evaluates ILM within ~2s of the due time, well inside the poll window.
s3-lifecycle-behavior-tests:
name: S3 Lifecycle Behavior Tests
# Also gated on e2e-tests, matching s3-implemented-tests: when the e2e smoke
# suite is already red this lane cannot tell us anything new, and it holds a
# sm-standard-4 for up to 30 minutes doing so. Both lanes only download the
# prebuilt debug binary (no cargo build), and s3-implemented-tests — which
# already waits on e2e-tests — finishes later anyway, so a green PR's total
# wall clock is unchanged.
needs: [ build-rustfs-debug-binary, e2e-tests ]
needs: [ build-rustfs-debug-binary ]
runs-on: sm-standard-4
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+4 -23
View File
@@ -22,18 +22,11 @@ on:
issue_comment:
types: [created, edited]
# Least privilege at the top, widened per job below. This workflow runs on
# pull_request_target and issue_comment, so it holds full secrets on every fork
# PR and on any comment anyone writes — the one place in this repository where a
# compromised action would be handed a repo-write token. It does not check out
# or execute PR code, so there is no pwn-request path today, but the blast
# radius should not depend on that staying true.
#
# contents: write in particular was never used: the signature records are
# written to rustfs/cla through the scoped app token created below, and nothing
# here writes to this repository's contents.
permissions:
contents: read
contents: write
pull-requests: write
issues: write
checks: write
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.event.issue.number || github.ref }}
@@ -43,26 +36,14 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request_target' && github.event.action == 'closed'
# Echoes one line; the run exists only so the concurrency group cancels the
# in-flight run of a closed PR.
permissions: {}
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
cla:
if: ${{ (github.event_name != 'issue_comment' || github.event.issue.pull_request) && (github.event_name != 'pull_request_target' || github.event.action != 'closed') }}
# checks: write reports the merge-queue check run; pull-requests and issues
# let cla-bot comment and label. contents stays read — see the note above.
permissions:
contents: read
checks: write
issues: write
pull-requests: write
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Report CLA result for merge queue
if: github.event_name == 'merge_group'
+1 -5
View File
@@ -62,16 +62,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-coverage
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: 'false'
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
@@ -118,8 +116,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
+21 -43
View File
@@ -82,10 +82,8 @@ jobs:
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main' &&
!contains(github.event.workflow_run.head_branch, '-preview'))
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
@@ -98,18 +96,11 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
@@ -210,9 +201,9 @@ jobs:
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
input_version="${{ github.event.inputs.version }}"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_push="${{ github.event.inputs.push_images }}"
should_build=true
# Get short SHA
@@ -220,7 +211,7 @@ jobs:
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🔧 Force rebuild: ${{ github.event.inputs.force_rebuild }}"
echo " 🚀 Push images: $should_push"
case "$input_version" in
@@ -229,13 +220,6 @@ jobs:
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
build_type="preview"
is_prerelease=true
should_build=false
should_push=false
echo "⏭️ Preview tags do not publish Docker images"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
@@ -306,8 +290,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
@@ -343,28 +325,32 @@ jobs:
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
# Convert version format for Dockerfile compatibility
case "$VERSION" in
"latest")
# For stable latest, use RELEASE=latest + release CHANNEL
DOCKER_RELEASE="latest"
DOCKER_CHANNEL="release"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
DOCKER_CHANNEL="release"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
DOCKER_CHANNEL="release"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "docker_channel=$DOCKER_CHANNEL" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
echo " - Docker CHANNEL: $DOCKER_CHANNEL"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
@@ -418,24 +404,18 @@ jobs:
push: ${{ needs.build-check.outputs.should_push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
cache-from: |
type=gha,scope=docker-${{ matrix.variant }}
cache-to: |
type=gha,mode=max,scope=docker-${{ matrix.variant }}
build-args: |
BUILDTIME=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
VERSION=${{ needs.build-check.outputs.version }}
BUILD_TYPE=${{ needs.build-check.outputs.build_type }}
REVISION=${{ github.sha }}
RELEASE=${{ steps.meta.outputs.docker_release }}
CHANNEL=${{ steps.meta.outputs.docker_channel }}
BUILDKIT_INLINE_CACHE=1
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
@@ -451,7 +431,6 @@ jobs:
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
@@ -506,7 +485,6 @@ jobs:
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
@@ -64,16 +64,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-e2e-repl
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: 'false'
# awscurl lets the STS dual-node test actually exercise its path. Without
# it the test skips gracefully with a visible log line
@@ -126,8 +124,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
-11
View File
@@ -45,13 +45,6 @@
# The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker
# via DEPLOY_MODE=binary and defers all pip setup to run.sh's self-bootstrap.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: e2e-s3tests
on:
@@ -142,8 +135,6 @@ jobs:
TEST_MODE: ${{ matrix.test-mode }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Provision Python explicitly rather than trusting the runner image to
# ship a working pip (ci-1: a bare python3 without pip is what broke the
@@ -363,8 +354,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
+1 -16
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Fuzz
on:
@@ -66,7 +59,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -87,14 +79,13 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: nightly
cache-shared-key: fuzz-${{ hashFiles('fuzz/Cargo.lock') }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
- name: Install cargo-fuzz
@@ -154,8 +145,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -211,8 +200,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -260,8 +247,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
+9 -35
View File
@@ -32,14 +32,12 @@ permissions:
jobs:
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
github.event_name == 'workflow_dispatch' ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
contains(github.event.workflow_run.head_branch, '.')
)
outputs:
@@ -50,26 +48,16 @@ jobs:
steps:
- name: Checkout helm chart repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
RAW="${{ github.event.inputs.version }}"
else
RAW="$RAW_BRANCH"
RAW="${{ github.event.workflow_run.head_branch }}"
fi
case "$RAW" in
@@ -84,13 +72,10 @@ jobs:
./scripts/helm_chart_version.sh "$RAW_TAG"
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
@@ -115,7 +100,6 @@ jobs:
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
@@ -123,8 +107,6 @@ jobs:
- name: Checkout helm package repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
@@ -140,19 +122,11 @@ jobs:
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git config --global user.name "${{ secrets.USERNAME }}"
git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
git push origin main
-8
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: "issue-translator"
on:
issue_comment:
@@ -33,7 +26,6 @@ permissions:
jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: usthe/issues-translate-action@b41f55ddc81d7d54bd542a4f289fe28ec081898e # v2.7
with:
+1 -9
View File
@@ -23,13 +23,6 @@
# Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
# unlike the self-hosted fleet, whose pods drift in Docker/pip availability
# (see the infra note in e2e-s3tests.yml). Nightly + manual only.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: minio-interop
on:
@@ -58,14 +51,13 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-minio-interop
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Generate real MinIO fixtures via Docker
-11
View File
@@ -45,13 +45,6 @@
# docker-capable self-hosted `dind-sm-standard-2` label was the alternative but
# has fewer cores and reintroduces fleet-state risk for no reliability gain.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: mint
on:
@@ -125,8 +118,6 @@ jobs:
timeout-minutes: 120
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Enable buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
@@ -272,8 +263,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
+2 -9
View File
@@ -19,12 +19,9 @@ on:
schedule:
- cron: '0 5 * * 0' # Weekly on Sunday 05:00 UTC (staggered after the midnight ci/build crons)
# GITHUB_TOKEN only needs to read the repository here: the branch push and the
# pull request are both created by update-flake-lock using the
# FLAKE_UPDATE_TOKEN PAT below, not by this token. Leaving write on it hands a
# repo-write credential to an unattended weekly job that does not use it.
permissions:
contents: read
contents: write
pull-requests: write
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -40,10 +37,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
# persist-credentials-exempt: update-flake-lock pushes the branch and opens
# the PR. It passes FLAKE_UPDATE_TOKEN to create-pull-request itself rather
# than reusing .git/config, but that is unverified — exempt until a
# workflow_dispatch run confirms it (rustfs/backlog#1602).
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@629b284231c2a82554b724e357e47fc6020833c8 # v3
-10
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Nix CI
on:
@@ -53,7 +46,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -71,8 +63,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@4eea0b33e3d1f02ecfe37cf16e7204c424009606 # v3.21.0
+4 -18
View File
@@ -22,13 +22,6 @@
# a deliberate correctness cost (e.g. the #4221 fsync durability fix) is
# recorded but does not block (rustfs/backlog#935 correction 1).
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Performance A/B
on:
@@ -99,8 +92,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
@@ -108,6 +99,7 @@ jobs:
rust-version: stable
cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build release rustfs
run: cargo build --release --bin rustfs
@@ -150,7 +142,6 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
fetch-depth: 0 # baseline is built from origin/main
- name: Setup Rust environment
@@ -159,6 +150,7 @@ jobs:
rust-version: stable
cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install warp
run: |
@@ -170,15 +162,13 @@ jobs:
- name: Decide exemption
id: exempt
env:
INPUT_ALLOW_REGRESSION: ${{ github.event.inputs.allow_regression }}
run: |
allow="false"
if [[ "${{ github.event_name }}" == "pull_request" ]] \
&& ${{ contains(github.event.pull_request.labels.*.name, 'perf-deliberate-tradeoff') }}; then
allow="true"
fi
if [[ "$INPUT_ALLOW_REGRESSION" == "true" ]]; then
if [[ "${{ github.event.inputs.allow_regression }}" == "true" ]]; then
allow="true"
fi
echo "allow_regression=$allow" >> "$GITHUB_OUTPUT"
@@ -234,8 +224,6 @@ jobs:
- name: Run warp A/B and gate
id: ab
env:
INPUT_DURATION: ${{ github.event.inputs.duration }}
run: |
set -euo pipefail
# Budget note: with perf-3's cached baseline the nightly does no source
@@ -246,7 +234,7 @@ jobs:
# budget, which the rig's previous 60s health poll undershot (the first
# two nightly failures). perf-6 recalibrates these once the noise study
# lands.
duration="${INPUT_DURATION:-12s}"
duration="${{ github.event.inputs.duration || '12s' }}"
baseline_sha="${{ steps.commits.outputs.baseline_sha }}"
candidate_sha="${{ steps.commits.outputs.candidate_sha }}"
baseline_hit="${{ steps.baseline_cache.outputs.cache-hit }}"
@@ -397,8 +385,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
@@ -24,13 +24,6 @@
# The run itself is expected to end red (the forced failure); only the
# alert-on-failure job result matters.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Schedule Failure Alert Drill
on:
@@ -63,8 +56,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
-8
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: "Mark stale issues"
on:
schedule:
@@ -27,7 +20,6 @@ on:
jobs:
stale:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/stale@5bef64f19d7facfb25b37b414482c7164d639639 # v9
with:
-1
View File
@@ -15,7 +15,6 @@ concurrency:
jobs:
update:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: overtrue/repo-visuals-action@72f34d24769ff5d341956da2f23952594ef2f1e2 # v1.3.0
with:
+19 -64
View File
@@ -105,78 +105,33 @@ CI) fails the build if anything is committed under `docs/superpowers/`, even via
## Verification Before PR
Convert changes into independently verifiable outcomes. This section controls
agent-run local validation; preparing a commit or PR does not by itself require
the broadest gate. Inspect only the final task-owned diff, classify it by
behavioral impact rather than line count or path alone, and run the smallest
set of checks that provides meaningful coverage. Do not let unrelated
worktree changes or a generic contributor checklist expand the scope.
Non-exempt changes must also pass Adversarial Validation (next section) before
the checks below count as completion.
Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
Non-exempt changes must also pass Adversarial Validation (next section) before the checks below count as completion.
### Validation floor
For code changes, run and pass the following before opening a PR:
- Every change that is not documentation-only must finish with
`cargo fmt --all --check` passing. An umbrella gate that runs this exact
check satisfies the requirement; do not run it twice. Use `cargo fmt --all`
only when formatting needs to be fixed. Run the configured formatter or
validator for other changed languages when one exists.
- Documentation-only or instruction-only means all task-owned changes are
prose or documentation assets and cannot affect runtime, builds, CI,
dependencies, generated code, or tests. Run `git diff --check` and any
relevant documentation guard, but skip Cargo formatting, compilation,
Clippy, tests, `make pre-commit`, and `make pre-pr`.
- Behavior changes require relevant existing or new tests. Prefer the most
focused test or affected package. A passing targeted test can also provide
sufficient compilation coverage when it builds every changed target and
feature involved; do not add a redundant `cargo check` in that case.
- `cargo check` supplements compilation coverage; it never substitutes for a
behavioral test. If a relevant test cannot reasonably be added or run, use
the narrowest compilation check and report the reason and remaining risk.
```bash
make pre-pr
```
### Validation tiers
Before committing code changes, prefer focused verification for the touched
surface and use the faster local gate when a broad smoke check is needed:
1. **Documentation/instruction-only:** Apply the exemption above. Run a guard
such as `make doc-paths-check` only when it is relevant to the edited text.
2. **Non-behavioral source change:** For comments, formatting, or another
demonstrably non-executable change, run the formatting floor. Compilation,
Clippy, and tests may be skipped only when the edit cannot affect
compilation or runtime behavior; run targeted doctests if executable
documentation examples changed.
3. **Localized or bounded behavior change:** Run the formatting floor and the
narrowest relevant tests. Add package-scoped `cargo check` or Clippy only
for changed targets, features, APIs, error handling, async behavior, or
control flow not already covered. When several crates are affected but the
dependency set is identifiable, validate those packages and known
dependents instead of the whole workspace. Use `make pre-commit` only when
a repository-wide fast gate adds useful confidence beyond those checks.
4. **Broad or high-risk change:** Run `make pre-pr` only when targeted coverage
cannot bound the impact, including:
- dependency, feature, build-script, procedural-macro, code-generation,
toolchain, or CI changes that alter compilation or the test matrix;
- cross-crate public APIs, shared foundational code, or broad refactors with
an unbounded dependent set;
- locking, storage durability or formats, erasure coding, replication,
RPC/protocol compatibility, IAM/KMS/auth, cryptography, or other
security-sensitive behavior;
- a targeted check that reveals wider impact, an explicit user request, or
a release policy that requires the full gate.
```bash
make pre-commit
```
Documentation-only and non-behavioral classifications take precedence over
path-based triggers. A small diff can still be high-risk, while a CI comment,
manifest comment, or release-note edit does not require full validation.
For migration batches, do not run the full `make pre-pr` gate before every
intermediate commit. Use focused tests and `make pre-commit` during
development, then reserve `make pre-pr` for the final PR-ready branch.
`make pre-pr` includes `make pre-commit` coverage. Never run both for the same
unchanged diff, and do not repeat equivalent checks during PR preparation or
because a local hook already ran them. Rerun only checks whose scope is affected
by later edits. Full workspace checks do not replace a relevant integration or
E2E test for changed behavior; run that focused test when required and
available, or report why it was not run and the remaining risk.
Before pushing code changes, make sure formatting is clean:
If `make` is unavailable, run the equivalent checks defined under
`.config/make/`. At handoff, list the checks actually run, checks intentionally
skipped, and the reason for the selected tier.
- Run `cargo fmt --all`.
- Run `cargo fmt --all --check` and ensure no files are modified unexpectedly.
If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any locally installed git pre-commit hooks may still run on commit unless explicitly skipped.
After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
Do not open a PR with code changes when the required checks fail.
Make a failing check pass by fixing the cause, never by weakening the gate:
Generated
+35 -135
View File
@@ -1528,7 +1528,7 @@ version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array 0.14.7",
"generic-array 0.14.9",
]
[[package]]
@@ -1547,7 +1547,7 @@ version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a8894febbff9f758034a5b8e12d87918f56dfc64a8e1fe757d65e29041538d93"
dependencies = [
"generic-array 0.14.7",
"generic-array 0.14.9",
]
[[package]]
@@ -1598,7 +1598,7 @@ version = "3.9.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6dee98b0db6a962de883bf5d20362dee4d7ca0d12fe39a7c6c73c844e1cd7c1f"
dependencies = [
"darling 0.20.11",
"darling 0.23.0",
"ident_case",
"prettyplease",
"proc-macro2",
@@ -1870,7 +1870,7 @@ version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad"
dependencies = [
"crypto-common 0.1.7",
"crypto-common 0.1.6",
"inout 0.1.4",
]
@@ -2328,7 +2328,7 @@ version = "0.5.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76"
dependencies = [
"generic-array 0.14.7",
"generic-array 0.14.9",
"rand_core 0.6.4",
"subtle",
"zeroize",
@@ -2353,11 +2353,11 @@ dependencies = [
[[package]]
name = "crypto-common"
version = "0.1.7"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
checksum = "1bfb12502f3fc46cca1bb51ac28df9d618d813cdc3d2f25b9fe775a34af26bb3"
dependencies = [
"generic-array 0.14.7",
"generic-array 0.14.9",
"typenum",
]
@@ -3554,7 +3554,7 @@ checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer 0.10.4",
"const-oid 0.9.6",
"crypto-common 0.1.7",
"crypto-common 0.1.6",
"subtle",
]
@@ -3672,7 +3672,6 @@ dependencies = [
"flate2",
"futures",
"hex",
"hotpath",
"http 1.5.0",
"http-body-util",
"hyper",
@@ -3814,7 +3813,7 @@ dependencies = [
"crypto-bigint 0.5.5",
"digest 0.10.7",
"ff 0.13.1",
"generic-array 0.14.7",
"generic-array 0.14.9",
"group 0.13.0",
"hkdf 0.12.4",
"pem-rfc7468 0.7.0",
@@ -4259,9 +4258,9 @@ dependencies = [
[[package]]
name = "generic-array"
version = "0.14.7"
version = "0.14.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
checksum = "4bb6743198531e02858aeaea5398fcc883e71851fcbcb5a2f773e2fb6cb1edf2"
dependencies = [
"typenum",
"version_check",
@@ -4274,7 +4273,7 @@ version = "1.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab4e5aa225bc56696909483320f0ff9b600f1a971b52e07a17d70f3d9b43254b"
dependencies = [
"generic-array 0.14.7",
"generic-array 0.14.9",
"rustversion",
"typenum",
]
@@ -4373,9 +4372,9 @@ dependencies = [
[[package]]
name = "google-cloud-auth"
version = "1.15.0"
version = "1.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f54aab44c16b8463ae11b165a87c3d484780231f157bb1ed65843d591beb5abd"
checksum = "a3494870d06f3cbbb3561ada6f234982549e3a2fb31e719ef258e6eadb9ae09a"
dependencies = [
"async-trait",
"aws-lc-rs",
@@ -4402,9 +4401,9 @@ dependencies = [
[[package]]
name = "google-cloud-gax"
version = "1.13.0"
version = "1.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9a46dd0fd026bbc4a5d84e6ab0c941cee6e3b057976a0bb107fdb5238ce598f"
checksum = "3103a4a9013f1aed573ca56e19a9680b0211643a99ea85caf524b397d6be8be3"
dependencies = [
"bytes",
"futures",
@@ -4421,9 +4420,9 @@ dependencies = [
[[package]]
name = "google-cloud-gax-internal"
version = "0.7.16"
version = "0.7.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb04c54317ace06d489213f761797240b3046142a9b7ce6b9a82a9d134e193d1"
checksum = "c0df265fba091ed7e00ecd0755009423310163f8b52820f007b6b4d97f4c6617"
dependencies = [
"bytes",
"futures",
@@ -4525,9 +4524,9 @@ dependencies = [
[[package]]
name = "google-cloud-storage"
version = "1.17.0"
version = "1.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9227f65175fa91a6e41f246797917697efdadfe09dd8ea84ad8b737a71efbd28"
checksum = "dc4b1d78c88db5c2530b12461e373a7d0d3a6caa3f6c1fc14e5d824cf2aeb307"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -4578,9 +4577,9 @@ dependencies = [
[[package]]
name = "google-cloud-wkt"
version = "1.7.0"
version = "1.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7fccf98cfd5481a5f5a285181ab0c62123d7d47cd2bb7299448440649349e4e7"
checksum = "46df1fcc3ab69164af3f4199ed21f45b5dbc56d9f03211eb4fa20116d442364b"
dependencies = [
"base64 0.22.1",
"bytes",
@@ -4917,31 +4916,19 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "66750a77f4f6b408a148be5102ef1f3ba7172def7ee92b1cfc75d9f7a3870453"
dependencies = [
"arc-swap",
"async-channel",
"async-trait",
"cfg-if",
"crossbeam-channel",
"flate2",
"futures-channel",
"futures-util",
"hdrhistogram",
"hotpath-macros",
"hotpath-meta",
"http 1.5.0",
"libc",
"object 0.36.7",
"parking_lot",
"pin-project-lite",
"prettytable-rs",
"quanta",
"regex",
"reqwest",
"reqwest-middleware",
"rustc-demangle",
"serde",
"serde_json",
"tiny_http",
"tokio",
]
[[package]]
@@ -4955,21 +4942,6 @@ dependencies = [
"syn 2.0.119",
]
[[package]]
name = "hotpath-macros-meta"
version = "0.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "21f2f70b29b6f42311acd2fb0b2f91a34d9a573c76fb8a7f51970670a1673a49"
[[package]]
name = "hotpath-meta"
version = "0.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8b771f77b2f409086bb40ff029f850ce99d17118d4e207df0f28cb32bd7568c7"
dependencies = [
"hotpath-macros-meta",
]
[[package]]
name = "htmlescape"
version = "0.3.1"
@@ -5051,9 +5023,9 @@ checksum = "15cdd26707701c53297e2fa6afb323d55fbc1d0810c3aec078ae3ef0424c3c15"
[[package]]
name = "hybrid-array"
version = "0.4.14"
version = "0.4.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b"
checksum = "818356c5132c1fede50f837ca96afbe78ff42413047f4abb886217845e1b6c8c"
dependencies = [
"ctutils",
"subtle",
@@ -5301,7 +5273,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01"
dependencies = [
"block-padding 0.3.3",
"generic-array 0.14.7",
"generic-array 0.14.9",
]
[[package]]
@@ -6811,15 +6783,6 @@ dependencies = [
"objc2-core-foundation",
]
[[package]]
name = "object"
version = "0.36.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "62948e14d923ea95ea2c7c86c71013138b66525b86bdc08d2dcc262bdb497b87"
dependencies = [
"memchr",
]
[[package]]
name = "object"
version = "0.37.3"
@@ -7857,7 +7820,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
dependencies = [
"heck",
"itertools 0.13.0",
"itertools 0.10.5",
"log",
"multimap",
"once_cell",
@@ -7877,7 +7840,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
dependencies = [
"heck",
"itertools 0.13.0",
"itertools 0.10.5",
"log",
"multimap",
"petgraph 0.8.3",
@@ -7898,7 +7861,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
dependencies = [
"anyhow",
"itertools 0.13.0",
"itertools 0.10.5",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -7911,7 +7874,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
dependencies = [
"anyhow",
"itertools 0.13.0",
"itertools 0.10.5",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -8551,21 +8514,6 @@ dependencies = [
"web-sys",
]
[[package]]
name = "reqwest-middleware"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "07bc3f1384cffa4f274dad2d4ddd73aed32fed8f786d96c6be8aa4e5fd3c3b58"
dependencies = [
"anyhow",
"async-trait",
"http 1.5.0",
"reqwest",
"serde",
"thiserror 2.0.19",
"tower-service",
]
[[package]]
name = "resolv-conf"
version = "0.7.6"
@@ -9040,7 +8988,6 @@ dependencies = [
"const-str",
"futures",
"hashbrown 0.17.1",
"hotpath",
"metrics",
"rustfs-config",
"rustfs-s3-types",
@@ -9061,7 +9008,6 @@ dependencies = [
"base64-simd",
"bytes",
"crc-fast",
"hotpath",
"http 1.5.0",
"md-5 0.11.0",
"pretty_assertions",
@@ -9075,7 +9021,6 @@ name = "rustfs-common"
version = "1.0.0-beta.12"
dependencies = [
"chrono",
"hotpath",
"metrics",
"rmp-serde",
"s3s",
@@ -9090,7 +9035,6 @@ dependencies = [
name = "rustfs-concurrency"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"insta",
"rustfs-io-core",
"serde",
@@ -9104,7 +9048,6 @@ name = "rustfs-config"
version = "1.0.0-beta.12"
dependencies = [
"const-str",
"hotpath",
"serde",
"serde_json",
]
@@ -9115,7 +9058,6 @@ version = "1.0.0-beta.12"
dependencies = [
"base64-simd",
"hmac 0.13.0",
"hotpath",
"rand 0.10.2",
"serde",
"serde_json",
@@ -9131,7 +9073,6 @@ dependencies = [
"argon2",
"base64-simd",
"chacha20poly1305",
"hotpath",
"jsonwebtoken 11.0.0",
"pbkdf2 0.13.0",
"rand 0.10.2",
@@ -9149,7 +9090,6 @@ name = "rustfs-data-usage"
version = "1.0.0-beta.12"
dependencies = [
"async-trait",
"hotpath",
"rmp-serde",
"rustfs-filemeta",
"serde",
@@ -9159,6 +9099,7 @@ dependencies = [
name = "rustfs-ecstore"
version = "1.0.0-beta.12"
dependencies = [
"aes-gcm",
"arc-swap",
"async-channel",
"async-recursion",
@@ -9174,6 +9115,7 @@ dependencies = [
"byteorder",
"bytes",
"bytesize",
"chacha20poly1305",
"chrono",
"criterion",
"enumset",
@@ -9227,6 +9169,7 @@ dependencies = [
"rustfs-erasure-codec",
"rustfs-filemeta",
"rustfs-io-metrics",
"rustfs-kms",
"rustfs-lifecycle",
"rustfs-lock",
"rustfs-madmin",
@@ -9295,7 +9238,6 @@ dependencies = [
name = "rustfs-extension-schema"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"serde",
"serde_json",
"thiserror 2.0.19",
@@ -9334,7 +9276,6 @@ dependencies = [
"async-trait",
"base64 0.23.0",
"futures",
"hotpath",
"http 1.5.0",
"metrics",
"rustfs-common",
@@ -9366,7 +9307,6 @@ dependencies = [
"async-trait",
"base64-simd",
"futures",
"hotpath",
"http 1.5.0",
"jsonwebtoken 11.0.0",
"moka",
@@ -9401,7 +9341,6 @@ name = "rustfs-io-core"
version = "1.0.0-beta.12"
dependencies = [
"bytes",
"hotpath",
"memmap2",
"rustfs-io-metrics",
"thiserror 2.0.19",
@@ -9414,13 +9353,10 @@ name = "rustfs-io-metrics"
version = "1.0.0-beta.12"
dependencies = [
"criterion",
"hotpath",
"metrics",
"metrics-util",
"num_cpus",
"rustfs-common",
"rustfs-s3-ops",
"rustfs-utils",
"sysinfo",
"thiserror 2.0.19",
"tokio",
@@ -9481,7 +9417,6 @@ version = "1.0.0-beta.12"
dependencies = [
"bytes",
"futures",
"hotpath",
"http 1.5.0",
"http-body 1.1.0",
"http-body-util",
@@ -9507,26 +9442,20 @@ name = "rustfs-kms"
version = "1.0.0-beta.12"
dependencies = [
"aes-gcm",
"anyhow",
"arc-swap",
"argon2",
"async-trait",
"base64 0.23.0",
"chacha20poly1305",
"hex",
"hotpath",
"insta",
"jiff",
"md-5 0.11.0",
"metrics",
"metrics-util",
"moka",
"rand 0.10.2",
"reqwest",
"rustfs-s3-types",
"rustfs-security-governance",
"rustfs-utils",
"rustify",
"serde",
"serde_json",
"sha2 0.11.0",
@@ -9535,7 +9464,6 @@ dependencies = [
"tempfile",
"thiserror 2.0.19",
"tokio",
"tokio-util",
"tracing",
"url",
"uuid",
@@ -9548,7 +9476,6 @@ name = "rustfs-lifecycle"
version = "1.0.0-beta.12"
dependencies = [
"async-trait",
"hotpath",
"metrics",
"metrics-util",
"proptest",
@@ -9573,7 +9500,6 @@ dependencies = [
"async-trait",
"crossbeam-queue",
"futures",
"hotpath",
"parking_lot",
"rand 0.10.2",
"rustfs-io-metrics",
@@ -9595,7 +9521,6 @@ version = "1.0.0-beta.12"
dependencies = [
"chrono",
"flate2",
"hotpath",
"regex",
"serde",
"serde_json",
@@ -9613,7 +9538,6 @@ name = "rustfs-madmin"
version = "1.0.0-beta.12"
dependencies = [
"chrono",
"hotpath",
"humantime",
"hyper",
"rmp-serde",
@@ -9634,7 +9558,6 @@ dependencies = [
"criterion",
"form_urlencoded",
"hashbrown 0.17.1",
"hotpath",
"metrics",
"percent-encoding",
"quick-xml",
@@ -9664,7 +9587,6 @@ version = "1.0.0-beta.12"
dependencies = [
"criterion",
"futures",
"hotpath",
"rustfs-config",
"rustfs-io-metrics",
"rustfs-utils",
@@ -9683,7 +9605,6 @@ version = "1.0.0-beta.12"
dependencies = [
"bytes",
"criterion",
"hotpath",
"metrics",
"metrics-util",
"moka",
@@ -9706,7 +9627,6 @@ dependencies = [
"flate2",
"futures-util",
"glob",
"hotpath",
"jiff",
"libc",
"metrics",
@@ -9755,7 +9675,6 @@ dependencies = [
"base64-simd",
"chrono",
"futures",
"hotpath",
"ipnetwork",
"jsonwebtoken 11.0.0",
"moka",
@@ -9792,7 +9711,6 @@ dependencies = [
"futures-util",
"hex",
"hmac 0.13.0",
"hotpath",
"http 1.5.0",
"http-body-util",
"hyper",
@@ -9845,7 +9763,6 @@ name = "rustfs-protos"
version = "1.0.0-beta.12"
dependencies = [
"flatbuffers",
"hotpath",
"prost 0.14.4",
"rmp-serde",
"rustfs-common",
@@ -9870,7 +9787,6 @@ version = "1.0.0-beta.12"
dependencies = [
"byteorder",
"bytes",
"hotpath",
"regex",
"rmp",
"rmp-serde",
@@ -9929,7 +9845,6 @@ dependencies = [
"chacha20poly1305",
"hex",
"hmac 0.13.0",
"hotpath",
"minlz",
"pin-project-lite",
"rand 0.10.2",
@@ -9947,7 +9862,6 @@ dependencies = [
name = "rustfs-s3-ops"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"rustfs-s3-types",
]
@@ -9955,7 +9869,6 @@ dependencies = [
name = "rustfs-s3-types"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"serde",
"serde_json",
]
@@ -9970,7 +9883,6 @@ dependencies = [
"datafusion",
"futures",
"futures-core",
"hotpath",
"http 1.5.0",
"metrics",
"parking_lot",
@@ -9999,7 +9911,6 @@ dependencies = [
"datafusion",
"derive_builder",
"futures",
"hotpath",
"parking_lot",
"rustfs-s3select-api",
"s3s",
@@ -10017,7 +9928,6 @@ dependencies = [
"futures",
"hex-simd",
"hmac 0.13.0",
"hotpath",
"http 1.5.0",
"metrics",
"rand 0.10.2",
@@ -10028,7 +9938,6 @@ dependencies = [
"rustfs-data-usage",
"rustfs-ecstore",
"rustfs-filemeta",
"rustfs-lock",
"rustfs-storage-api",
"rustfs-utils",
"s3s",
@@ -10051,7 +9960,6 @@ dependencies = [
name = "rustfs-security-governance"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"thiserror 2.0.19",
]
@@ -10061,7 +9969,6 @@ version = "1.0.0-beta.12"
dependencies = [
"base64-simd",
"bytes",
"hotpath",
"http 1.5.0",
"hyper",
"rustfs-utils",
@@ -10078,7 +9985,6 @@ name = "rustfs-storage-api"
version = "1.0.0-beta.12"
dependencies = [
"async-trait",
"hotpath",
"insta",
"rustfs-filemeta",
"serde",
@@ -10100,7 +10006,6 @@ dependencies = [
"deadpool-postgres",
"futures-util",
"hashbrown 0.17.1",
"hotpath",
"hyper",
"hyper-rustls",
"lapin",
@@ -10146,7 +10051,6 @@ dependencies = [
name = "rustfs-test-utils"
version = "1.0.0-beta.12"
dependencies = [
"hotpath",
"rustfs-data-usage",
"rustfs-ecstore",
"rustfs-storage-api",
@@ -10163,7 +10067,6 @@ name = "rustfs-tls-runtime"
version = "1.0.0-beta.12"
dependencies = [
"arc-swap",
"hotpath",
"metrics",
"rcgen",
"rustfs-common",
@@ -10185,7 +10088,6 @@ version = "1.0.0-beta.12"
dependencies = [
"async-trait",
"axum",
"hotpath",
"http 1.5.0",
"ipnetwork",
"metrics",
@@ -10232,7 +10134,6 @@ dependencies = [
"hex-simd",
"highway",
"hmac 0.13.0",
"hotpath",
"http 1.5.0",
"hyper",
"local-ip-address",
@@ -10265,7 +10166,6 @@ dependencies = [
"astral-tokio-tar",
"async-compression",
"criterion",
"hotpath",
"tempfile",
"thiserror 2.0.19",
"tokio",
@@ -10609,7 +10509,7 @@ checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc"
dependencies = [
"base16ct 0.2.0",
"der 0.7.10",
"generic-array 0.14.7",
"generic-array 0.14.9",
"pkcs8 0.10.2",
"subtle",
"zeroize",
@@ -11599,7 +11499,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.4.3",
"getrandom 0.3.4",
"once_cell",
"rustix",
"windows-sys 0.61.2",
+3 -4
View File
@@ -250,8 +250,8 @@ enumset = "1.1.14"
faster-hex = "0.10.0"
flate2 = "1.1.9"
glob = "0.3.4"
google-cloud-storage = "1.17.0"
google-cloud-auth = "1.15.0"
google-cloud-storage = "1.16.0"
google-cloud-auth = "1.14.0"
hashbrown = { version = "0.17.1" }
hex = "0.4.3"
hex-simd = "0.8.0"
@@ -285,7 +285,6 @@ reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.33.3" }
redis = { version = "1.5.0" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.4" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
@@ -348,7 +347,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = "0.1.52"
hotpath = { version = "0.22.0", default-features = false }
hotpath = "0.22.0"
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
-26
View File
@@ -25,33 +25,7 @@ documentation = "https://docs.rs/rustfs-audit/latest/rustfs_audit/"
keywords = ["audit", "target", "management", "fan-out", "RustFS"]
categories = ["web-programming", "development-tools", "asynchronous", "api-bindings"]
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"rustfs-config/hotpath",
"rustfs-s3-types/hotpath",
"rustfs-targets/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-s3-types/hotpath-alloc",
"rustfs-targets/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-s3-types/hotpath-cpu",
"rustfs-targets/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
rustfs-targets = { workspace = true }
rustfs-config = { workspace = true, features = ["audit", "server-config-model"] }
rustfs-s3-types = { workspace = true }
-7
View File
@@ -28,14 +28,7 @@ documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
crc-fast = { workspace = true }
http = { workspace = true }
-7
View File
@@ -27,14 +27,7 @@ categories = ["web-programming", "development-tools", "data-structures"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
tonic = { workspace = true, features = ["gzip", "deflate"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
-7
View File
@@ -13,14 +13,7 @@ categories = ["concurrency", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-io-core/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-io-core/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-io-core/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
# Internal crates
rustfs-io-core = { workspace = true }
serde = { workspace = true, features = ["derive"] }
-4
View File
@@ -25,7 +25,6 @@ keywords = ["configuration", "settings", "management", "rustfs", "Minio"]
categories = ["web-programming", "development-tools", "config"]
[dependencies]
hotpath.workspace = true
const-str = { workspace = true, optional = true, features = ["std", "proc"] }
serde = { workspace = true, optional = true, features = ["derive"] }
serde_json = { workspace = true, optional = true, features = ["raw_value"] }
@@ -35,9 +34,6 @@ workspace = true
[features]
default = ["constants"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
audit = ["dep:const-str", "constants"]
constants = ["dep:const-str"]
notify = ["dep:const-str", "constants"]
-7
View File
@@ -24,14 +24,7 @@ description = "Credentials management utilities for RustFS, enabling secure hand
keywords = ["rustfs", "Minio", "credentials", "authentication", "authorization"]
categories = ["web-programming", "development-tools", "data-structures", "security"]
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
base64-simd = { workspace = true }
hmac = { workspace = true }
rand = { workspace = true, features = ["serde"] }
-4
View File
@@ -29,7 +29,6 @@ documentation = "https://docs.rs/rustfs-crypto/latest/rustfs_crypto/"
workspace = true
[dependencies]
hotpath.workspace = true
aes-gcm = { workspace = true, optional = true, features = ["rand_core"] }
argon2 = { workspace = true, optional = true }
chacha20poly1305 = { workspace = true, optional = true }
@@ -50,9 +49,6 @@ time = { workspace = true, features = ["parsing", "formatting", "macros", "serde
[features]
default = ["crypto", "fips"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
fips = []
crypto = [
"dep:aes-gcm",
-7
View File
@@ -27,14 +27,7 @@ categories = ["data-structures", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "rustfs-filemeta/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-filemeta/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-filemeta/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
serde = { workspace = true, features = ["derive"] }
rmp-serde = { workspace = true }
async-trait = { workspace = true }
-48
View File
@@ -25,58 +25,10 @@ workspace = true
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"hotpath/reqwest-0-13",
"rustfs-config/hotpath",
"rustfs-credentials/hotpath",
"rustfs-data-usage/hotpath",
"rustfs-ecstore/hotpath",
"rustfs-filemeta/hotpath",
"rustfs-lock/hotpath",
"rustfs-madmin/hotpath",
"rustfs-protos/hotpath",
"rustfs-rio/hotpath",
"rustfs-signer/hotpath",
"rustfs-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-credentials/hotpath-alloc",
"rustfs-data-usage/hotpath-alloc",
"rustfs-ecstore/hotpath-alloc",
"rustfs-filemeta/hotpath-alloc",
"rustfs-lock/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-protos/hotpath-alloc",
"rustfs-rio/hotpath-alloc",
"rustfs-signer/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-credentials/hotpath-cpu",
"rustfs-data-usage/hotpath-cpu",
"rustfs-ecstore/hotpath-cpu",
"rustfs-filemeta/hotpath-cpu",
"rustfs-lock/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-protos/hotpath-cpu",
"rustfs-rio/hotpath-cpu",
"rustfs-signer/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
]
ftps = []
sftp = []
[dependencies]
hotpath.workspace = true
rustfs-config = { workspace = true, features = ["constants"] }
rustfs-credentials.workspace = true
rustfs-ecstore.workspace = true
+60 -1
View File
@@ -26,16 +26,75 @@
//! Later batches tracked on backlog#1154: config get/set, info, pools status,
//! group lifecycle, import/export IAM.
use crate::common::{RustFSTestEnvironment, admin_ok, admin_request, init_logging};
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::{Client, Config};
use http::header::{CONTENT_TYPE, HOST};
use reqwest::StatusCode;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::error::Error;
use tokio::time::{Duration, sleep};
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
type BoxError = Box<dyn Error + Send + Sync>;
/// Signs and sends an admin HTTP request with the given credential, returning
/// status and body. Native `/rustfs/admin/v3` requests and responses are plain
/// JSON (the MinIO-compat encryption applies only to `/minio/admin/v3` paths).
async fn admin_request(
base_url: &str,
method: http::Method,
path_and_query: &str,
body: Option<String>,
access_key: &str,
secret_key: &str,
) -> Result<(StatusCode, String), BoxError> {
let url = format!("{base_url}{path_and_query}");
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
let mut builder = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
if body.is_some() {
builder = builder.header(CONTENT_TYPE, "application/json");
}
let content_len = body.as_ref().map(|b| b.len() as i64).unwrap_or_default();
let signed = sign_v4(builder.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
let mut request = local_http_client().request(reqwest_method, &url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
if let Some(body) = body {
request = request.body(body);
}
let response = request.send().await?;
let status = response.status();
let text = response.text().await.unwrap_or_default();
Ok((status, text))
}
/// Root-credential admin request that must succeed; returns the response body.
async fn admin_ok(
env: &RustFSTestEnvironment,
method: http::Method,
path_and_query: &str,
body: Option<String>,
) -> Result<String, BoxError> {
let (status, text) = admin_request(&env.url, method.clone(), path_and_query, body, &env.access_key, &env.secret_key).await?;
if !status.is_success() {
return Err(format!("{method} {path_and_query} failed: {status} {text}").into());
}
Ok(text)
}
fn build_s3_client(url: &str, access_key: &str, secret_key: &str) -> Client {
let config = Config::builder()
-57
View File
@@ -24,12 +24,7 @@
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::{Client, Config};
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use http::header::{CONTENT_TYPE, HOST};
use reqwest::Client as HttpClient;
use reqwest::StatusCode;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::ffi::OsStr;
use std::fs as stdfs;
use std::path::{Path, PathBuf};
@@ -80,58 +75,6 @@ pub fn local_http_client() -> HttpClient {
.expect("failed to build local reqwest client")
}
/// Signs and sends an admin HTTP request with the given credentials.
pub(crate) async fn admin_request(
base_url: &str,
method: http::Method,
path_and_query: &str,
body: Option<String>,
access_key: &str,
secret_key: &str,
) -> Result<(StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{base_url}{path_and_query}");
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
let mut request = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
if body.is_some() {
request = request.header(CONTENT_TYPE, "application/json");
}
let content_length = i64::try_from(body.as_ref().map_or(0, String::len)).map_err(|_| "admin request body is too large")?;
let signed = sign_v4(request.body(Body::empty())?, content_length, access_key, secret_key, "", "us-east-1");
let mut request = local_http_client().request(method, &url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
if let Some(body) = body {
request = request.body(body);
}
let response = request.send().await?;
let status = response.status();
let body = response.text().await?;
Ok((status, body))
}
/// Sends a root-credential admin request and returns its successful response body.
pub(crate) async fn admin_ok(
env: &RustFSTestEnvironment,
method: http::Method,
path_and_query: &str,
body: Option<String>,
) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
let (status, response_body) =
admin_request(&env.url, method.clone(), path_and_query, body, &env.access_key, &env.secret_key).await?;
if !status.is_success() {
return Err(format!("{method} {path_and_query} failed: {status} {response_body}").into());
}
Ok(response_body)
}
/// Resolve the RustFS binary relative to the workspace.
pub fn rustfs_binary_path() -> PathBuf {
rustfs_binary_path_with_features(requested_rustfs_build_features().as_deref())
@@ -117,14 +117,9 @@ mod tests {
.key("assets/explicit-copy.js")
.copy_source(format!("{bucket}/{key}"))
.metadata_directive(MetadataDirective::Copy)
.customize()
.mutate_request(|request| {
request.headers_mut().insert("content-type", "application/octet-stream");
request.headers_mut().insert("x-amz-meta-request-only", "ignored");
})
.send()
.await
.expect("explicit COPY directive with request metadata failed");
.expect("explicit COPY directive failed");
let explicit_copy_head = client
.head_object()
.bucket(bucket)
@@ -133,18 +128,6 @@ mod tests {
.await
.expect("HEAD failed after explicit COPY");
assert_eq!(explicit_copy_head.cache_control(), Some("max-age=60"));
assert_eq!(explicit_copy_head.content_type(), Some("text/javascript; charset=utf-8"));
assert_eq!(
explicit_copy_head.metadata().and_then(|metadata| metadata.get("mtime")),
Some(&"1777992333".to_string())
);
assert_eq!(
explicit_copy_head
.metadata()
.and_then(|metadata| metadata.get("request-only")),
None,
"COPY must ignore request metadata"
);
assert_eq!(
explicit_copy_head.website_redirect_location(),
None,
@@ -588,6 +571,20 @@ mod tests {
Some("InvalidArgument")
);
let ignored_replacement = client
.copy_object()
.bucket(bucket)
.key(key)
.copy_source(format!("{bucket}/{key}"))
.content_type("application/ignored")
.send()
.await
.expect_err("Replacement fields without REPLACE should be rejected");
assert_eq!(
ignored_replacement.as_service_error().and_then(|error| error.code()),
Some("InvalidRequest")
);
let unchanged = client
.get_object()
.bucket(bucket)
@@ -56,21 +56,6 @@ mod tests {
);
}
async fn assert_current_list_hides_delete_marker(client: &Client, bucket: &str, key: &str) {
let listed = client
.list_objects_v2()
.bucket(bucket)
.prefix(key)
.send()
.await
.expect("list current objects after delete marker");
assert!(
listed.contents().iter().all(|object| object.key() != Some(key)),
"ListObjectsV2 must hide an object whose latest version is a delete marker"
);
}
#[tokio::test]
#[serial]
async fn test_versioning_only_delete_marker_has_minio_compatible_visibility_for_migration_proof() {
@@ -109,7 +94,6 @@ mod tests {
assert_eq!(markers[0].version_id(), Some(delete_marker_version_id));
assert_eq!(markers[0].is_latest(), Some(true));
assert_current_get_is_delete_marker_not_found(&client, bucket, key).await;
assert_current_list_hides_delete_marker(&client, bucket, key).await;
}
#[tokio::test]
@@ -134,17 +118,6 @@ mod tests {
.await
.expect("put historical version");
let data_version_id = put.version_id().expect("put should return data version id");
let listed_before_delete = client
.list_objects_v2()
.bucket(bucket)
.prefix(key)
.send()
.await
.expect("list current object before creating delete marker");
assert!(
listed_before_delete.contents().iter().any(|object| object.key() == Some(key)),
"ListObjectsV2 must include the current object before it is deleted"
);
let delete_marker = client
.delete_object()
@@ -172,7 +145,6 @@ mod tests {
assert_eq!(markers[0].version_id(), Some(delete_marker_version_id));
assert_eq!(markers[0].is_latest(), Some(true));
assert_current_get_is_delete_marker_not_found(&client, bucket, key).await;
assert_current_list_hides_delete_marker(&client, bucket, key).await;
let historical = client
.get_object()
@@ -2136,20 +2136,11 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
assert_eq!(terminal["bucket"].as_str(), Some(bucket.as_str()));
assert_eq!(terminal["prefix"].as_str(), Some(prefix));
assert_eq!(terminal["dry_run"].as_bool(), Some(false));
let terminal_status = terminal["status"].as_str();
assert!(
matches!(terminal_status, Some("partial" | "unknown")),
assert_eq!(
terminal["status"].as_str(),
Some("partial"),
"small transition queue should surface terminal backpressure: {terminal}"
);
if terminal_status == Some("unknown") {
let failure_reason = terminal["failure_reason"]
.as_str()
.ok_or_else(|| format!("unknown terminal status omitted failure_reason: {terminal}"))?;
assert!(
failure_reason.contains("worker result was not persisted before the transition queue drained"),
"unknown terminal status should identify lost worker-result persistence: {terminal}"
);
}
let skipped_queue_full = terminal["report"]["skipped_queue_full"]
.as_u64()
.ok_or_else(|| format!("terminal status omitted report.skipped_queue_full: {terminal}"))?;
@@ -26,7 +26,6 @@
use super::common::*;
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::{ByteStream, DateTimeFormat};
use aws_sdk_s3::types::{
CompletedMultipartUpload, CompletedPart, Delete, MetadataDirective, ObjectIdentifier, ObjectLockLegalHoldStatus,
@@ -2121,127 +2120,6 @@ async fn test_multipart_default_retention_fixed_at_create() {
// Versioning Auto-Enable Tests
// ============================================================================
#[tokio::test]
#[serial]
async fn test_unretained_object_lock_object_delete_and_bucket_cleanup() {
init_logging();
info!("🧪 Test: Unretained Object Lock object delete and bucket cleanup (Issue #5339)");
let mut env = ObjectLockTestEnvironment::new()
.await
.expect("failed to create Object Lock test environment");
env.start_rustfs().await.expect("failed to start RustFS");
let bucket = "test-object-lock-delete-cleanup";
let key = "unretained-object";
env.create_object_lock_bucket(bucket)
.await
.expect("failed to create Object Lock bucket");
let client = env.s3_client();
let put_response = client
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"unretained data"))
.send()
.await
.expect("failed to upload unretained object");
let object_version_id = put_response
.version_id()
.expect("Object Lock buckets must create versioned objects")
.to_string();
let delete_response = client
.delete_object()
.bucket(bucket)
.key(key)
.send()
.await
.expect("failed to create delete marker");
assert_eq!(delete_response.delete_marker(), Some(true));
let delete_marker_version_id = delete_response
.version_id()
.expect("Deleting without a version ID must create a delete marker")
.to_string();
let get_error = client
.get_object()
.bucket(bucket)
.key(key)
.send()
.await
.expect_err("GET must not return an object hidden by a delete marker");
assert_eq!(get_error.raw_response().map(|response| response.status().as_u16()), Some(404));
assert_eq!(get_error.as_service_error().and_then(|error| error.code()), Some("NoSuchKey"));
let listed_objects = client
.list_objects_v2()
.bucket(bucket)
.send()
.await
.expect("failed to list current objects");
assert!(
listed_objects.contents().iter().all(|object| object.key() != Some(key)),
"ListObjectsV2 must hide objects whose latest version is a delete marker"
);
let listed_versions = client
.list_object_versions()
.bucket(bucket)
.send()
.await
.expect("failed to list object versions");
assert!(
listed_versions
.versions()
.iter()
.any(|version| version.key() == Some(key) && version.version_id() == Some(object_version_id.as_str())),
"The data version must remain until it is explicitly deleted"
);
assert!(
listed_versions
.delete_markers()
.iter()
.any(|marker| marker.key() == Some(key) && marker.version_id() == Some(delete_marker_version_id.as_str())),
"ListObjectVersions must expose the delete marker"
);
client
.delete_object()
.bucket(bucket)
.key(key)
.version_id(object_version_id)
.send()
.await
.expect("failed to delete the data version");
client
.delete_object()
.bucket(bucket)
.key(key)
.version_id(delete_marker_version_id)
.send()
.await
.expect("failed to delete the delete marker");
let remaining_versions = client
.list_object_versions()
.bucket(bucket)
.send()
.await
.expect("failed to list versions after cleanup");
assert!(remaining_versions.versions().is_empty());
assert!(remaining_versions.delete_markers().is_empty());
client
.delete_bucket()
.bucket(bucket)
.send()
.await
.expect("Deleting every version must remove xl.meta so the bucket can be deleted normally");
}
#[tokio::test]
#[serial]
async fn test_versioning_auto_enabled_with_object_lock() {
+46 -18
View File
@@ -382,6 +382,7 @@ struct ManualTransitionRunReport {
skipped_delete_marker: u64,
skipped_directory: u64,
skipped_replication: u64,
skipped_already_transitioned: u64,
skipped_already_in_flight: u64,
skipped_queue_full: u64,
skipped_queue_closed: u64,
@@ -407,6 +408,41 @@ fn assert_completed_or_in_flight_partial(state: &str, report: &ManualTransitionR
}
}
fn assert_conflict_winner_report(state: &str, report: &ManualTransitionRunReport, expected_objects: u64, context: &str) {
assert_completed_or_in_flight_partial(state, report, context);
if report.skipped_already_in_flight > 0 {
assert!(
report.scanned <= expected_objects,
"{context}: scanned more objects than the conflict scope contains: {report:#?}"
);
assert!(
report.eligible <= expected_objects,
"{context}: marked more objects eligible than the conflict scope contains: {report:#?}"
);
assert_eq!(
report.enqueued + report.skipped_already_in_flight,
report.eligible,
"{context}: partial in-flight accounting must cover every eligible object: {report:#?}"
);
} else {
assert_eq!(report.scanned, expected_objects, "{context}: {report:#?}");
assert_eq!(
report.eligible + report.skipped_already_transitioned,
expected_objects,
"{context}: {report:#?}"
);
assert_eq!(
report.enqueued + report.skipped_already_in_flight,
expected_objects,
"{context}: {report:#?}"
);
}
assert_eq!(
report.transition_completed, report.enqueued,
"{context}: winner must wait for all queued transitions: {report:#?}"
);
}
#[derive(Debug, Deserialize)]
struct ManualTransitionQueueSnapshot {
queue_capacity: u64,
@@ -1240,15 +1276,8 @@ async fn test_manual_transition_async_scope_conflicts_report_active_job() -> Tes
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(
vec![],
&[
("RUSTFS_SCANNER_ENABLED", "false"),
("RUSTFS_SCANNER_CYCLE", "3600"),
(MANUAL_TRANSITION_CANCEL_BARRIER_ENV, "1"),
],
)
.await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
@@ -1318,21 +1347,20 @@ async fn test_manual_transition_async_scope_conflicts_report_active_job() -> Tes
assert_eq!(conflict.cancel_endpoint, status_endpoint);
assert!(!conflict.scope_key.is_empty());
manual_transition_job_cancel(&hot, cancel_endpoint).await?;
let terminal = wait_for_manual_transition_job_terminal(&hot, status_endpoint, MANUAL_ASYNC_CONFLICT_TERMINAL_TIMEOUT).await?;
assert_eq!(terminal.job_id, job_id);
assert_eq!(terminal.status, "cancelled", "terminal conflict winner response: {terminal:#?}");
assert!(!terminal.report.dry_run);
assert_eq!(terminal.report.bucket, MANUAL_ASYNC_CONFLICT_BUCKET);
assert_eq!(terminal.report.prefix, accepted.report.prefix);
assert!(terminal.report.cancelled, "terminal conflict winner response: {terminal:#?}");
assert_eq!(terminal.report.scanned, 0, "terminal conflict winner response: {terminal:#?}");
assert_eq!(terminal.report.enqueued, 0, "terminal conflict winner response: {terminal:#?}");
assert_eq!(
terminal.report.transition_completed, 0,
"terminal conflict winner response: {terminal:#?}"
assert_conflict_winner_report(
&terminal.status,
&terminal.report,
MANUAL_ASYNC_CONFLICT_OBJECTS as u64,
"terminal conflict winner response",
);
assert_eq!(terminal.report.dry_run_eligible, 0, "terminal conflict winner response: {terminal:#?}");
assert_eq!(terminal.report.transition_failed, 0, "terminal conflict winner response: {terminal:#?}");
assert_eq!(terminal.report.tier_failure, 0, "terminal conflict winner response: {terminal:#?}");
let after_remote_count = cold_tier_object_count(&cold_client).await?;
assert!(after_remote_count >= before_remote_count);
assert!(after_remote_count <= before_remote_count + MANUAL_ASYNC_CONFLICT_OBJECTS);
@@ -3237,6 +3237,116 @@ async fn test_bucket_replication_converges_delete_marker_and_version_purge() ->
.await?;
assert_eq!(retained.body.collect().await?.into_bytes().as_ref(), b"versioned replication payload v2");
source_client
.delete_object()
.bucket(source_bucket)
.key(object_key)
.version_id(delete_marker_version_id)
.send()
.await?;
assert_replication_converged(&source_client, source_bucket, &target_client, target_bucket).await?;
let target_state = list_replication_state(&target_client, target_bucket).await?;
assert!(
target_state.iter().all(|entry| entry.version_id != delete_marker_version_id),
"target retained the explicitly purged delete-marker version"
);
assert!(
target_state
.iter()
.any(|entry| !entry.delete_marker && entry.version_id == retained_version_id),
"target removed the retained object version while purging the delete marker: {target_state:?}"
);
Ok(())
}
#[tokio::test]
#[serial]
async fn test_bucket_replication_disabled_version_delete_preserves_target_versions_issue_5442() -> TestResult {
init_logging();
let mut source_env = RustFSTestEnvironment::new().await?;
let mut source_env_vars = replication_fast_env();
source_env_vars.extend_from_slice(LOOPBACK_REPLICATION_TARGET_ENV);
source_env.start_rustfs_server_with_env(vec![], &source_env_vars).await?;
let mut target_env = RustFSTestEnvironment::new().await?;
target_env.start_rustfs_server_without_cleanup(vec![]).await?;
let source_bucket = "replication-no-version-delete-src";
let target_bucket = "replication-no-version-delete-dst";
let object_key = "retained-versions.txt";
let source_client = source_env.create_s3_client();
let target_client = target_env.create_s3_client();
source_client.create_bucket().bucket(source_bucket).send().await?;
target_client.create_bucket().bucket(target_bucket).send().await?;
enable_bucket_versioning(&source_env, source_bucket).await?;
enable_bucket_versioning(&target_env, target_bucket).await?;
let target_arn = set_replication_target(&source_env, source_bucket, &target_env, target_bucket).await?;
put_bucket_replication_with_delete_statuses(&source_env, source_bucket, &target_arn, "Enabled", None).await?;
let put = source_client
.put_object()
.bucket(source_bucket)
.key(object_key)
.body(ByteStream::from_static(b"permanent delete replication disabled"))
.send()
.await?;
let object_version_id = put.version_id().ok_or("source PUT omitted version ID")?.to_string();
assert_replication_converged(&source_client, source_bucket, &target_client, target_bucket).await?;
let delete = source_client
.delete_object()
.bucket(source_bucket)
.key(object_key)
.send()
.await?;
let delete_marker_version_id = delete
.version_id()
.ok_or("source DELETE omitted marker version ID")?
.to_string();
assert_eq!(delete.delete_marker(), Some(true));
assert_replication_converged(&source_client, source_bucket, &target_client, target_bucket).await?;
let expected_target_state = list_replication_state(&target_client, target_bucket).await?;
assert_eq!(expected_target_state.len(), 2);
assert!(
expected_target_state
.iter()
.any(|entry| !entry.delete_marker && entry.version_id == object_version_id)
);
assert!(
expected_target_state
.iter()
.any(|entry| entry.delete_marker && entry.version_id == delete_marker_version_id)
);
for version_id in [&object_version_id, &delete_marker_version_id] {
source_client
.delete_object()
.bucket(source_bucket)
.key(object_key)
.version_id(version_id)
.send()
.await?;
}
assert!(list_replication_state(&source_client, source_bucket).await?.is_empty());
let observation_deadline = tokio::time::Instant::now() + Duration::from_secs(10);
loop {
let target_state = list_replication_state(&target_client, target_bucket).await?;
assert_eq!(
target_state, expected_target_state,
"disabled permanent-delete replication changed target versions"
);
if tokio::time::Instant::now() >= observation_deadline {
break;
}
sleep(Duration::from_millis(100)).await;
}
Ok(())
}
+37 -421
View File
@@ -12,35 +12,22 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::common::{RustFSTestEnvironment, admin_ok, init_logging};
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_sts::config::retry::RetryConfig;
use aws_sdk_sts::config::{Credentials, Region};
use aws_sdk_sts::error::ProvideErrorMetadata;
use aws_sdk_sts::operation::RequestId;
use aws_sdk_sts::{Client, Config};
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use bytes::Bytes;
use http::header::{AUTHORIZATION, CONTENT_TYPE};
use http::{Request, Response};
use http_body_util::{BodyExt, Full};
use hyper::body::Incoming;
use hyper::server::conn::http1;
use hyper::service::service_fn;
use hyper_util::rt::TokioIo;
use serde_json::Value;
use http::header::{CONTENT_TYPE, HOST};
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::collections::BTreeSet;
use std::convert::Infallible;
use std::error::Error;
use std::sync::Arc;
use tokio::net::TcpListener;
use tokio::sync::{Notify, mpsc};
use tokio::task::{JoinHandle, JoinSet};
use tokio::time::{Duration, timeout};
type BoxError = Box<dyn Error + Send + Sync>;
type TestResult = Result<(), BoxError>;
const OPA_AUTH_TOKEN: &str = "sts-opa-token";
fn sts_client(url: &str, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
let mut config = Config::builder()
@@ -62,14 +49,32 @@ fn sts_client(url: &str, access_key: &str, secret_key: &str, session_token: Opti
}
async fn create_root_service_account(env: &RustFSTestEnvironment) -> Result<(String, String), BoxError> {
let body = admin_ok(
env,
http::Method::PUT,
"/rustfs/admin/v3/add-service-accounts",
Some(serde_json::json!({ "targetUser": env.access_key.clone() }).to_string()),
)
.await?;
let response: Value = serde_json::from_str(&body)?;
let path = "/rustfs/admin/v3/add-service-accounts";
let url = format!("{}{path}", env.url);
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
let body = serde_json::json!({ "targetUser": env.access_key.clone() }).to_string();
let request = http::Request::builder()
.method(http::Method::PUT)
.uri(uri)
.header(HOST, authority)
.header(CONTENT_TYPE, "application/json")
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
.body(Body::empty())?;
let content_length = i64::try_from(body.len()).map_err(|_| "service account request body is too large")?;
let signed = sign_v4(request, content_length, &env.access_key, &env.secret_key, "", "us-east-1");
let mut request = local_http_client().put(&url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
let response = request.body(body).send().await?;
let status = response.status();
let body = response.text().await?;
if !status.is_success() {
return Err(format!("create service account failed: {status} {body}").into());
}
let response: serde_json::Value = serde_json::from_str(&body)?;
let access_key = response["credentials"]["accessKey"]
.as_str()
.ok_or("service account response should contain credentials.accessKey")?
@@ -81,237 +86,28 @@ async fn create_root_service_account(env: &RustFSTestEnvironment) -> Result<(Str
Ok((access_key, secret_key))
}
async fn create_user_with_policy(
env: &RustFSTestEnvironment,
user: &str,
secret: &str,
policy_name: &str,
statements: Value,
) -> TestResult {
create_user(env, user, secret).await?;
admin_ok(
env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-canned-policy?name={policy_name}"),
Some(
serde_json::json!({
"Version": "2012-10-17",
"Statement": statements,
})
.to_string(),
),
)
.await?;
admin_ok(
env,
http::Method::POST,
"/rustfs/admin/v3/idp/builtin/policy/attach",
Some(serde_json::json!({ "policies": [policy_name], "user": user }).to_string()),
)
.await?;
Ok(())
}
async fn create_user(env: &RustFSTestEnvironment, user: &str, secret: &str) -> TestResult {
admin_ok(
env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-user?accessKey={user}"),
Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
)
.await?;
Ok(())
}
async fn assert_access_denied(client: &Client, context: &str) -> TestResult {
async fn assert_chaining_denied(client: &Client, credential_kind: &str) -> TestResult {
let error = client
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/test")
.role_session_name("sts-query-compat-e2e")
.send()
.await
.expect_err("AssumeRole must be denied");
.expect_err("credential chaining must be denied");
let service_error = error
.as_service_error()
.ok_or_else(|| format!("{context} should deserialize as an STS service error: {error:?}"))?;
.ok_or_else(|| format!("{credential_kind} denial should deserialize as an STS service error: {error:?}"))?;
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(403));
assert_eq!(service_error.code(), Some("AccessDenied"));
assert_eq!(service_error.message(), Some("Access Denied"));
assert!(
error.request_id().is_some_and(|request_id| !request_id.is_empty()),
"{context} should include a request ID"
"{credential_kind} denial should include a request ID"
);
Ok(())
}
async fn handle_opa_request(
request: Request<Incoming>,
requests: mpsc::UnboundedSender<Value>,
validation_started: mpsc::UnboundedSender<()>,
validation_mode: OpaValidationMode,
expected_authorization: Option<String>,
) -> Result<Response<Full<Bytes>>, Infallible> {
if let Some(expected_authorization) = expected_authorization
&& request.headers().get(AUTHORIZATION).and_then(|value| value.to_str().ok()) != Some(expected_authorization.as_str())
{
return Ok(Response::builder()
.status(401)
.body(Full::new(Bytes::new()))
.expect("static OPA unauthorized response must be valid"));
}
let body = match request.into_body().collect().await {
Ok(body) => body.to_bytes(),
Err(error) => {
return Ok(Response::builder()
.status(400)
.body(Full::new(Bytes::from(error.to_string())))
.expect("static OPA error response must be valid"));
}
};
let payload = if body.is_empty() {
None
} else {
match serde_json::from_slice::<Value>(&body) {
Ok(payload) => Some(payload),
Err(error) => {
return Ok(Response::builder()
.status(400)
.body(Full::new(Bytes::from(error.to_string())))
.expect("static OPA error response must be valid"));
}
}
};
if payload.is_none() {
let _ = validation_started.send(());
if let OpaValidationMode::DelayedUnavailable(release) = validation_mode {
release.notified().await;
return Ok(Response::builder()
.status(503)
.body(Full::new(Bytes::new()))
.expect("static OPA unavailable response must be valid"));
}
}
let allow = match payload.as_ref().and_then(|value| value.pointer("/input/identity/account")) {
Some(Value::String(account)) if account == "opaallow" => payload
.as_ref()
.and_then(|value| value.pointer("/input/context/deny_only"))
.and_then(Value::as_bool)
.unwrap_or(false),
Some(Value::String(account)) if account == "opadeny" => false,
None => true,
_ => false,
};
if let Some(payload) = payload {
let _ = requests.send(payload);
}
let body =
serde_json::to_vec(&serde_json::json!({ "result": { "allow": allow } })).expect("static OPA response must serialize");
Ok(Response::builder()
.header(CONTENT_TYPE, "application/json")
.body(Full::new(Bytes::from(body)))
.expect("static OPA response must be valid"))
}
#[derive(Clone)]
enum OpaValidationMode {
Ready,
DelayedUnavailable(Arc<Notify>),
}
struct OpaMock {
url: String,
requests: mpsc::UnboundedReceiver<Value>,
validation_started: mpsc::UnboundedReceiver<()>,
validation_release: Option<Arc<Notify>>,
task: JoinHandle<()>,
}
impl OpaMock {
async fn start() -> Result<Self, BoxError> {
Self::start_with_mode(OpaValidationMode::Ready, Some(OPA_AUTH_TOKEN)).await
}
async fn start_delayed_unavailable() -> Result<Self, BoxError> {
let release = Arc::new(Notify::new());
Self::start_with_mode(OpaValidationMode::DelayedUnavailable(release), None).await
}
async fn start_with_mode(validation_mode: OpaValidationMode, auth_token: Option<&str>) -> Result<Self, BoxError> {
let listener = TcpListener::bind("127.0.0.1:0").await?;
let url = format!("http://{}/v1/data/rustfs/authz/allow", listener.local_addr()?);
let (requests_tx, requests) = mpsc::unbounded_channel();
let (validation_started_tx, validation_started) = mpsc::unbounded_channel();
let expected_authorization = auth_token.map(|token| format!("Bearer {token}"));
let validation_release = match &validation_mode {
OpaValidationMode::Ready => None,
OpaValidationMode::DelayedUnavailable(release) => Some(Arc::clone(release)),
};
let task = tokio::spawn(async move {
let mut connections = JoinSet::new();
loop {
tokio::select! {
accepted = listener.accept() => {
let Ok((stream, _)) = accepted else { break };
let requests = requests_tx.clone();
let validation_started = validation_started_tx.clone();
let validation_mode = validation_mode.clone();
let expected_authorization = expected_authorization.clone();
connections.spawn(async move {
let handler = service_fn(move |request| {
handle_opa_request(
request,
requests.clone(),
validation_started.clone(),
validation_mode.clone(),
expected_authorization.clone(),
)
});
let _ = http1::Builder::new()
.serve_connection(TokioIo::new(stream), handler)
.await;
});
}
_ = connections.join_next(), if !connections.is_empty() => {}
}
}
});
Ok(Self {
url,
requests,
validation_started,
validation_release,
task,
})
}
async fn next_request(&mut self) -> Result<Value, BoxError> {
timeout(Duration::from_secs(5), self.requests.recv())
.await?
.ok_or_else(|| "OPA request channel closed".into())
}
async fn wait_for_validation(&mut self) -> TestResult {
timeout(Duration::from_secs(5), self.validation_started.recv())
.await?
.ok_or_else(|| "OPA validation channel closed".into())
}
fn release_validation(&self) {
if let Some(release) = &self.validation_release {
release.notify_one();
}
}
}
impl Drop for OpaMock {
fn drop(&mut self) {
self.task.abort();
}
}
#[tokio::test]
#[serial]
async fn test_sts_query_responses_are_aws_sdk_compatible() -> TestResult {
@@ -359,7 +155,7 @@ async fn test_sts_query_responses_are_aws_sdk_compatible() -> TestResult {
"signature rejection should include a request ID"
);
assert_access_denied(
assert_chaining_denied(
&sts_client(
&env.url,
temporary.access_key_id(),
@@ -371,187 +167,7 @@ async fn test_sts_query_responses_are_aws_sdk_compatible() -> TestResult {
.await?;
let (service_access_key, service_secret_key) = create_root_service_account(&env).await?;
assert_access_denied(
&sts_client(&env.url, &service_access_key, &service_secret_key, None),
"service-account denial",
)
.await?;
let implicit_user = "stsimplicit";
let explicit_allow_user = "stsallow";
let explicit_deny_user = "stsdeny";
let policyless_user = "stspolicyless";
let secret = "stsAuthzSecret123";
create_user(&env, policyless_user, secret).await?;
assert_access_denied(&sts_client(&env.url, policyless_user, secret, None), "policyless user").await?;
create_user_with_policy(
&env,
implicit_user,
secret,
"sts-implicit-policy",
serde_json::json!([{
"Effect": "Allow",
"Action": ["s3:ListAllMyBuckets"],
"Resource": ["arn:aws:s3:::*"],
}]),
)
.await?;
create_user_with_policy(
&env,
explicit_allow_user,
secret,
"sts-allow-policy",
serde_json::json!([{
"Effect": "Allow",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"],
}]),
)
.await?;
create_user_with_policy(
&env,
explicit_deny_user,
secret,
"sts-deny-policy",
serde_json::json!([
{
"Effect": "Allow",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"],
},
{
"Effect": "Deny",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"],
}
]),
)
.await?;
for user in [implicit_user, explicit_allow_user] {
let output = sts_client(&env.url, user, secret, None)
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/test")
.role_session_name("sts-authz-e2e")
.send()
.await
.map_err(|error| format!("{user} should be allowed to call AssumeRole: {error:?}"))?;
let credentials = output
.credentials()
.ok_or_else(|| format!("{user} AssumeRole response should contain credentials"))?;
assert!(!credentials.access_key_id().is_empty());
assert!(!credentials.secret_access_key().is_empty());
assert!(!credentials.session_token().is_empty());
}
assert_access_denied(&sts_client(&env.url, explicit_deny_user, secret, None), "explicit sts:AssumeRole Deny").await?;
env.stop_server();
Ok(())
}
#[tokio::test]
#[serial]
async fn test_sts_assume_role_opa_contract() -> TestResult {
init_logging();
let mut opa = OpaMock::start().await?;
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(
vec![],
&[
("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
],
)
.await?;
let secret = "stsOpaSecret123";
create_user_with_policy(
&env,
"opaallow",
secret,
"sts-opa-local-deny-policy",
serde_json::json!([{
"Effect": "Deny",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"],
}]),
)
.await?;
create_user_with_policy(
&env,
"opadeny",
secret,
"sts-opa-local-allow-policy",
serde_json::json!([{
"Effect": "Allow",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"],
}]),
)
.await?;
sts_client(&env.url, "opaallow", secret, None)
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/test")
.role_session_name("sts-opa-contract")
.send()
.await
.map_err(|error| format!("OPA allow should override the local explicit Deny: {error:?}"))?;
assert_access_denied(
&sts_client(&env.url, "opadeny", secret, None),
"OPA denial despite local sts:AssumeRole Allow",
)
.await?;
let mut accounts = BTreeSet::new();
for _ in 0..2 {
let request = opa.next_request().await?;
assert_eq!(request.pointer("/input/action").and_then(Value::as_str), Some("sts:AssumeRole"));
assert_eq!(request.pointer("/input/context/deny_only").and_then(Value::as_bool), Some(true));
let account = request
.pointer("/input/identity/account")
.and_then(Value::as_str)
.ok_or("OPA input should include identity.account")?;
accounts.insert(account.to_owned());
}
assert_eq!(accounts, BTreeSet::from(["opaallow".to_owned(), "opadeny".to_owned()]));
env.stop_server();
Ok(())
}
#[tokio::test]
#[serial]
async fn test_sts_assume_role_fails_closed_while_opa_is_unavailable() -> TestResult {
init_logging();
let mut opa = OpaMock::start_delayed_unavailable().await?;
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str())])
.await?;
opa.wait_for_validation().await?;
let user = "opaunavailable";
let secret = "stsOpaUnavailableSecret123";
create_user_with_policy(
&env,
user,
secret,
"sts-opa-unavailable-local-policy",
serde_json::json!([{
"Effect": "Allow",
"Action": ["s3:ListAllMyBuckets"],
"Resource": ["arn:aws:s3:::*"],
}]),
)
.await?;
assert_access_denied(&sts_client(&env.url, user, secret, None), "configured OPA initialization").await?;
opa.release_validation();
tokio::time::sleep(Duration::from_millis(200)).await;
assert_access_denied(&sts_client(&env.url, user, secret, None), "configured OPA validation failure").await?;
assert_chaining_denied(&sts_client(&env.url, &service_access_key, &service_secret_key, None), "service account").await?;
env.stop_server();
Ok(())
+5 -86
View File
@@ -33,98 +33,14 @@ workspace = true
[features]
default = []
rio-v2 = ["dep:rustfs-rio-v2"]
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"hotpath/async-channel",
"hotpath/parking_lot",
"hotpath/reqwest-0-13",
"rustfs-checksums/hotpath",
"rustfs-common/hotpath",
"rustfs-concurrency/hotpath",
"rustfs-config/hotpath",
"rustfs-credentials/hotpath",
"rustfs-data-usage/hotpath",
"rustfs-filemeta/hotpath",
"rustfs-io-metrics/hotpath",
"rustfs-lifecycle/hotpath",
"rustfs-lock/hotpath",
"rustfs-madmin/hotpath",
"rustfs-object-capacity/hotpath",
"rustfs-policy/hotpath",
"rustfs-protos/hotpath",
"rustfs-replication/hotpath",
"rustfs-rio/hotpath",
"rustfs-rio-v2?/hotpath",
"rustfs-s3-types/hotpath",
"rustfs-signer/hotpath",
"rustfs-storage-api/hotpath",
"rustfs-tls-runtime/hotpath",
"rustfs-utils/hotpath",
"rustfs-crypto/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-checksums/hotpath-alloc",
"rustfs-common/hotpath-alloc",
"rustfs-concurrency/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-credentials/hotpath-alloc",
"rustfs-data-usage/hotpath-alloc",
"rustfs-filemeta/hotpath-alloc",
"rustfs-io-metrics/hotpath-alloc",
"rustfs-lifecycle/hotpath-alloc",
"rustfs-lock/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-object-capacity/hotpath-alloc",
"rustfs-policy/hotpath-alloc",
"rustfs-protos/hotpath-alloc",
"rustfs-replication/hotpath-alloc",
"rustfs-rio/hotpath-alloc",
"rustfs-rio-v2?/hotpath-alloc",
"rustfs-s3-types/hotpath-alloc",
"rustfs-signer/hotpath-alloc",
"rustfs-storage-api/hotpath-alloc",
"rustfs-tls-runtime/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
"rustfs-crypto/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-checksums/hotpath-cpu",
"rustfs-common/hotpath-cpu",
"rustfs-concurrency/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-credentials/hotpath-cpu",
"rustfs-data-usage/hotpath-cpu",
"rustfs-filemeta/hotpath-cpu",
"rustfs-io-metrics/hotpath-cpu",
"rustfs-lifecycle/hotpath-cpu",
"rustfs-lock/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-object-capacity/hotpath-cpu",
"rustfs-policy/hotpath-cpu",
"rustfs-protos/hotpath-cpu",
"rustfs-replication/hotpath-cpu",
"rustfs-rio/hotpath-cpu",
"rustfs-rio-v2?/hotpath-cpu",
"rustfs-s3-types/hotpath-cpu",
"rustfs-signer/hotpath-cpu",
"rustfs-storage-api/hotpath-cpu",
"rustfs-tls-runtime/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
"rustfs-crypto/hotpath-cpu",
]
hotpath = ["dep:hotpath", "hotpath/hotpath", "rustfs-filemeta/hotpath", "rustfs-rio/hotpath"]
# Exposes shared lifecycle/tier test utilities (MockWarmBackend, fault
# injection, xl.meta transition assertions) via `api::tier::test_util`.
# Enable only from `[dev-dependencies]` (rustfs/backlog#1148 ilm-6).
test-util = []
[dependencies]
hotpath.workspace = true
hotpath = { workspace = true, optional = true }
rustfs-filemeta.workspace = true
rustfs-utils = { workspace = true, features = ["full"] }
rustfs-rio.workspace = true
@@ -141,6 +57,7 @@ rustfs-policy.workspace = true
rustfs-protos.workspace = true
rustfs-replication.workspace = true
rustfs-lifecycle.workspace = true
rustfs-kms.workspace = true
rustfs-s3-types = { workspace = true }
rustfs-data-usage.workspace = true
rustfs-object-capacity.workspace = true
@@ -208,6 +125,8 @@ libc.workspace = true
rustix = { workspace = true, features = ["process", "fs"] }
rustfs-madmin.workspace = true
reqwest = { workspace = true }
aes-gcm = { workspace = true, features = ["rand_core"] }
chacha20poly1305.workspace = true
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
urlencoding = { workspace = true }
smallvec = { workspace = true, features = ["serde"] }
+10 -13
View File
@@ -267,13 +267,12 @@ pub mod config {
pub mod com {
pub use crate::config::com::{
COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config,
is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
save_server_config_no_lock, save_server_config_snapshot, save_server_config_snapshot_with_generation,
server_config_path, try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock,
with_server_config_read_lock, with_server_config_write_lock,
ServerConfigCorruptError, ServerConfigSnapshot, delete_config, is_server_config_corrupt_error, lookup_configs,
read_config, read_config_no_lock, read_config_with_metadata, read_config_without_migrate,
read_config_without_migrate_no_lock, read_existing_server_config_no_lock, read_server_config_snapshot, save_config,
save_config_no_lock, save_config_with_opts, save_server_config, save_server_config_no_lock,
save_server_config_snapshot, server_config_path, try_migrate_server_config, with_config_object_read_lock,
with_config_object_write_lock, with_server_config_read_lock, with_server_config_write_lock,
};
}
@@ -392,12 +391,10 @@ pub mod notification {
pub mod object {
pub use crate::object_api::{
BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, ObjectEncryptionResolver, ObjectInfo,
ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode,
ReadEncryptionRequest, StreamConsumer, get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook,
register_get_object_body_cache_hook, register_object_mutation_hook, unregister_get_object_body_cache_hook,
unregister_object_mutation_hook,
BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectBodyCacheHookLookup, GetObjectBodySource,
GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, StreamConsumer,
get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
};
pub use crate::store::PreparedGetObjectReader;
}
@@ -1278,6 +1278,12 @@ fn build_remove_object_headers(version_id: Option<&str>, opts: &RemoveObjectOpti
fn resolve_delete_api_version_id(version_id: Option<String>, opts: &RemoveObjectOptions) -> Option<String> {
if opts.replication_request && opts.replication_delete_marker {
None
} else if opts.replication_request
&& version_id
.as_deref()
.is_some_and(|version_id| Uuid::parse_str(version_id).is_ok_and(|version_id| version_id.is_nil()))
{
Some("null".to_string())
} else {
version_id
}
@@ -2259,6 +2265,12 @@ mod tests {
assert_eq!(got.as_deref(), Some(vid.as_str()));
}
#[test]
fn null_version_purge_sends_s3_null_version_id() {
let got = resolve_delete_api_version_id(Some(Uuid::nil().to_string()), &remove_opts(true, false));
assert_eq!(got.as_deref(), Some("null"));
}
#[test]
fn delete_marker_propagation_omits_versionid_query_param() {
// Propagating a delete-marker CREATION (delete_marker=true): the target
+6 -93
View File
@@ -579,26 +579,8 @@ pub struct BucketMetadataSys {
/// Serializes metadata-map commits and their derived cache updates for one
/// bucket. Namespace locks, when present, are acquired before this lock.
metadata_publish_locks: Arc<MetadataPublishLockRegistry>,
/// Deduplicates concurrent lazy loads of one bucket's metadata, so N
/// simultaneous cache misses issue a single disk read instead of N.
///
/// This is the `singleflight` that upstream applies to its own lazy
/// `GetConfig`. Without it the namespace *read* lock the load holds is no
/// help: read locks are shared, so it excludes concurrent config writers
/// but not concurrent readers, and every caller still pays a full
/// erasure-set metadata fanout. A separate registry from
/// `metadata_publish_locks`, reusing the same per-bucket lock machinery.
///
/// Lock order: this lock, then the namespace lock, then the publish lock,
/// then the metadata map. It is only ever taken as the first of those, so
/// it cannot invert against a path that already holds one of the others.
lazy_load_locks: Arc<MetadataPublishLockRegistry>,
#[cfg(test)]
lazy_load_lock_probe: std::sync::atomic::AtomicBool,
/// Counts disk loads taken by the lazy `get_config` path, so a test can
/// prove concurrent misses collapse into one.
#[cfg(test)]
lazy_disk_loads: std::sync::atomic::AtomicUsize,
/// Buckets recently observed to have no persisted metadata. Serving the
/// fabricated default from here (instead of re-reading disk) keeps the
/// per-request cost of repeated lookups for such names bounded — without
@@ -617,13 +599,8 @@ impl BucketMetadataSys {
metadata_publish_locks: Arc::new(MetadataPublishLockRegistry {
locks: StdMutex::new(HashMap::new()),
}),
lazy_load_locks: Arc::new(MetadataPublishLockRegistry {
locks: StdMutex::new(HashMap::new()),
}),
#[cfg(test)]
lazy_load_lock_probe: std::sync::atomic::AtomicBool::new(false),
#[cfg(test)]
lazy_disk_loads: std::sync::atomic::AtomicUsize::new(0),
absent_metadata: moka::future::Cache::builder()
.max_capacity(ABSENT_BUCKET_METADATA_MAX_ENTRIES)
.time_to_live(ABSENT_BUCKET_METADATA_TTL)
@@ -638,22 +615,16 @@ impl BucketMetadataSys {
}
fn metadata_publish_lock(&self, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
Self::bucket_lock_in(&self.metadata_publish_locks, bucket)
}
/// Per-bucket gate for the lazy `get_config` disk load. See
/// [`Self::lazy_load_locks`].
fn lazy_load_lock(&self, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
Self::bucket_lock_in(&self.lazy_load_locks, bucket)
}
fn bucket_lock_in(registry: &Arc<MetadataPublishLockRegistry>, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
let mut locks = registry.locks.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let mut locks = self
.metadata_publish_locks
.locks
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
locks.get(bucket).and_then(Weak::upgrade).unwrap_or_else(|| {
let lock = Arc::new_cyclic(|lock| {
Mutex::new(MetadataPublishLockState {
bucket: bucket.to_string(),
registry: Arc::downgrade(registry),
registry: Arc::downgrade(&self.metadata_publish_locks),
lock: lock.clone(),
})
});
@@ -1108,27 +1079,6 @@ impl BucketMetadataSys {
return Ok((Arc::new(bm), true));
}
// Collapse concurrent misses for this bucket into one disk load.
// Taken before the namespace lock — see `lazy_load_locks` for the
// ordering rule.
let load_lock = self.lazy_load_lock(bucket);
let _load_guard = load_lock.lock_owned().await;
// Re-check both caches: whoever held the gate before us may have
// already answered this exact question, and repeating the fanout
// is the whole cost this gate exists to avoid.
if let Some(bm) = self.metadata_map.read().await.get(bucket).cloned() {
return Ok((bm, true));
}
if self.absent_metadata.get(bucket).await.is_some() {
let mut bm = BucketMetadata::new(bucket);
bm.default_timestamps();
return Ok((Arc::new(bm), true));
}
#[cfg(test)]
self.lazy_disk_loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let lock = self.api.new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
#[cfg(test)]
@@ -1481,43 +1431,6 @@ mod tests {
use serial_test::serial;
use tokio::time::timeout;
/// Concurrent cache misses for one bucket must collapse into a single disk
/// load.
///
/// The namespace read lock the lazy path already holds does not provide
/// this: read locks are shared, so it excludes concurrent config writers
/// but not concurrent readers. Without the dedup gate every caller pays its
/// own namespace-lock acquisition plus a full erasure-set metadata fanout —
/// and the paths that reach `get_config` are per-request, so the multiplier
/// is request concurrency.
#[tokio::test]
async fn concurrent_lazy_loads_of_one_bucket_issue_a_single_disk_read() {
use std::sync::atomic::Ordering;
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = Arc::new(BucketMetadataSys::new(ecstore));
// A name with no persisted metadata: every caller misses the map, and
// the absent-cache entry does not exist until the first load records it.
let bucket = "singleflight-bucket";
let waiters = 8;
let results = futures::future::join_all((0..waiters).map(|_| {
let sys = Arc::clone(&sys);
async move { sys.get_config(bucket).await.map(|(bm, _)| bm.name.clone()) }
}))
.await;
for result in results {
assert_eq!(result.expect("every caller must get an answer"), bucket);
}
assert_eq!(
sys.lazy_disk_loads.load(Ordering::Relaxed),
1,
"concurrent misses for one bucket must share a single disk load"
);
}
/// Pins the fail-closed caching contract of the lazy `get_config` path
/// and the refresh no-replace rule: fabricated defaults are returned but
/// never served by the map-only `get()`, persisted metadata is cached on
@@ -13,6 +13,6 @@
// limitations under the License.
pub use rustfs_replication::{
ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, replication_target_arns,
should_remove_replication_target, validate_replication_config_target_arns,
ObjectOpts, ReplicationConfigurationExt, ReplicationTargetValidationError, delete_replication_target_arns,
replication_target_arns, should_remove_replication_target, validate_replication_config_target_arns,
};
@@ -28,7 +28,7 @@ use super::replication_logging::{EVENT_RESYNC_CONFIG_LOOKUP_SKIPPED, LOG_COMPONE
use super::replication_metadata_boundary::ReplicationMetadataStore;
use super::replication_object_decision_boundary::{
MustReplicateOptions, ReplicationDeleteSource, ReplicationResyncTargetObject, delete_replication_missing_source_decision,
delete_replication_object_opts, resync_target_for_object,
delete_replication_object_opts, delete_replication_version_id, resync_target_for_object,
};
use super::replication_storage_boundary::{ObjectInfo, ObjectOptions, ObjectToDelete, object_to_delete_for_replication};
use super::replication_target_boundary::{BucketTargets, ReplicationTargetStore};
@@ -73,7 +73,7 @@ impl ReplicationConfig {
if oi.delete_marker {
let opts = ObjectOpts {
name: oi.name.clone(),
version_id: oi.version_id,
version_id: delete_replication_version_id(oi.delete_marker, oi.version_id, !oi.version_purge_status.is_empty()),
delete_marker: true,
op_type: ReplicationType::Delete,
existing_object: true,
@@ -299,8 +299,14 @@ pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplic
#[cfg(test)]
mod tests {
use s3s::dto::{Destination, ReplicationRule, ReplicationRuleStatus};
use s3s::dto::{
DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DeleteReplicationStatus, Destination,
ReplicationRule, ReplicationRuleStatus,
};
use uuid::Uuid;
use super::super::replication_filemeta_boundary::VersionPurgeStatusType;
use super::super::replication_target_boundary::BucketTarget;
use super::*;
fn replication_rule() -> ReplicationRule {
@@ -360,4 +366,56 @@ mod tests {
assert!(options.is_replication_request());
assert_eq!(options.user_tags(), "env=prod");
}
#[tokio::test]
async fn resync_distinguishes_delete_marker_creation_from_version_purge() {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule();
rule.destination.bucket = arn.to_string();
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
});
rule.delete_replication = Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED),
});
let config = ReplicationConfig::new(
Some(ReplicationConfiguration {
role: String::new(),
rules: vec![rule],
}),
Some(BucketTargets {
targets: vec![BucketTarget {
arn: arn.to_string(),
endpoint: "target.example".to_string(),
..Default::default()
}],
}),
);
let marker_version_id = Uuid::new_v4();
let marker = ObjectInfo {
bucket: "source".to_string(),
name: "object".to_string(),
delete_marker: true,
version_id: Some(marker_version_id),
replication_status: ReplicationStatusType::Pending,
..Default::default()
};
let creation = config
.resync(marker.clone(), ReplicateDecision::default(), &HashMap::new())
.await;
assert!(creation.targets.contains_key(arn));
let purge = config
.resync(
ObjectInfo {
version_purge_status: VersionPurgeStatusType::Pending,
..marker
},
ReplicateDecision::default(),
&HashMap::new(),
)
.await;
assert!(purge.targets.is_empty());
}
}
@@ -13,14 +13,14 @@
// limitations under the License.
pub use rustfs_replication::{
MustReplicateOptions, ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, delete_replication_state_from_config,
delete_replication_version_id, should_schedule_delete_replication, should_use_existing_delete_replication_info,
should_use_existing_delete_replication_source,
MustReplicateOptions, ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, delete_replication_parts,
delete_replication_state_from_config, delete_replication_version_id, should_schedule_delete_replication,
should_use_existing_delete_replication_info, should_use_existing_delete_replication_source,
};
pub(crate) use rustfs_replication::{
ReplicationDeleteSource, ReplicationMultipartPartInput, ReplicationResyncTargetObject,
delete_replication_missing_source_decision, delete_replication_object_opts, heal_uses_delete_replication_path,
is_retryable_delete_replication_head_error, is_version_delete_replication, replication_etags_match,
replication_multipart_complete_actual_size, replication_multipart_part_plan, resync_target_for_object,
should_retry_delete_marker_purge,
should_retry_delete_marker_purge, version_purge_target_missing,
};
@@ -67,6 +67,7 @@ const EVENT_REPLICATION_BACKPRESSURE: &str = "replication_backpressure";
const EVENT_REPLICATION_RESYNC_LOAD_SKIPPED: &str = "replication_resync_load_skipped";
const EVENT_REPLICATION_RESYNC_RECOVERED: &str = "replication_resync_recovered";
const EVENT_REPLICATION_MRF_QUEUE_UNAVAILABLE: &str = "replication_mrf_queue_unavailable";
const EVENT_REPLICATION_MRF_ENTRY_SKIPPED: &str = "replication_mrf_entry_skipped";
#[derive(Debug, Default)]
pub struct DurableMrfBacklog {
@@ -660,6 +661,21 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
for entry in entries.iter() {
match entry.op {
MrfOpKind::Delete => {
let Some(delete_parts) = entry.delete_parts_for_replay() else {
debug!(
event = EVENT_REPLICATION_MRF_ENTRY_SKIPPED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
bucket = %entry.bucket,
object = %entry.object,
reason = "invalid_delete_version_ids",
"Skipped invalid persisted replication delete"
);
continue;
};
let version_purge_id = delete_parts
.version_id
.or_else(|| delete_parts.delete_marker_version_id.filter(|_| !delete_parts.delete_marker));
// Reconstruct a heal delete and re-queue it. We do NOT call
// get_object_info here because the delete-marker or version may
// already be absent from the local store — that is expected.
@@ -674,15 +690,20 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
let oi = ObjectInfo {
bucket: entry.bucket.clone(),
name: entry.object.clone(),
version_id: entry.version_id,
delete_marker: entry.delete_marker,
version_id: version_purge_id,
delete_marker: delete_parts.delete_marker,
replication_status: if entry.replica {
ReplicationStatusType::Replica
} else {
ReplicationStatusType::Empty
},
..Default::default()
};
let dsc = check_replicate_delete(
&entry.bucket,
&ObjectToDelete {
object_name: entry.object.clone(),
version_id: entry.version_id,
version_id: version_purge_id,
..Default::default()
},
&oi,
@@ -708,9 +729,9 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
let dv = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: entry.object.clone(),
version_id: entry.version_id,
delete_marker_version_id: entry.delete_marker_version_id,
delete_marker: entry.delete_marker,
version_id: delete_parts.version_id,
delete_marker_version_id: delete_parts.delete_marker_version_id,
delete_marker: delete_parts.delete_marker,
delete_marker_mtime,
replication_state: Some(rstate),
..Default::default()
@@ -2333,6 +2354,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
};
let second = MrfReplicateEntry {
@@ -2446,6 +2468,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
};
@@ -2479,6 +2502,7 @@ mod tests {
op: MrfOpKind::Delete,
delete_marker_version_id: Some(dm_vid),
delete_marker: true,
replica: false,
delete_marker_mtime: Some(mtime_nanos),
};
@@ -2512,6 +2536,7 @@ mod tests {
op: MrfOpKind::Delete,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
};
@@ -2540,6 +2565,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
},
MrfReplicateEntry {
@@ -2551,6 +2577,7 @@ mod tests {
op: MrfOpKind::Delete,
delete_marker_version_id: Some(del_dm_vid),
delete_marker: true,
replica: false,
delete_marker_mtime: None,
},
];
@@ -2580,6 +2607,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
};
assert_eq!(obj_entry.op, MrfOpKind::Object);
@@ -2594,6 +2622,7 @@ mod tests {
op: MrfOpKind::Delete,
delete_marker_version_id: Some(Uuid::new_v4()),
delete_marker: true,
replica: false,
delete_marker_mtime: None,
};
assert_eq!(del_entry.op, MrfOpKind::Delete);
@@ -2609,6 +2638,7 @@ mod tests {
op: MrfOpKind::default(),
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
};
assert_eq!(legacy_entry.op, MrfOpKind::Object, "legacy default must be Object");
@@ -2672,6 +2702,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
}];
let encoded = encode_mrf_file(&entries).expect("durable MRF backlog should encode");
@@ -2702,6 +2733,7 @@ mod tests {
op: MrfOpKind::Object,
delete_marker_version_id: None,
delete_marker: false,
replica: false,
delete_marker_mtime: None,
}])
.expect("invalid persisted entry should still encode for boundary testing");
@@ -13,7 +13,7 @@
// limitations under the License.
use super::replication_bandwidth_boundary;
use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt as _};
use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt as _, delete_replication_target_arns};
use super::replication_config_store::ReplicationConfigStore;
use super::replication_error_boundary::{Result, is_err_object_not_found, is_err_version_not_found};
use super::replication_event_sink::{EventArgs, send_event, send_local_event};
@@ -29,9 +29,10 @@ use super::replication_metadata_boundary::ReplicationMetadataStore;
use super::replication_msgp_boundary::ReplicationMsgpCodec;
use super::replication_object_config::{ReplicationConfig, check_replicate_delete, get_replication_config, must_replicate};
use super::replication_object_decision_boundary::{
MustReplicateOptions, ReplicationMultipartPartInput, heal_uses_delete_replication_path,
MustReplicateOptions, ReplicationMultipartPartInput, delete_replication_parts, heal_uses_delete_replication_path,
is_retryable_delete_replication_head_error, is_version_delete_replication, replication_etags_match,
replication_multipart_complete_actual_size, replication_multipart_part_plan, should_retry_delete_marker_purge,
version_purge_target_missing,
};
use super::replication_queue_boundary::DeletedObjectReplicationInfo;
use super::replication_resync_boundary::ResyncStatusType;
@@ -49,7 +50,7 @@ use super::replication_target_boundary::{
PutObjectOptions, PutObjectPartOptions, ReplicationTargetStore, TargetClient, replication_action_for_target_head,
replication_complete_multipart_options, replication_delete_marker_purge_remove_options, replication_delete_remove_options,
replication_force_delete_remove_options, replication_object_is_ssec_encrypted, replication_put_object_header_size,
replication_put_object_options, replication_target_head_is_newer_null_version,
replication_put_object_options, replication_target_head_is_newer_null_version, replication_target_version_id,
};
use super::replication_versioning_boundary::ReplicationVersioningStore;
use super::runtime_boundary as runtime_sources;
@@ -70,9 +71,8 @@ use rustfs_utils::http::{
AMZ_TAGGING_DIRECTIVE, SUFFIX_REPLICATION_RESET, SUFFIX_REPLICATION_STATUS, has_internal_suffix, insert_str,
};
use rustfs_utils::{DEFAULT_SIP_HASH_KEY, sip_hash};
#[cfg(test)]
use s3s::dto::ReplicationConfiguration;
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use std::fmt::Display;
use std::sync::Arc;
use time::OffsetDateTime;
@@ -97,6 +97,8 @@ const EVENT_RESYNC_TASK_FAILED: &str = "replication_resync_task_failed";
const EVENT_RESYNC_TARGET_OPERATION_FAILED: &str = "replication_resync_target_operation_failed";
const EVENT_RESYNC_RUNTIME_CHANNEL_FAILED: &str = "replication_resync_runtime_channel_failed";
const ERR_REPLICATION_METADATA_COPY_UNSUPPORTED: &str = "metadata-only replication is not implemented";
const ERR_VERSION_PURGE_TARGET_STILL_EXISTS: &str = "target version still exists after replication purge";
const ERR_VERSION_PURGE_MISSING_VERSION_ID: &str = "version purge record is missing version ID";
const REPLICATION_TARGET_OFFLINE_ERROR_MARKERS: &[&str] = &[
"dispatch failure",
"timeouterror",
@@ -786,19 +788,46 @@ impl ReplicationResyncer {
}
if roi.delete_marker || !roi.version_purge_status.is_empty() {
let (version_id, dm_version_id) = if roi.version_purge_status.is_empty() {
(None, roi.version_id)
} else {
(roi.version_id, None)
let Some(parts) =
delete_replication_parts(roi.delete_marker, roi.version_id, !roi.version_purge_status.is_empty())
else {
debug!(
event = EVENT_RESYNC_OBJECT_PROCESSED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket_name,
object = %roi.name,
reason = "version_purge_missing_version_id",
"Failed to process resync version purge"
);
let status = TargetReplicationResyncStatus {
bucket: roi.bucket.clone(),
object: roi.name.clone(),
failed_count: 1,
error: Some(ERR_VERSION_PURGE_MISSING_VERSION_ID.to_string()),
..Default::default()
};
if let Err(err) = results_tx.send(status).await {
error!(
event = EVENT_RESYNC_RUNTIME_CHANNEL_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket_name,
reason = "status_channel_send_failed",
error = %err,
"Failed to send resync status"
);
}
continue;
};
let doi = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: roi.name.clone(),
delete_marker_version_id: dm_version_id,
version_id,
delete_marker_version_id: parts.delete_marker_version_id,
version_id: parts.version_id,
replication_state: roi.replication_state.clone(),
delete_marker: roi.delete_marker,
delete_marker: parts.delete_marker,
delete_marker_mtime: roi.mod_time,
..Default::default()
},
@@ -821,21 +850,41 @@ impl ReplicationResyncer {
};
let reset_id = target_client.reset_id.clone();
let is_version_purge = !roi.version_purge_status.is_empty();
let head_result = head_object_with_proxy_stats(
&bucket_name,
target_client.as_ref(),
&target_client.bucket,
&roi.name,
roi.version_id.map(|v| v.to_string()),
replication_target_version_id(roi.version_id, is_version_purge),
)
.await;
let (size, err) = match head_result {
Ok(_) if is_version_purge => {
st.failed_count += 1;
(0, Some(ERR_VERSION_PURGE_TARGET_STILL_EXISTS.to_string()))
}
Ok(_) => {
st.replicated_count += 1;
st.replicated_size += roi.size;
(roi.size, None)
}
Err(err) if is_version_purge => {
let (is_not_found, code) = err
.as_service_error()
.map(|service_err| (service_err.is_not_found(), service_err.code()))
.unwrap_or((false, None));
let raw_status = err.raw_response().map(|response| response.status().as_u16());
let missing = version_purge_target_missing(is_not_found, code, raw_status);
if missing {
st.replicated_count += 1;
(0, None)
} else {
st.failed_count += 1;
(0, resync_target_error_detail(&err))
}
}
Err(err) if roi.delete_marker => {
// Verifying a replicated delete marker: only a
// definitive 404/NoSuchKey or 405/MethodNotAllowed
@@ -852,7 +901,7 @@ impl ReplicationResyncer {
};
if retryable {
st.failed_count += 1;
(0, Some(err))
(0, resync_target_error_detail(&err))
} else {
st.replicated_count += 1;
(0, None)
@@ -871,17 +920,17 @@ impl ReplicationResyncer {
}
Ok(None) => {
st.failed_count += 1;
(0, Some(err))
(0, resync_target_error_detail(&err))
}
Err(e2) => {
st.failed_count += 1;
(0, Some(e2))
(0, resync_target_error_detail(&e2))
}
}
}
Err(err) => {
st.failed_count += 1;
(0, Some(err))
(0, resync_target_error_detail(&err))
}
};
@@ -911,7 +960,7 @@ impl ReplicationResyncer {
"Processed resync object"
);
}
st.error = err.as_ref().and_then(resync_target_error_detail);
st.error = err;
if cancel_token.is_cancelled() {
return;
@@ -1065,22 +1114,27 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
}
let dsc = if heal_uses_delete_replication_path(oi.delete_marker, &oi.version_purge_status) {
check_replicate_delete(
oi.bucket.as_str(),
&ObjectToDelete {
object_name: oi.name.clone(),
version_id: oi.version_id,
..Default::default()
},
&oi,
&ObjectOptions {
versioned: ReplicationVersioningStore::prefix_enabled(&oi.bucket, &oi.name).await,
version_suspended: ReplicationVersioningStore::prefix_suspended(&oi.bucket, &oi.name).await,
..Default::default()
},
None,
)
.await
match delete_replication_parts(oi.delete_marker, oi.version_id, !oi.version_purge_status.is_empty()) {
Some(parts) => {
check_replicate_delete(
oi.bucket.as_str(),
&ObjectToDelete {
object_name: oi.name.clone(),
version_id: parts.version_id,
..Default::default()
},
&oi,
&ObjectOptions {
versioned: ReplicationVersioningStore::prefix_enabled(&oi.bucket, &oi.name).await,
version_suspended: ReplicationVersioningStore::prefix_suspended(&oi.bucket, &oi.name).await,
..Default::default()
},
None,
)
.await
}
None => ReplicateDecision::default(),
}
} else {
must_replicate(
oi.bucket.as_str(),
@@ -1150,7 +1204,6 @@ pub async fn replicate_delete<S: ReplicationStorage>(dobj: DeletedObjectReplicat
} else {
dobj.delete_object.version_id
};
let _rcfg = match get_replication_config(&bucket).await {
Ok(Some(config)) => config,
Ok(None) => {
@@ -1469,8 +1522,8 @@ pub async fn replicate_delete<S: ReplicationStorage>(dobj: DeletedObjectReplicat
delete_marker_version_id,
)
.await
&& replicate_delete_marker_purge_to_targets(&bucket_clone, &dobj_clone, &dsc_clone).await
{
replicate_delete_marker_purge_to_targets(&bucket_clone, &dobj_clone, &dsc_clone).await;
break;
}
tokio::time::sleep(TokioDuration::from_secs(1)).await;
@@ -1600,9 +1653,50 @@ async fn source_delete_marker_missing<S: EcstoreObjectOperations>(
}
}
async fn replicate_delete_marker_purge_to_targets(bucket: &str, dobj: &DeletedObjectReplicationInfo, dsc: &ReplicateDecision) {
fn delete_marker_purge_target_arns(config: &ReplicationConfiguration, dobj: &DeletedObjectReplicationInfo) -> HashSet<String> {
let replica = dobj
.delete_object
.replication_state
.as_ref()
.is_some_and(|state| state.replica_status == ReplicationStatusType::Replica);
delete_replication_target_arns(config, &dobj.delete_object.object_name, replica)
}
#[derive(Debug, PartialEq, Eq)]
enum DeleteMarkerPurgeConfig {
Apply(HashSet<String>),
Stop,
Retry,
}
fn delete_marker_purge_config<E>(
result: std::result::Result<Option<ReplicationConfiguration>, E>,
dobj: &DeletedObjectReplicationInfo,
) -> DeleteMarkerPurgeConfig {
match result {
Ok(Some(config)) => DeleteMarkerPurgeConfig::Apply(delete_marker_purge_target_arns(&config, dobj)),
Ok(None) => DeleteMarkerPurgeConfig::Stop,
Err(_) => DeleteMarkerPurgeConfig::Retry,
}
}
async fn replicate_delete_marker_purge_to_targets(
bucket: &str,
dobj: &DeletedObjectReplicationInfo,
dsc: &ReplicateDecision,
) -> bool {
let Some(delete_marker_version_id) = dobj.delete_object.delete_marker_version_id else {
return;
return true;
};
let marker_creation_purge_targets = if dobj.delete_object.delete_marker {
match delete_marker_purge_config(get_replication_config(bucket).await, dobj) {
DeleteMarkerPurgeConfig::Apply(targets) => Some(targets),
DeleteMarkerPurgeConfig::Stop => return true,
DeleteMarkerPurgeConfig::Retry => return false,
}
} else {
None
};
for tgt_entry in dsc.targets_map.values() {
@@ -1612,6 +1706,12 @@ async fn replicate_delete_marker_purge_to_targets(bucket: &str, dobj: &DeletedOb
if !dobj.target_arn.is_empty() && dobj.target_arn != tgt_entry.arn {
continue;
}
if marker_creation_purge_targets
.as_ref()
.is_some_and(|targets| !targets.contains(&tgt_entry.arn))
{
continue;
}
let Some(tgt_client) = ReplicationTargetStore::remote_target_client(bucket, &tgt_entry.arn).await else {
continue;
};
@@ -1625,6 +1725,7 @@ async fn replicate_delete_marker_purge_to_targets(bucket: &str, dobj: &DeletedOb
)
.await;
}
true
}
async fn replicate_force_delete_to_targets<S: ReplicationStorage>(dobj: &DeletedObjectReplicationInfo, storage: Arc<S>) {
@@ -1851,10 +1952,10 @@ async fn replicate_force_delete_to_targets<S: ReplicationStorage>(dobj: &Deleted
}
async fn replicate_delete_to_target(dobj: &DeletedObjectReplicationInfo, tgt_client: Arc<TargetClient>) -> ReplicatedTargetInfo {
let version_id = if let Some(version_id) = &dobj.delete_object.delete_marker_version_id {
version_id.to_owned()
let version_id = if let Some(version_id) = dobj.delete_object.delete_marker_version_id {
Some(version_id)
} else {
dobj.delete_object.version_id.unwrap_or_default()
dobj.delete_object.version_id
};
let mut rinfo = dobj
@@ -1889,11 +1990,7 @@ async fn replicate_delete_to_target(dobj: &DeletedObjectReplicationInfo, tgt_cli
return rinfo;
}
let version_id = if version_id.is_nil() {
None
} else {
Some(version_id.to_string())
};
let version_id = replication_target_version_id(version_id, is_version_purge);
if dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() {
match head_object_with_proxy_stats(
@@ -3143,6 +3240,10 @@ async fn replicate_object_with_multipart<S: ReplicationObjectIO>(ctx: MultipartR
#[cfg(test)]
mod tests {
use super::*;
use s3s::dto::{
DeleteReplication, DeleteReplicationStatus, Destination, ReplicaModifications, ReplicaModificationsStatus,
ReplicationRule, ReplicationRuleAndOperator, ReplicationRuleFilter, ReplicationRuleStatus, SourceSelectionCriteria, Tag,
};
use std::collections::HashMap;
use time::OffsetDateTime;
use uuid::Uuid;
@@ -3452,6 +3553,110 @@ mod tests {
);
}
#[test]
fn test_delete_marker_purge_targets_follow_delete_and_replica_modification_rules() {
fn rule(arn: &str, delete_status: &'static str) -> ReplicationRule {
ReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(delete_status),
}),
destination: Destination {
bucket: arn.to_string(),
..Default::default()
},
existing_object_replication: None,
filter: None,
id: Some(arn.to_string()),
prefix: Some("logs/".to_string()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
}
}
let enabled_arn = "arn:rustfs:replication:us-east-1:target:enabled";
let disabled_arn = "arn:rustfs:replication:us-east-1:target:disabled";
let delete_marker_version_id = Uuid::new_v4();
let mut config = ReplicationConfiguration {
role: String::new(),
rules: vec![
rule(enabled_arn, DeleteReplicationStatus::ENABLED),
rule(disabled_arn, DeleteReplicationStatus::DISABLED),
],
};
let mut dobj = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: "logs/object.txt".to_string(),
delete_marker: true,
delete_marker_version_id: Some(delete_marker_version_id),
..Default::default()
},
..Default::default()
};
assert_eq!(delete_marker_purge_target_arns(&config, &dobj), HashSet::from([enabled_arn.to_string()]));
assert_eq!(
delete_marker_purge_config::<()>(Ok(Some(config.clone())), &dobj),
DeleteMarkerPurgeConfig::Apply(HashSet::from([enabled_arn.to_string()]))
);
dobj.delete_object.replication_state = Some(Default::default());
dobj.delete_object
.replication_state
.as_mut()
.expect("test replication state")
.replica_status = ReplicationStatusType::Replica;
assert!(delete_marker_purge_target_arns(&config, &dobj).is_empty());
config.rules[0].source_selection_criteria = Some(SourceSelectionCriteria {
replica_modifications: Some(ReplicaModifications {
status: ReplicaModificationsStatus::from_static(ReplicaModificationsStatus::ENABLED),
}),
sse_kms_encrypted_objects: None,
});
assert_eq!(delete_marker_purge_target_arns(&config, &dobj), HashSet::from([enabled_arn.to_string()]));
config.rules[0].prefix = None;
config.rules[0].filter = Some(ReplicationRuleFilter {
tag: Some(Tag {
key: Some("env".to_string()),
value: Some("prod".to_string()),
}),
..Default::default()
});
assert!(delete_marker_purge_target_arns(&config, &dobj).is_empty());
config.rules[0].filter = Some(ReplicationRuleFilter {
and: Some(ReplicationRuleAndOperator {
prefix: Some("logs/".to_string()),
tags: Some(vec![Tag {
key: Some("env".to_string()),
value: Some("prod".to_string()),
}]),
}),
..Default::default()
});
assert!(delete_marker_purge_target_arns(&config, &dobj).is_empty());
}
#[test]
fn test_delete_marker_purge_config_errors_are_retryable() {
let delete_marker_version_id = Uuid::new_v4();
let dobj = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: "object.txt".to_string(),
delete_marker: true,
delete_marker_version_id: Some(delete_marker_version_id),
..Default::default()
},
..Default::default()
};
assert_eq!(delete_marker_purge_config::<()>(Ok(None), &dobj), DeleteMarkerPurgeConfig::Stop);
assert_eq!(delete_marker_purge_config::<()>(Err(()), &dobj), DeleteMarkerPurgeConfig::Retry);
}
#[test]
fn test_is_retryable_delete_replication_head_error_allows_delete_marker_head_responses() {
assert!(
@@ -31,10 +31,13 @@ use rustfs_utils::http::{
};
use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
use uuid::Uuid;
pub(crate) use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient,
};
#[cfg(test)]
pub(crate) use crate::bucket::target::BucketTarget;
pub(crate) use crate::bucket::target::BucketTargets;
use super::replication_config_store::ReplicationConfigStore;
@@ -305,6 +308,14 @@ pub(crate) fn replication_target_head_is_newer_null_version(object_info: &Object
target_is_newer_than_source_null_version(&replication_source_object(object_info), &replication_target_object(target))
}
pub(crate) fn replication_target_version_id(version_id: Option<Uuid>, version_purge: bool) -> Option<String> {
match version_id {
Some(version_id) if version_id.is_nil() && version_purge => Some("null".to_string()),
Some(version_id) if !version_id.is_nil() => Some(version_id.to_string()),
_ => None,
}
}
pub(crate) fn replication_delete_remove_options(
delete_marker: bool,
replication_mtime: Option<OffsetDateTime>,
@@ -519,6 +530,18 @@ mod tests {
assert!(force.replication_request);
}
#[test]
fn replication_target_version_id_preserves_null_purges() {
assert_eq!(replication_target_version_id(Some(Uuid::nil()), true).as_deref(), Some("null"));
assert_eq!(replication_target_version_id(Some(Uuid::nil()), false), None);
let version_id = Uuid::new_v4();
assert_eq!(
replication_target_version_id(Some(version_id), true).as_deref(),
Some(version_id.to_string().as_str())
);
}
#[test]
fn replication_complete_multipart_options_sets_actual_size() {
let options = replication_complete_multipart_options("1024".to_string());
+15 -1
View File
@@ -46,6 +46,16 @@ lazy_static! {
m.insert("x-amz-replication-status".to_string(), true);
m
};
static ref SSE_HEADERS: HashMap<String, bool> = {
let mut m = HashMap::new();
m.insert("x-amz-server-side-encryption".to_string(), true);
m.insert("x-amz-server-side-encryption-aws-kms-key-id".to_string(), true);
m.insert("x-amz-server-side-encryption-context".to_string(), true);
m.insert("x-amz-server-side-encryption-customer-algorithm".to_string(), true);
m.insert("x-amz-server-side-encryption-customer-key".to_string(), true);
m.insert("x-amz-server-side-encryption-customer-key-md5".to_string(), true);
m
};
}
pub fn is_standard_query_value(qs_key: &str) -> bool {
@@ -60,12 +70,16 @@ pub fn is_standard_header(header_key: &str) -> bool {
*SUPPORTED_HEADERS.get(&header_key.to_lowercase()).unwrap_or(&false)
}
pub fn is_sse_header(header_key: &str) -> bool {
*SSE_HEADERS.get(&header_key.to_lowercase()).unwrap_or(&false)
}
pub fn is_amz_header(header_key: &str) -> bool {
let key = header_key.to_lowercase();
key.starts_with("x-amz-meta-")
|| key.starts_with("x-amz-grant-")
|| key == "x-amz-acl"
|| rustfs_utils::http::is_sse_header(header_key)
|| is_sse_header(header_key)
|| key.starts_with("x-amz-checksum-")
}
+53 -642
View File
@@ -53,14 +53,12 @@ use std::sync::LazyLock;
use std::sync::{Arc, RwLock};
use tokio::sync::{OwnedRwLockWriteGuard, RwLock as AsyncRwLock};
use tracing::{debug, error, info, instrument, warn};
use uuid::Uuid;
pub const CONFIG_PREFIX: &str = "config";
const SERVER_CONFIG_OBJECT: &str = "config/config.json";
const CONFIG_TRANSACTION_LOCK_SUFFIX: &str = ".transaction.lock";
// Server-config lock order: SERVER_CONFIG_LOCK -> transaction lock ->
// SERVER_CONFIG_OBJECT. Readers and writers must never reverse this order.
// Server-config lock order: SERVER_CONFIG_LOCK -> distributed namespace lock
// for SERVER_CONFIG_OBJECT. Readers and writers must never reverse this order.
static SERVER_CONFIG_LOCK: LazyLock<Arc<AsyncRwLock<()>>> = LazyLock::new(|| Arc::new(AsyncRwLock::new(())));
fn config_task_join_error(operation: &'static str, error: tokio::task::JoinError) -> Error {
@@ -78,11 +76,8 @@ where
T: Send + 'static,
{
tokio::spawn(async move {
// Lock order: SERVER_CONFIG_LOCK -> transaction lock -> object lock.
// Lock order: SERVER_CONFIG_LOCK -> namespace write lock.
let _local_guard = SERVER_CONFIG_LOCK.write().await;
let transaction_lock = server_config_transaction_lock_path();
let transaction_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &transaction_lock).await?;
let _transaction_guard = transaction_lock.get_write_lock(get_lock_acquire_timeout()).await?;
let namespace_lock = store.new_ns_lock(RUSTFS_META_BUCKET, SERVER_CONFIG_OBJECT).await?;
let _write_guard = namespace_lock.get_write_lock(get_lock_acquire_timeout()).await?;
Ok(operation().await)
@@ -101,11 +96,8 @@ where
T: Send + 'static,
{
tokio::spawn(async move {
// Lock order: SERVER_CONFIG_LOCK -> transaction lock -> object lock.
// Lock order: SERVER_CONFIG_LOCK -> namespace read lock.
let _local_guard = SERVER_CONFIG_LOCK.read().await;
let transaction_lock = server_config_transaction_lock_path();
let transaction_lock = store.new_ns_lock(RUSTFS_META_BUCKET, &transaction_lock).await?;
let _transaction_guard = transaction_lock.get_read_lock(get_lock_acquire_timeout()).await?;
let namespace_lock = store.new_ns_lock(RUSTFS_META_BUCKET, SERVER_CONFIG_OBJECT).await?;
let _read_guard = namespace_lock.get_read_lock(get_lock_acquire_timeout()).await?;
Ok(operation().await)
@@ -575,21 +567,6 @@ where
}
pub async fn save_config_with_opts<S>(api: Arc<S>, file: &str, data: Vec<u8>, opts: &ObjectOptions) -> Result<()>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
>,
{
save_config_with_opts_and_metadata(api, file, data, opts).await.map(|_| ())
}
async fn save_config_with_opts_and_metadata<S>(api: Arc<S>, file: &str, data: Vec<u8>, opts: &ObjectOptions) -> Result<ObjectInfo>
where
S: ObjectIO<
Error = Error,
@@ -602,13 +579,11 @@ where
>,
{
let mut put_data = PutObjReader::from_vec(data);
match api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts).await {
Ok(object_info) => Ok(object_info),
Err(err) => {
error!("save_config_with_opts: err: {:?}, file: {}", err, file);
Err(err)
}
if let Err(err) = api.put_object(RUSTFS_META_BUCKET, file, &mut put_data, opts).await {
error!("save_config_with_opts: err: {:?}, file: {}", err, file);
return Err(err);
}
Ok(())
}
fn new_server_config() -> Config {
@@ -619,12 +594,8 @@ async fn new_and_save_server_config<S>(api: Arc<S>) -> Result<Config>
where
S: EcstoreObjectIO + StorageAdminApi + NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
{
let snapshot = read_server_config_snapshot(api.clone()).await?;
if snapshot.object_exists() {
return Ok(snapshot.config.clone());
}
let cfg = new_server_config();
save_server_config_snapshot(api, &cfg, &snapshot).await?;
save_server_config(api, &cfg).await?;
Ok(cfg)
}
@@ -646,10 +617,6 @@ pub fn server_config_path() -> String {
SERVER_CONFIG_OBJECT.to_string()
}
fn server_config_transaction_lock_path() -> String {
format!("{}{CONFIG_TRANSACTION_LOCK_SUFFIX}", server_config_path())
}
fn storage_class_kvs_mut(cfg: &mut Config) -> &mut KVS {
let sub_cfg = cfg.0.entry(STORAGE_CLASS_SUB_SYS.to_string()).or_insert_with(|| {
let mut section = HashMap::new();
@@ -852,9 +819,6 @@ fn apply_external_scalar_config_map(
let Some(config_value) = root.get(descriptor.subsystem_key) else {
return Ok(false);
};
if descriptor.subsystem_key == HEAL_SUB_SYS && config_value.is_null() {
return Ok(false);
}
let overrides = decode_scalar_config_value(config_value, descriptor)?;
if overrides.is_empty() {
@@ -1499,142 +1463,21 @@ fn build_audit_object(cfg: &Config) -> Map<String, Value> {
build_target_object(cfg, &audit_target_descriptors())
}
fn sync_rendered_target_instance(existing: Value, rendered: Option<&Value>, valid_keys: &[&str]) -> Option<Value> {
match existing {
Value::Object(mut instance) => {
for key in valid_keys {
instance.remove(*key);
}
if let Some(Value::Object(rendered)) = rendered {
instance.extend(rendered.clone());
}
(!instance.is_empty()).then_some(Value::Object(instance))
}
Value::Array(entries) => {
let mut pending = rendered
.and_then(Value::as_object)
.map(|rendered| {
rendered
.iter()
.filter_map(|(key, value)| parse_target_scalar_value(key, value).map(|value| (key.clone(), value)))
.collect::<HashMap<_, _>>()
})
.unwrap_or_default();
let mut updated = Vec::with_capacity(entries.len().saturating_add(pending.len()));
for entry in entries {
let Some(entry_obj) = entry.as_object() else {
updated.push(entry);
continue;
};
let Some(key) = entry_obj.get("key").and_then(Value::as_str) else {
updated.push(entry);
continue;
};
if !valid_keys.contains(&key) {
updated.push(entry);
continue;
}
let Some(value) = pending.remove(key) else {
continue;
};
let mut entry_obj = entry_obj.clone();
entry_obj.insert("value".to_string(), Value::String(value));
updated.push(Value::Object(entry_obj));
}
updated.extend(rendered_scalar_config_kvs_entries(&pending));
(!updated.is_empty()).then_some(Value::Array(updated))
}
value if rendered.is_none() => Some(value),
_ => rendered.cloned(),
}
}
fn sync_rendered_target_object(
target_obj: &mut Map<String, Value>,
rendered_target: &Map<String, Value>,
descriptors: &[TargetConfigDescriptor],
) {
for descriptor in descriptors {
let existing = target_obj.remove(descriptor.external_key);
let alias = target_obj.remove(descriptor.subsystem_key);
let mut section = existing
.or(alias)
.and_then(|value| value.as_object().cloned())
.unwrap_or_default();
let rendered = rendered_target.get(descriptor.external_key).and_then(Value::as_object);
if is_target_instance_shorthand(&section, descriptor.valid_keys) {
let has_named_instances = rendered.is_some_and(|instances| instances.keys().any(|name| name != "default"));
if !has_named_instances {
if let Some(section) = sync_rendered_target_instance(
Value::Object(section),
rendered.and_then(|instances| instances.get("default")),
descriptor.valid_keys,
) {
target_obj.insert(descriptor.external_key.to_string(), section);
}
continue;
match rendered_target.get(descriptor.external_key) {
Some(Value::Object(v)) => {
target_obj.insert(descriptor.external_key.to_string(), Value::Object(v.clone()));
target_obj.remove(descriptor.subsystem_key);
}
let mut nested = Map::new();
if let Some(default) = sync_rendered_target_instance(
Value::Object(section),
rendered.and_then(|instances| instances.get("default")),
descriptor.valid_keys,
) {
nested.insert("default".to_string(), default);
_ => {
target_obj.remove(descriptor.external_key);
target_obj.remove(descriptor.subsystem_key);
}
if let Some(rendered) = rendered {
for (instance_name, instance) in rendered {
if instance_name != "default" {
nested.insert(instance_name.clone(), instance.clone());
}
}
}
if !nested.is_empty() {
target_obj.insert(descriptor.external_key.to_string(), Value::Object(nested));
}
continue;
}
if let Some(default_alias) = section.remove(DEFAULT_DELIMITER) {
if let Some(default) = section.get_mut("default") {
if let Some(alias) = sync_rendered_target_instance(default_alias, None, descriptor.valid_keys) {
match (default, alias) {
(Value::Object(default), Value::Object(alias)) => {
for (key, value) in alias {
default.entry(key).or_insert(value);
}
}
(Value::Array(default), Value::Array(alias)) => default.extend(alias),
_ => {}
}
}
} else {
section.insert("default".to_string(), default_alias);
}
}
let mut merged = Map::new();
for (instance_name, instance) in section {
if let Some(instance) = sync_rendered_target_instance(
instance,
rendered.and_then(|instances| instances.get(&instance_name)),
descriptor.valid_keys,
) {
merged.insert(instance_name, instance);
}
}
if let Some(rendered) = rendered {
for (instance_name, instance) in rendered {
if !merged.contains_key(instance_name) {
merged.insert(instance_name.clone(), instance.clone());
}
}
}
if !merged.is_empty() {
target_obj.insert(descriptor.external_key.to_string(), Value::Object(merged));
}
}
}
@@ -1653,14 +1496,6 @@ fn encode_server_config_blob(cfg: &Config, seed: Option<&[u8]>) -> Result<Vec<u8
Some(Value::Object(v)) => v,
_ => Map::new(),
};
for key in [
storageclass::CLASS_STANDARD,
storageclass::CLASS_RRS,
storageclass::OPTIMIZE,
storageclass::INLINE_BLOCK,
] {
sc_obj.remove(key);
}
for (k, v) in build_storageclass_object(cfg) {
sc_obj.insert(k, v);
}
@@ -1668,10 +1503,7 @@ fn encode_server_config_blob(cfg: &Config, seed: Option<&[u8]>) -> Result<Vec<u8
root.remove("storage_class");
for descriptor in [scanner_config_descriptor(), heal_config_descriptor()] {
let mut existing = root.remove(descriptor.subsystem_key);
if descriptor.subsystem_key == HEAL_SUB_SYS && existing.as_ref().is_some_and(Value::is_null) {
existing = None;
}
let existing = root.remove(descriptor.subsystem_key);
let rendered = build_scalar_config_object(cfg, descriptor);
if let Some(config_value) = sync_rendered_scalar_config_value(existing, &rendered, descriptor)? {
root.insert(descriptor.subsystem_key.to_string(), config_value);
@@ -1728,7 +1560,6 @@ fn is_standard_object_server_config(data: &[u8]) -> bool {
matches!(root.get("version"), Some(Value::String(v)) if !v.trim().is_empty())
&& matches!(root.get("storageclass"), Some(Value::Object(_)))
&& !root.contains_key("storage_class")
&& !matches!(root.get(HEAL_SUB_SYS), Some(Value::Null))
}
fn configs_semantically_equal(lhs: &Config, rhs: &Config) -> bool {
@@ -1762,7 +1593,7 @@ where
FileInfo = FileInfo,
ObjectToDelete = ObjectToDelete,
DeletedObject = DeletedObject,
> + NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
>,
{
if let Some(decrypt) = &decrypt_fn {
register_server_config_decrypt_fn(decrypt.clone());
@@ -1770,7 +1601,14 @@ where
let config_file = server_config_path();
match api
.get_object_info(RUSTFS_META_BUCKET, &config_file, &ObjectOptions::default())
.get_object_info(
RUSTFS_META_BUCKET,
&config_file,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
{
Ok(_) => {
@@ -1786,6 +1624,7 @@ where
let opts = ObjectOptions {
max_parity: true,
no_lock: true,
..Default::default()
};
@@ -1838,33 +1677,7 @@ where
}
};
let snapshot = match read_server_config_snapshot(api.clone()).await {
Ok(snapshot) => snapshot,
Err(err) => {
warn!("recheck target server config failed, skip migration: {:?}", err);
return;
}
};
if snapshot.object_exists() {
debug!("server config was created while legacy migration was preparing, skip migration");
return;
}
match save_config_with_opts(
api,
&config_file,
normalized,
&ObjectOptions {
max_parity: true,
http_preconditions: Some(HTTPPreconditions {
if_none_match: Some("*".to_string()),
..Default::default()
}),
..Default::default()
},
)
.await
{
match save_config(api, &config_file, normalized).await {
Ok(()) => {
info!("Migrated compatible server config from legacy metadata bucket");
}
@@ -1956,13 +1769,8 @@ where
{
let config_file = server_config_path();
// Try to read the configuration file.
let data = if namespace_lock_held {
read_config_no_lock(api.clone(), &config_file).await
} else {
read_config(api.clone(), &config_file).await
};
match data {
// Try to read the configuration file
match read_config_no_lock(api.clone(), &config_file).await {
Ok(data) => read_server_config(api, &data, namespace_lock_held).await,
Err(Error::ConfigNotFound) => handle_missing_config(api, "Read the main configuration", namespace_lock_held).await,
Err(err) => handle_config_read_error(err, &config_file),
@@ -1979,12 +1787,7 @@ where
warn!("Received empty configuration data, try to reread from '{}'", config_file);
// Try to read the configuration again
let data = if namespace_lock_held {
read_config_no_lock(api.clone(), &config_file).await
} else {
read_config(api.clone(), &config_file).await
};
match data {
match read_config_no_lock(api.clone(), &config_file).await {
Ok(cfg_data) => {
let cfg = decode_persisted_server_config(&cfg_data)?;
return Ok(cfg.merge());
@@ -2233,16 +2036,11 @@ pub struct ServerConfigSnapshot {
raw: Option<Vec<u8>>,
seed: Option<Vec<u8>>,
etag: Option<String>,
generation: Option<Uuid>,
_local_guard: OwnedRwLockWriteGuard<()>,
_guard: rustfs_lock::NamespaceLockGuard,
}
impl ServerConfigSnapshot {
pub fn object_exists(&self) -> bool {
self.raw.is_some()
}
pub fn ensure_lock_held(&self) -> Result<()> {
if self._guard.is_lock_lost() {
return Err(Error::other("server config transaction lock was lost"));
@@ -2253,34 +2051,12 @@ impl ServerConfigSnapshot {
pub fn is_lock_lost(&self) -> bool {
self._guard.is_lock_lost()
}
pub fn generation(&self) -> Option<Uuid> {
self.generation
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ServerConfigSaveResult {
persisted: bool,
generation: Option<Uuid>,
}
impl ServerConfigSaveResult {
pub fn persisted(&self) -> bool {
self.persisted
}
pub fn generation(&self) -> Option<Uuid> {
self.generation
}
}
/// Read a server config transaction snapshot while holding a dedicated
/// transaction lock. The config object's normal namespace lock remains
/// available to fence reads and the conditional write at commit time.
/// The transaction guard remains live until the snapshot is dropped,
/// serializing persistence and history ordering across admin nodes. Runtime
/// state is reloaded from the durable object after this guard is released.
/// Read a server config transaction snapshot while holding the same local and
/// distributed write locks used by every other server-config writer. Internal
/// reads and the later conditional write use no-lock object I/O; the guards
/// remain live until the snapshot is dropped.
pub async fn read_server_config_snapshot<S>(api: Arc<S>) -> Result<ServerConfigSnapshot>
where
S: ObjectIO<
@@ -2295,10 +2071,12 @@ where
{
let config_file = server_config_path();
let local_guard = SERVER_CONFIG_LOCK.clone().write_owned().await;
let transaction_lock = server_config_transaction_lock_path();
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &transaction_lock).await?;
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &config_file).await?;
let guard = lock.get_write_lock(get_lock_acquire_timeout()).await?;
let read_options = ObjectOptions::default();
let read_options = ObjectOptions {
no_lock: true,
..Default::default()
};
match read_config_with_metadata_inner(api, &config_file, &read_options, true).await {
Ok((raw, object_info)) => {
let (config, seed) = decode_persisted_server_config_with_seed(&raw)?;
@@ -2307,7 +2085,6 @@ where
raw: Some(raw),
seed: Some(seed),
etag: object_info.etag,
generation: object_info.data_dir.filter(|generation| !generation.is_nil()),
_local_guard: local_guard,
_guard: guard,
})
@@ -2317,7 +2094,6 @@ where
raw: None,
seed: None,
etag: None,
generation: None,
_local_guard: local_guard,
_guard: guard,
}),
@@ -2332,27 +2108,6 @@ where
/// lock, so a concurrent update or transaction lease loss cannot commit an
/// unfenced overwrite.
pub async fn save_server_config_snapshot<S>(api: Arc<S>, cfg: &Config, snapshot: &ServerConfigSnapshot) -> Result<bool>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
> + NamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
{
save_server_config_snapshot_with_generation(api, cfg, snapshot)
.await
.map(|result| result.persisted())
}
pub async fn save_server_config_snapshot_with_generation<S>(
api: Arc<S>,
cfg: &Config,
snapshot: &ServerConfigSnapshot,
) -> Result<ServerConfigSaveResult>
where
S: ObjectIO<
Error = Error,
@@ -2371,19 +2126,13 @@ where
&& configs_semantically_equal(&snapshot.config, cfg)
{
debug!("server config unchanged and already in standard object shape, skip write");
return Ok(ServerConfigSaveResult {
persisted: false,
generation: snapshot.generation(),
});
return Ok(false);
}
let data = encode_server_config_blob(cfg, snapshot.seed.as_deref())?;
if snapshot.raw.as_deref().is_some_and(|current| current == data.as_slice()) {
debug!("server config bytes unchanged after encode, skip write");
return Ok(ServerConfigSaveResult {
persisted: false,
generation: snapshot.generation(),
});
return Ok(false);
}
let http_preconditions = if snapshot.raw.is_some() {
@@ -2403,22 +2152,19 @@ where
}
};
snapshot.ensure_lock_held()?;
let object_info = save_config_with_opts_and_metadata(
save_config_with_opts(
api,
&config_file,
data,
&ObjectOptions {
max_parity: true,
no_lock: true,
http_preconditions: Some(http_preconditions),
..Default::default()
},
)
.await?;
Ok(ServerConfigSaveResult {
persisted: true,
generation: object_info.data_dir.filter(|generation| !generation.is_nil()),
})
Ok(true)
}
/// Saves the server config while an upper layer holds the namespace write
@@ -2555,9 +2301,8 @@ mod tests {
use super::{
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, config_task_join_error,
configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
lookup_configs, new_and_save_server_config, read_config, read_config_preserve_empty, read_config_with_metadata,
read_config_without_migrate, read_server_config_snapshot, save_server_config, save_server_config_snapshot,
save_server_config_snapshot_with_generation, server_config_transaction_lock_path, storage_class_kvs_mut,
lookup_configs, read_config, read_config_preserve_empty, read_config_with_metadata, read_config_without_migrate,
read_server_config_snapshot, save_server_config, save_server_config_snapshot, server_config_path, storage_class_kvs_mut,
};
use crate::config::{audit, heal, notify, oidc, scanner};
use crate::disk::endpoint::Endpoint;
@@ -2566,9 +2311,7 @@ mod tests {
use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader};
use crate::runtime::sources as runtime_sources;
use crate::set_disk::SetDisks;
use crate::storage_api_contracts::{
admin::StorageAdminApi, namespace::NamespaceLocking as _, object::HTTPPreconditions, range::HTTPRangeSpec,
};
use crate::storage_api_contracts::{admin::StorageAdminApi, namespace::NamespaceLocking as _, range::HTTPRangeSpec};
use http::HeaderMap;
use rustfs_config::audit::{AUDIT_AMQP_SUB_SYS, AUDIT_KAFKA_SUB_SYS, AUDIT_MQTT_SUB_SYS, AUDIT_WEBHOOK_SUB_SYS};
use rustfs_config::notify::{
@@ -3361,85 +3104,6 @@ mod tests {
);
}
#[test]
fn root_heal_null_decodes_as_no_override_and_is_canonicalized_on_save() {
let seed = br#"{
"version":"33",
"storageclass":{"standard":"","rrs":""},
"heal":null,
"future_root":{"mode":"keep"},
"openid":{"default":{
"config_url":"https://issuer.example/.well-known/openid-configuration",
"client_id":"console",
"client_secret":"oidc-secret",
"future_provider_control":"keep"
}},
"notify":{"webhook":{"primary":{
"enable":true,
"endpoint":"https://notify.example/hook",
"auth_token":"notify-secret",
"future_notify_control":"keep"
}}},
"logger":{"webhook":{"primary":{
"enable":true,
"endpoint":"https://audit.example/hook",
"auth_token":"audit-secret",
"future_audit_control":"keep"
}}}
}"#;
let cfg = decode_server_config_blob(seed).expect("root heal null should mean no persisted override");
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
assert!(!is_standard_object_server_config(seed));
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("legacy seed should canonicalize on an authorized save");
let value: Value = serde_json::from_slice(&encoded).expect("canonical config should be valid JSON");
assert!(value.get(HEAL_SUB_SYS).is_none());
assert_eq!(value["future_root"]["mode"].as_str(), Some("keep"));
assert_eq!(value["openid"]["default"]["client_secret"].as_str(), Some("oidc-secret"));
assert_eq!(value["openid"]["default"]["future_provider_control"].as_str(), Some("keep"));
assert_eq!(value["notify"]["webhook"]["primary"]["auth_token"].as_str(), Some("notify-secret"));
assert_eq!(value["notify"]["webhook"]["primary"]["future_notify_control"].as_str(), Some("keep"));
assert_eq!(value["logger"]["webhook"]["primary"]["auth_token"].as_str(), Some("audit-secret"));
assert_eq!(value["logger"]["webhook"]["primary"]["future_audit_control"].as_str(), Some("keep"));
assert!(is_standard_object_server_config(&encoded));
}
#[test]
fn invalid_scalar_and_nested_null_config_shapes_remain_rejected() {
let invalid_sections = [
r#""scanner":null"#,
r#""heal":"""#,
r#""heal":false"#,
r#""heal":0"#,
r#""heal":{"default":null}"#,
r#""heal":{"_":null}"#,
r#""heal":{"bitrot_cycle":null}"#,
r#""heal":[{"key":"bitrot_cycle","value":null}]"#,
];
for section in invalid_sections {
let input = format!(r#"{{"version":"33","storageclass":{{"standard":"","rrs":""}},{section}}}"#);
let err = decode_server_config_blob(input.as_bytes()).expect_err("invalid scalar shape must remain rejected");
assert!(
err.to_string().contains("expected"),
"invalid section {section} returned an unrelated error: {err}"
);
}
}
#[test]
fn valid_heal_object_and_kvs_array_shapes_remain_accepted() {
let empty_object = br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":{}}"#;
let cfg = decode_server_config_blob(empty_object).expect("empty heal object should decode as no override");
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
let kvs_array =
br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":[{"key":"bitrot_cycle","value":"off"}]}"#;
let cfg = decode_server_config_blob(kvs_array).expect("heal KVS array should decode");
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_some());
}
#[test]
fn scanner_update_preserves_unknown_root_and_oidc_provider_fields() {
let seed = br#"{
@@ -3467,171 +3131,6 @@ mod tests {
assert_eq!(value["openid"]["default"]["client_id"].as_str(), Some("console"));
}
#[test]
fn storageclass_reset_removes_stale_inline_block_from_seed() {
let seed = br#"{
"version":"33",
"storageclass":{
"standard":"EC:2",
"rrs":"EC:1",
"optimize":"availability",
"inline_block":"64KiB",
"future_storage_control":"keep"
}
}"#;
let encoded = encode_server_config_blob(&Config::new(), Some(seed)).expect("storageclass reset should encode");
let value: Value = serde_json::from_slice(&encoded).expect("encoded config should be valid json");
let storageclass = value["storageclass"].as_object().expect("storageclass object");
assert!(storageclass.get(crate::config::storageclass::INLINE_BLOCK).is_none());
assert_eq!(storageclass["future_storage_control"].as_str(), Some("keep"));
}
#[test]
fn target_update_preserves_unknown_fields_without_restoring_removed_instances() {
let seed = br#"{
"version":"33",
"storageclass":{"standard":"","rrs":""},
"notify":{"webhook":{
"primary":{
"enable":true,
"endpoint":"https://notify.example/old",
"auth_token":"notify-secret",
"future_control":"keep"
},
"removed":{"enable":true,"endpoint":"https://notify.example/removed"},
"retained":{"enable":true,"endpoint":"https://notify.example/retained","future_control":"keep"},
"enable":{"enable":true,"endpoint":"https://notify.example/named-enable","future_control":"keep"}
}}
}"#;
let mut cfg = decode_server_config_blob(seed).expect("target seed should decode");
let webhook = cfg
.0
.get_mut(NOTIFY_WEBHOOK_SUB_SYS)
.expect("notify webhook subsystem should exist");
webhook
.get_mut("primary")
.expect("primary target should exist")
.insert(rustfs_config::WEBHOOK_ENDPOINT.to_string(), "https://notify.example/new".to_string());
webhook.remove("removed");
webhook.remove("retained");
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("target update should encode");
let value: Value = serde_json::from_slice(&encoded).expect("encoded config should be valid json");
let webhook = value["notify"]["webhook"].as_object().expect("webhook section");
assert_eq!(webhook["primary"]["endpoint"].as_str(), Some("https://notify.example/new"));
assert_eq!(webhook["primary"]["auth_token"].as_str(), Some("notify-secret"));
assert_eq!(webhook["primary"]["future_control"].as_str(), Some("keep"));
assert!(webhook.get("removed").is_none());
assert_eq!(webhook["retained"]["future_control"].as_str(), Some("keep"));
assert!(webhook["retained"].get("enable").is_none());
assert!(webhook["retained"].get("endpoint").is_none());
assert_eq!(webhook["enable"]["endpoint"].as_str(), Some("https://notify.example/named-enable"));
assert_eq!(webhook["enable"]["future_control"].as_str(), Some("keep"));
}
#[test]
fn shorthand_target_update_preserves_shape_and_unknown_nested_fields() {
let seed = br#"{
"version":"33",
"storageclass":{"standard":"","rrs":""},
"notify":{"webhook":{
"enable":true,
"endpoint":"https://notify.example/old",
"future_control":{"endpoint":"leave-untouched","mode":"keep"}
}}
}"#;
let mut cfg = decode_server_config_blob(seed).expect("shorthand target should decode");
cfg.0
.get_mut(NOTIFY_WEBHOOK_SUB_SYS)
.and_then(|targets| targets.get_mut(DEFAULT_DELIMITER))
.expect("default webhook target should exist")
.insert(rustfs_config::WEBHOOK_ENDPOINT.to_string(), "https://notify.example/new".to_string());
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("shorthand target update should encode");
let value: Value = serde_json::from_slice(&encoded).expect("encoded config should be valid json");
let webhook = value["notify"]["webhook"].as_object().expect("webhook shorthand object");
assert_eq!(webhook["endpoint"].as_str(), Some("https://notify.example/new"));
assert!(webhook.get("default").is_none());
assert_eq!(webhook["future_control"]["endpoint"].as_str(), Some("leave-untouched"));
assert_eq!(webhook["future_control"]["mode"].as_str(), Some("keep"));
let decoded = decode_server_config_blob(&encoded).expect("updated shorthand target should remain decodable");
assert_eq!(
decoded
.get_value(NOTIFY_WEBHOOK_SUB_SYS, DEFAULT_DELIMITER)
.expect("updated default webhook target")
.get(rustfs_config::WEBHOOK_ENDPOINT),
"https://notify.example/new"
);
}
#[test]
fn target_kvs_update_preserves_unknown_entries_and_attributes() {
let seed = br#"{
"version":"33",
"storageclass":{"standard":"","rrs":""},
"notify":{"webhook":{"primary":[
{"key":"enable","value":"on","hidden_if_empty":false},
{"key":"endpoint","value":"https://notify.example/old","future_attribute":"keep-endpoint"},
{"key":"future_control","value":"keep","future_attribute":"keep-control"}
]}}
}"#;
let mut cfg = decode_server_config_blob(seed).expect("target KVS seed should decode");
cfg.0
.get_mut(NOTIFY_WEBHOOK_SUB_SYS)
.and_then(|targets| targets.get_mut("primary"))
.expect("primary webhook target should exist")
.insert(rustfs_config::WEBHOOK_ENDPOINT.to_string(), "https://notify.example/new".to_string());
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("target KVS update should encode");
let value: Value = serde_json::from_slice(&encoded).expect("encoded config should be valid json");
let entries = value["notify"]["webhook"]["primary"]
.as_array()
.expect("target KVS shape should be preserved");
let endpoint = entries
.iter()
.find(|entry| entry["key"].as_str() == Some(rustfs_config::WEBHOOK_ENDPOINT))
.expect("endpoint entry should remain");
let future = entries
.iter()
.find(|entry| entry["key"].as_str() == Some("future_control"))
.expect("unknown target entry should remain");
assert_eq!(endpoint["value"].as_str(), Some("https://notify.example/new"));
assert_eq!(endpoint["future_attribute"].as_str(), Some("keep-endpoint"));
assert_eq!(future["value"].as_str(), Some("keep"));
assert_eq!(future["future_attribute"].as_str(), Some("keep-control"));
}
#[test]
fn target_default_alias_is_canonicalized_without_losing_unknown_fields() {
let seed = br#"{
"version":"33",
"storageclass":{"standard":"","rrs":""},
"notify":{"webhook":{
"_":{"enable":false,"endpoint":"https://notify.example/alias","future_alias":"keep"},
"default":{"enable":true,"endpoint":"https://notify.example/default","future_default":"keep"}
}}
}"#;
let cfg = decode_server_config_blob(seed).expect("dual default aliases should decode");
let expected_endpoint = cfg
.get_value(NOTIFY_WEBHOOK_SUB_SYS, DEFAULT_DELIMITER)
.expect("default webhook target should exist")
.get(rustfs_config::WEBHOOK_ENDPOINT);
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("default alias should canonicalize");
let value: Value = serde_json::from_slice(&encoded).expect("encoded config should be valid json");
let webhook = value["notify"]["webhook"].as_object().expect("webhook section");
assert!(webhook.get(DEFAULT_DELIMITER).is_none());
assert_eq!(webhook["default"]["endpoint"].as_str(), Some(expected_endpoint.as_str()));
assert_eq!(webhook["default"]["future_alias"].as_str(), Some("keep"));
assert_eq!(webhook["default"]["future_default"].as_str(), Some("keep"));
}
#[test]
fn test_scanner_config_changes_are_semantically_significant() {
let baseline = Config::new();
@@ -4625,7 +4124,6 @@ mod tests {
/// What reads of the config object currently return.
enum RecoveryReadState {
Missing,
Blob(Vec<u8>),
QuorumError,
}
@@ -4638,7 +4136,6 @@ mod tests {
heal_calls: AtomicUsize,
write_calls: AtomicUsize,
last_put_no_lock: AtomicBool,
last_put_preconditions: Mutex<Option<HTTPPreconditions>>,
revision: AtomicUsize,
drive_counts: Vec<usize>,
lock_manager: Arc<rustfs_lock::GlobalLockManager>,
@@ -4653,7 +4150,6 @@ mod tests {
heal_calls: AtomicUsize::new(0),
write_calls: AtomicUsize::new(0),
last_put_no_lock: AtomicBool::new(false),
last_put_preconditions: Mutex::new(None),
revision: AtomicUsize::new(1),
drive_counts: vec![2],
lock_manager: Arc::new(rustfs_lock::GlobalLockManager::new()),
@@ -4710,7 +4206,6 @@ mod tests {
_opts: &ObjectOptions,
) -> Result<GetObjectReader> {
let data = match &*self.state.lock().expect("state lock poisoned") {
RecoveryReadState::Missing => return Err(Error::ConfigNotFound),
RecoveryReadState::Blob(data) => data.clone(),
RecoveryReadState::QuorumError => return Err(Error::ErasureReadQuorum),
};
@@ -4718,9 +4213,6 @@ mod tests {
size: data.len() as i64,
actual_size: data.len() as i64,
etag: Some(format!("config-{}", self.revision.load(Ordering::SeqCst))),
data_dir: Some(uuid::Uuid::from_u128(
u128::try_from(self.revision.load(Ordering::SeqCst)).expect("test revision should fit in u128"),
)),
..Default::default()
};
Ok(GetObjectReader {
@@ -4739,19 +4231,15 @@ mod tests {
opts: &ObjectOptions,
) -> Result<ObjectInfo> {
let current_etag = format!("config-{}", self.revision.load(Ordering::SeqCst));
let object_exists = matches!(&*self.state.lock().expect("state lock poisoned"), RecoveryReadState::Blob(_));
if let Some(preconditions) = &opts.http_preconditions
&& (preconditions
.if_match_value()
.is_some_and(|etag| !object_exists || etag != current_etag)
|| (object_exists && preconditions.if_none_match_value() == Some("*")))
&& (preconditions.if_match_value().is_some_and(|etag| etag != current_etag)
|| preconditions.if_none_match_value() == Some("*"))
{
return Err(Error::PreconditionFailed);
}
let mut body = Vec::new();
data.stream.read_to_end(&mut body).await?;
self.last_put_no_lock.store(opts.no_lock, Ordering::SeqCst);
*self.last_put_preconditions.lock().expect("preconditions lock poisoned") = opts.http_preconditions.clone();
self.write_calls.fetch_add(1, Ordering::SeqCst);
*self.state.lock().expect("state lock poisoned") = RecoveryReadState::Blob(body.clone());
let revision = self.revision.fetch_add(1, Ordering::SeqCst) + 1;
@@ -4759,7 +4247,6 @@ mod tests {
size: i64::try_from(body.len()).expect("test config should fit in i64"),
actual_size: i64::try_from(body.len()).expect("test config should fit in i64"),
etag: Some(format!("config-{revision}")),
data_dir: Some(uuid::Uuid::from_u128(u128::try_from(revision).expect("test revision should fit in u128"))),
..Default::default()
})
}
@@ -4797,18 +4284,10 @@ mod tests {
.expect("scanner-only config change should be persisted");
assert_eq!(store.write_calls.load(Ordering::SeqCst), 1);
assert!(!store.last_put_no_lock.load(Ordering::SeqCst));
let preconditions = store
.last_put_preconditions
.lock()
.expect("preconditions lock poisoned")
.clone()
.expect("existing config update must be conditional");
assert_eq!(preconditions.if_match_value(), Some("config-1"));
assert_eq!(preconditions.if_none_match_value(), None);
assert!(store.last_put_no_lock.load(Ordering::SeqCst));
assert_eq!(
store.lock_resources.lock().expect("lock resources mutex poisoned").as_slice(),
&[server_config_transaction_lock_path()]
&[server_config_path()]
);
let decoded = read_config_without_migrate(store)
.await
@@ -4822,73 +4301,6 @@ mod tests {
);
}
#[tokio::test]
async fn server_config_snapshot_save_returns_committed_generation() {
let baseline = encode_server_config_blob(&Config::new(), None).expect("baseline config should encode");
let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(baseline), None));
let snapshot = read_server_config_snapshot(store.clone())
.await
.expect("server config snapshot");
let result = save_server_config_snapshot_with_generation(store, &config_with_scanner_cycle("61"), &snapshot)
.await
.expect("conditional config save");
assert!(result.persisted());
assert_eq!(result.generation(), Some(uuid::Uuid::from_u128(2)));
}
#[tokio::test]
async fn missing_server_config_is_created_with_if_none_match() {
let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Missing, None));
let cfg = config_with_scanner_cycle("61");
save_server_config(store.clone(), &cfg)
.await
.expect("missing config should be created conditionally");
assert_eq!(store.write_calls.load(Ordering::SeqCst), 1);
assert!(!store.last_put_no_lock.load(Ordering::SeqCst));
let preconditions = store
.last_put_preconditions
.lock()
.expect("preconditions lock poisoned")
.clone()
.expect("missing config create must be conditional");
assert_eq!(preconditions.if_match_value(), None);
assert_eq!(preconditions.if_none_match_value(), Some("*"));
let persisted = read_config_without_migrate(store)
.await
.expect("created config should reload");
assert_eq!(
persisted
.get_value(SCANNER_SUB_SYS, DEFAULT_DELIMITER)
.expect("persisted scanner config")
.get(SCANNER_CYCLE),
"61"
);
}
#[tokio::test]
async fn missing_config_initialization_recheck_preserves_concurrent_config() {
let existing = config_with_scanner_cycle("73");
let baseline = encode_server_config_blob(&existing, None).expect("existing config should encode");
let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(baseline), None));
let observed = new_and_save_server_config(store.clone())
.await
.expect("initialization recheck should return the config created by another writer");
assert_eq!(store.write_calls.load(Ordering::SeqCst), 0);
assert_eq!(
observed
.get_value(SCANNER_SUB_SYS, DEFAULT_DELIMITER)
.expect("concurrent scanner config")
.get(SCANNER_CYCLE),
"73"
);
}
#[tokio::test]
async fn stale_server_config_snapshot_cannot_overwrite_newer_update() {
let baseline = encode_server_config_blob(&Config::new(), None).expect("baseline config should encode");
@@ -4945,7 +4357,7 @@ mod tests {
let lock = rustfs_lock::NamespaceLock::new("server-config-lease-loss".to_string(), client.clone());
let guard = lock
.lock_guard(
rustfs_lock::ObjectKey::new(crate::disk::RUSTFS_META_BUCKET, server_config_transaction_lock_path()),
rustfs_lock::ObjectKey::new(crate::disk::RUSTFS_META_BUCKET, server_config_path()),
"server-config-lease-loss",
std::time::Duration::from_secs(1),
std::time::Duration::from_millis(120),
@@ -4960,7 +4372,6 @@ mod tests {
raw: Some(baseline.clone()),
seed: None,
etag: Some("config-0".to_string()),
generation: Some(uuid::Uuid::from_u128(1)),
_local_guard: local_guard,
_guard: guard,
};
-4
View File
@@ -1049,10 +1049,6 @@ impl LocalDiskWrapper {
Ok(())
}
pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) {
*self.disk_id.write().await = id;
}
/// Get the current disk ID
pub async fn get_current_disk_id(&self) -> Option<Uuid> {
*self.disk_id.read().await
+2 -121
View File
@@ -5078,7 +5078,7 @@ impl LocalDisk {
Ok((buf, mtime))
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
async fn read_metadata_with_dmtime(&self, file_path: impl AsRef<Path>) -> Result<(Vec<u8>, Option<OffsetDateTime>)> {
check_path_length(file_path.as_ref().to_string_lossy().as_ref())?;
@@ -5121,7 +5121,7 @@ impl LocalDisk {
Ok((data, modtime))
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
async fn read_all_data(&self, volume: &str, volume_dir: impl AsRef<Path>, file_path: impl AsRef<Path>) -> Result<Vec<u8>> {
// TODO: timeout support
let (data, _) = self.read_all_data_with_dmtime(volume, volume_dir, file_path).await?;
@@ -6641,49 +6641,6 @@ impl LocalDisk {
let xl_path = object_dir.join(STORAGE_FORMAT_FILE);
restore_delete_rollback_after_error(object_dir, &xl_path, Some(rollback_dir), volume, object, stage, err).await
}
/// Execute every deferred data-dir deletion pending on `volume` right now,
/// even while snapshot leases are still held. Bucket deletion requires it:
/// a streaming reader defers the physical cleanup of an already-deleted
/// version, and a non-force `delete_volume` would otherwise fail closed
/// with `VolumeNotEmpty` on those remnants even though the bucket is
/// logically empty. The still-active readers keep their open descriptors;
/// only path-based reopens observe the removal.
async fn settle_pending_snapshot_deletes(&self, volume: &str) {
let pending: Vec<(SnapshotLeaseKey, DeleteOptions)> = {
let mut registry = self.snapshot_leases.lock().await;
registry
.entries
.iter_mut()
.filter(|(key, entry)| key.volume == volume && !entry.deleting && entry.pending_delete.is_some())
.map(|(key, entry)| {
entry.deleting = true;
(key.clone(), entry.pending_delete.clone().expect("filtered on Some"))
})
.collect()
};
for (key, opts) in pending {
let result = self.delete_unleased(&key.volume, &key.path, &opts).await;
let mut registry = self.snapshot_leases.lock().await;
match result {
Ok(()) => {
registry.entries.remove(&key);
}
Err(err) => {
if let Some(entry) = registry.entries.get_mut(&key) {
entry.deleting = false;
}
warn!(
volume = %key.volume,
path = %key.path,
error = %err,
"failed to settle deferred data-dir deletion before volume removal"
);
}
}
}
}
}
#[async_trait::async_trait]
@@ -9165,14 +9122,6 @@ impl DiskAPI for LocalDisk {
let p = self.get_bucket_path(volume)?;
let _volume_mutation_guard = os::disk_volume_mutation_lock(&self.root, volume).write_owned().await;
// A streaming reader's snapshot lease defers the physical cleanup of
// data dirs whose version delete already committed. Those remnants are
// logically deleted, so run the parked cleanups now instead of letting
// the non-force removal below fail closed on them (the s3-tests SSE-C
// teardown races exactly this way: DeleteObjects, then DeleteBucket
// while an abandoned GET body still pins the lease).
self.settle_pending_snapshot_deletes(volume).await;
// Non-force removes empty directory remnants children-first with
// non-recursive rmdir calls. A file that exists during the scan, or
// appears before its parent is removed, fails closed with
@@ -16032,74 +15981,6 @@ mod test {
assert!(matches!(disk.read_all(volume, &first_part).await, Err(DiskError::FileNotFound)));
}
#[tokio::test]
async fn delete_volume_settles_lease_deferred_cleanup() {
use tempfile::tempdir;
let root_dir = tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let volume = "snapshot-lease-bucket-delete";
let object = "multipart_enc";
let version_id = Uuid::new_v4();
let data_dir = Uuid::new_v4();
let rollback_dir = Uuid::new_v4();
let data_path = path_join_buf(&[object, &data_dir.to_string()]);
let part = path_join_buf(&[&data_path, "part.1"]);
ensure_test_volume(&disk, volume).await;
disk.write_all(volume, &part, Bytes::from_static(b"payload"))
.await
.expect("shard should be written");
let fi = test_file_info(object, version_id, Some(data_dir), None);
disk.write_all(volume, &path_join_buf(&[object, STORAGE_FORMAT_FILE]), test_meta(fi.clone()).into())
.await
.expect("metadata should be written");
// An abandoned streaming GET pins the data dir with a snapshot lease.
let snapshot = disk
.acquire_snapshot_lease(volume, &data_path)
.await
.expect("snapshot lease should be acquired");
disk.delete_version(
volume,
object,
fi.clone(),
false,
DeleteOptions {
old_data_dir: Some(rollback_dir),
..Default::default()
},
)
.await
.expect("version delete should commit metadata");
disk.delete(
volume,
&format!("{object}/{rollback_dir}"),
DeleteOptions {
recursive: true,
immediate: true,
..Default::default()
},
)
.await
.expect("version delete should schedule physical cleanup");
// The bucket is logically empty; a non-force volume delete must settle
// the deferred data-dir cleanup instead of failing with VolumeNotEmpty.
disk.delete_volume(volume, false)
.await
.expect("bucket delete must not observe lease-deferred remnants");
assert!(matches!(
disk.read_all(volume, &part).await,
Err(DiskError::FileNotFound | DiskError::VolumeNotFound)
));
// The late lease release finds nothing pending and stays idempotent.
disk.release_snapshot_lease(volume, &data_path, snapshot)
.await
.expect("releasing the lease after bucket deletion should be a no-op");
}
#[tokio::test]
async fn version_delete_cleanup_intent_survives_local_disk_restart() {
use tempfile::tempdir;
-78
View File
@@ -132,18 +132,6 @@ pub enum Disk {
Remote(Box<RemoteDisk>),
}
impl Disk {
pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) -> Result<()> {
match self {
Disk::Local(local_disk) => {
local_disk.set_disk_id_state(id).await;
Ok(())
}
Disk::Remote(remote_disk) => remote_disk.set_disk_id(id).await,
}
}
}
#[async_trait::async_trait]
impl DiskAPI for Disk {
fn to_string(&self) -> String {
@@ -1564,72 +1552,6 @@ mod tests {
let _ = fs::remove_dir_all(&test_dir).await;
}
#[tokio::test]
#[serial_test::serial]
async fn local_disk_id_state_does_not_publish_to_the_process_registry() {
let local_dir = tempfile::tempdir().expect("local disk tempdir should be created");
let mut endpoint =
Endpoint::try_from(local_dir.path().to_str().expect("tempdir path should be utf8")).expect("endpoint should parse");
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(0);
let local_disk = LocalDisk::new(&endpoint, false).await.expect("local disk should initialize");
let disk = Disk::Local(Box::new(LocalDiskWrapper::new(Arc::new(local_disk), false)));
let disk_id = Uuid::new_v4();
disk.set_disk_id_state(Some(disk_id))
.await
.expect("local wrapper state should accept a disk ID");
let Disk::Local(local_disk) = &disk else {
panic!("test disk should remain local");
};
assert_eq!(local_disk.get_current_disk_id().await, Some(disk_id));
assert!(
!crate::runtime::global::current_ctx()
.local_disk_id_map()
.read()
.await
.contains_key(&disk_id),
"state-only startup publication must not update the process disk-ID registry"
);
disk.set_disk_id_state(None)
.await
.expect("local wrapper state should clear a disk ID");
assert_eq!(local_disk.get_current_disk_id().await, None);
}
#[tokio::test]
async fn remote_disk_id_state_delegates_some_and_none() {
let mut endpoint = Endpoint::try_from("http://remote-server:9000/data").expect("remote endpoint should parse");
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(0);
let remote_disk = RemoteDisk::new(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
Arc::new(crate::cluster::rpc::TcpHttpInternodeDataTransport),
)
.await
.expect("remote disk should initialize");
let disk = Disk::Remote(Box::new(remote_disk));
let disk_id = Uuid::new_v4();
disk.set_disk_id_state(Some(disk_id))
.await
.expect("remote state should accept a disk ID");
assert_eq!(disk.get_disk_id().await.expect("remote disk ID should be readable"), Some(disk_id));
disk.set_disk_id_state(None)
.await
.expect("remote state should clear a disk ID");
assert_eq!(disk.get_disk_id().await.expect("remote disk ID should be readable"), None);
}
#[tokio::test]
async fn reset_health_for_store_init_retry_delegates_to_disk_variants() {
let local_dir = tempfile::tempdir().unwrap();
+3 -3
View File
@@ -103,7 +103,7 @@ where
/// or `out` is larger than one shard. On error `out`'s contents are
/// unspecified but never contain bytes that failed the hash check — the copy
/// happens only after verification.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn read(&mut self, out: &mut [u8]) -> std::io::Result<usize> {
let want = out.len();
self.begin_read(want)?;
@@ -303,7 +303,7 @@ where
/// Write a (hash+data) block. Returns the number of data bytes written.
/// Returns an error if called after a short write or if data exceeds shard_size.
#[hotpath::measure(label = "BitrotWriter::write")]
#[cfg_attr(feature = "hotpath", hotpath::measure(label = "BitrotWriter::write"))]
pub async fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
if buf.is_empty() {
return Ok(0);
@@ -455,7 +455,7 @@ pub fn bitrot_shard_file_size(size: usize, shard_size: usize, algo: HashAlgorith
/// stores those as whole-file bitrot with no interleaved hash, so the size guard
/// on the next line would reject a genuinely healthy part. Reading legacy V1
/// whole-file-bitrot objects would need a separate verification path.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn bitrot_verify<R: AsyncRead + Unpin + Send>(
mut r: R,
want_size: usize,
+12 -119
View File
@@ -691,7 +691,7 @@ impl<R> ParallelReader<R>
where
R: crate::erasure::coding::ShardSource,
{
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn read(&mut self) -> (Vec<Option<Vec<u8>>>, Vec<Option<Error>>) {
// On the reconstruction-verifying GET path, read every live shard reader
// in lockstep so all readers advance one block per stripe and stay
@@ -1505,7 +1505,7 @@ where
}
impl Erasure {
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn decode<W, R>(
&self,
writer: &mut W,
@@ -1645,28 +1645,15 @@ impl Erasure {
}
Err(e) => {
record_get_stage_duration_if_enabled(GET_OBJECT_PATH_LEGACY_DUPLEX, GET_STAGE_EMIT, emit_stage_start);
let reason = classify_io_error(&e);
if reason == GetObjectFailureReason::DownstreamClosed {
debug!(
block_offset,
block_length,
bytes_written = *written,
stage = GET_STAGE_EMIT,
reason = reason.as_str(),
error = ?e,
"Erasure decode stopped after downstream closed"
);
} else {
error!(
block_offset,
block_length,
bytes_written = *written,
stage = GET_STAGE_EMIT,
reason = reason.as_str(),
error = ?e,
"Erasure decode failed to emit reconstructed data"
);
}
error!(
block_offset,
block_length,
bytes_written = *written,
stage = GET_STAGE_EMIT,
reason = classify_io_error(&e).as_str(),
error = ?e,
"Erasure decode failed to emit reconstructed data"
);
*ret_err = Some(e);
return StripeFlow::Stop;
}
@@ -1958,7 +1945,7 @@ mod tests {
use std::io::Cursor;
use std::pin::Pin;
use std::sync::{
Arc, Mutex,
Arc,
atomic::{AtomicUsize, Ordering},
};
use std::task::{Context, Poll};
@@ -2133,59 +2120,6 @@ mod tests {
}
}
struct DownstreamClosedWriter;
impl AsyncWrite for DownstreamClosedWriter {
fn poll_write(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &[u8]) -> Poll<io::Result<usize>> {
Poll::Ready(Err(crate::diagnostics::get::mark_get_object_downstream_closed(io::Error::new(
ErrorKind::BrokenPipe,
"injected downstream close",
))))
}
fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
Poll::Ready(Ok(()))
}
fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
Poll::Ready(Ok(()))
}
}
#[derive(Clone, Default)]
struct CapturedLogs(Arc<Mutex<Vec<u8>>>);
struct CapturedLogWriter(Arc<Mutex<Vec<u8>>>);
impl CapturedLogs {
fn contents(&self) -> String {
String::from_utf8(self.0.lock().expect("captured logs mutex should not be poisoned").clone())
.expect("captured logs should be valid UTF-8")
}
}
impl std::io::Write for CapturedLogWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.0
.lock()
.expect("captured logs mutex should not be poisoned")
.extend_from_slice(buf);
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for CapturedLogs {
type Writer = CapturedLogWriter;
fn make_writer(&'a self) -> Self::Writer {
CapturedLogWriter(Arc::clone(&self.0))
}
}
#[test]
fn parallel_reader_constructor_variants_preserve_read_cost_and_verification_flags() {
let erasure = Erasure::new(2, 1, 64);
@@ -2281,47 +2215,6 @@ mod tests {
assert_eq!(err.to_string(), "injected emit failure");
}
#[tokio::test(flavor = "current_thread")]
async fn erasure_decode_logs_reconstructed_downstream_close_at_debug() {
let logs = CapturedLogs::default();
let subscriber = tracing_subscriber::fmt()
.with_max_level(tracing::Level::DEBUG)
.with_writer(logs.clone())
.with_ansi(false)
.without_time()
.finish();
let _guard = tracing::subscriber::set_default(subscriber);
let erasure = Erasure::new(2, 1, 64);
let data: Vec<u8> = (0..64).collect();
let shard_size = erasure.shard_size();
let encoded = erasure.encode_data(&data).expect("test data should encode");
let readers = vec![
None,
Some(BitrotReader::new(
Cursor::new(encoded[1].to_vec()),
shard_size,
HashAlgorithm::None,
false,
)),
Some(BitrotReader::new(
Cursor::new(encoded[2].to_vec()),
shard_size,
HashAlgorithm::None,
false,
)),
];
let mut writer = DownstreamClosedWriter;
let (written, err) = erasure.decode(&mut writer, readers, 0, data.len(), data.len()).await;
assert_eq!(written, 0);
assert_eq!(err.expect("downstream close must still terminate the GET").kind(), ErrorKind::BrokenPipe);
let captured = logs.contents();
assert!(captured.contains("Erasure decode stopped after downstream closed"));
assert!(!captured.contains("Erasure decode failed to emit reconstructed data"));
}
#[tokio::test]
async fn test_erasure_decode_rejects_reader_count_and_range_overflow() {
let erasure = Erasure::new(2, 1, 64);
+4 -4
View File
@@ -504,7 +504,7 @@ impl Erasure {
Ok((reader, total))
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn encode<R>(
self: Arc<Self>,
reader: R,
@@ -670,7 +670,7 @@ impl Erasure {
Ok((reader, total))
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn encode_batched<R>(
self: Arc<Self>,
mut reader: R,
@@ -798,7 +798,7 @@ impl Erasure {
/// Fast path for small inline objects: skip tokio::spawn + mpsc channel.
/// Reads all data, encodes directly, writes shards sequentially.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn encode_inline_small<R>(
self: Arc<Self>,
reader: R,
@@ -813,7 +813,7 @@ impl Erasure {
/// Fast path for single-block non-inline objects: avoids the producer/consumer
/// pipeline in `encode()` while keeping the same writer/quorum/shutdown semantics.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn encode_single_block_non_inline<R>(
self: Arc<Self>,
reader: R,
+5 -5
View File
@@ -640,7 +640,7 @@ impl Erasure {
/// # Returns
/// A vector of encoded shards as `Bytes`.
#[tracing::instrument(level = "debug", skip_all, fields(data_len=data.len()))]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub fn encode_data(&self, data: &[u8]) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
@@ -688,7 +688,7 @@ impl Erasure {
/// Encode owned data, avoiding a copy when the caller already has a heap buffer.
/// Falls back to copying into a new buffer if zero-copy conversion fails.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub fn encode_data_owned(&self, data: Vec<u8>) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
@@ -752,7 +752,7 @@ impl Erasure {
/// block), the `resize(need_total_size)` below stays within capacity for every
/// `data_len <= block_size` — both shard-size formulas are monotone in
/// `data_len` — so this function never reallocates the buffer.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub fn encode_data_bytes_mut(&self, mut data_buffer: BytesMut, data_len: usize) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
@@ -805,7 +805,7 @@ impl Erasure {
///
/// # Returns
/// Ok if reconstruction succeeds, error otherwise.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub fn decode_data(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
if self.parity_shards > 0 {
if self.uses_legacy {
@@ -825,7 +825,7 @@ impl Erasure {
}
/// Decode and reconstruct missing data shards, then regenerate parity shards.
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub fn decode_data_and_parity(&self, shards: &mut [Option<Vec<u8>>]) -> io::Result<()> {
if self.parity_shards > 0 {
if self.uses_legacy {
+1 -58
View File
@@ -22,8 +22,6 @@ use rustfs_config::{
ENV_STARTUP_TOPOLOGY_WAIT_MODE, ENV_STARTUP_TOPOLOGY_WAIT_TIMEOUT, ENV_UNSAFE_BYPASS_DISK_CHECK,
};
use rustfs_utils::{XHost, check_local_server_addr, get_env_opt_str, get_host_ip, is_local_host};
#[cfg(test)]
use std::sync::{LazyLock, Mutex};
use std::{
collections::{BTreeMap, BTreeSet, HashMap, HashSet, hash_map::Entry},
future::Future,
@@ -600,58 +598,6 @@ const DNS_RETRY_JITTER_PERCENT: u64 = 20;
/// wait does not flood the log with one line per backoff tick.
const TOPOLOGY_WARN_THROTTLE: Duration = Duration::from_secs(30);
#[cfg(test)]
static FORCED_LOCAL_HOST_RESOLUTION_TIMEOUTS: LazyLock<Mutex<HashSet<String>>> = LazyLock::new(|| Mutex::new(HashSet::new()));
#[cfg(test)]
struct LocalHostResolutionTimeoutGuard {
hosts: Vec<String>,
}
#[cfg(test)]
impl Drop for LocalHostResolutionTimeoutGuard {
fn drop(&mut self) {
let mut forced_hosts = FORCED_LOCAL_HOST_RESOLUTION_TIMEOUTS
.lock()
.expect("local-host test resolver mutex poisoned");
for host in &self.hosts {
forced_hosts.remove(host);
}
}
}
#[cfg(test)]
fn force_local_host_resolution_timeout_for_test(hosts: &[&str]) -> LocalHostResolutionTimeoutGuard {
let hosts = hosts.iter().map(|host| (*host).to_string()).collect::<Vec<_>>();
let mut forced_hosts = FORCED_LOCAL_HOST_RESOLUTION_TIMEOUTS
.lock()
.expect("local-host test resolver mutex poisoned");
forced_hosts.extend(hosts.iter().cloned());
LocalHostResolutionTimeoutGuard { hosts }
}
#[cfg(test)]
fn local_host_resolution_timeout_forced(host: &Host<&str>) -> bool {
let host = match host {
Host::Domain(domain) => (*domain).to_string(),
Host::Ipv4(ip) => ip.to_string(),
Host::Ipv6(ip) => ip.to_string(),
};
FORCED_LOCAL_HOST_RESOLUTION_TIMEOUTS
.lock()
.expect("local-host test resolver mutex poisoned")
.contains(&host)
}
fn endpoint_is_local_host(host: Host<&str>, port: u16, local_port: u16) -> Result<bool> {
#[cfg(test)]
if local_host_resolution_timeout_forced(&host) {
return Err(Error::new(ErrorKind::TimedOut, "resolver timeout"));
}
is_local_host(host, port, local_port)
}
struct DnsRetryDeadline {
started: Instant,
timeout: Duration,
@@ -748,7 +694,7 @@ async fn resolve_local_host_with_retry(
retry_dns_operation(
|| {
let host = host.clone();
async move { endpoint_is_local_host(host, port, local_port) }
async move { is_local_host(host, port, local_port) }
},
async_sleep,
dns_retry_deadline,
@@ -2285,9 +2231,6 @@ mod test {
#[serial]
#[tokio::test]
async fn create_server_endpoints_bounds_kubernetes_alias_dns_fallback() {
let _resolution_timeout =
force_local_host_resolution_timeout_for_test(&["unrelated-0.example.invalid", "unrelated-1.example.invalid"]);
async_with_vars(
[
(ENV_LOCAL_ENDPOINT_HOST, None),
+1 -3
View File
@@ -12,10 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#![recursion_limit = "256"]
/// Scope-based hotpath measurement for `#[async_trait]` methods, where
/// `#[hotpath::measure]` would only time the boxed-future construction.
/// `#[cfg_attr(feature = "hotpath", hotpath::measure)]` would only time the boxed-future construction.
/// The guard records wall time from this statement until the enclosing
/// (desugared) async block completes, including early returns via `?`.
#[cfg(feature = "hotpath")]
@@ -1,80 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use async_trait::async_trait;
use http::{HeaderMap, HeaderValue};
use std::collections::HashMap;
use std::error::Error;
use std::fmt::{Display, Formatter};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ReadEncryptionMode {
Direct { base_nonce: [u8; 12] },
Object,
}
pub struct ReadEncryptionMaterial {
pub key_bytes: [u8; 32],
pub mode: ReadEncryptionMode,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum EncryptionResolutionErrorKind {
InvalidRequest,
InvalidMetadata,
ServiceUnavailable,
DecryptionFailed,
}
#[derive(Debug)]
pub struct EncryptionResolutionError {
kind: EncryptionResolutionErrorKind,
message: String,
}
impl EncryptionResolutionError {
pub fn new(kind: EncryptionResolutionErrorKind, message: impl Into<String>) -> Self {
Self {
kind,
message: message.into(),
}
}
pub fn kind(&self) -> EncryptionResolutionErrorKind {
self.kind
}
}
impl Display for EncryptionResolutionError {
fn fmt(&self, formatter: &mut Formatter<'_>) -> std::fmt::Result {
formatter.write_str(&self.message)
}
}
impl Error for EncryptionResolutionError {}
pub struct ReadEncryptionRequest<'a> {
pub bucket: &'a str,
pub object: &'a str,
pub metadata: &'a HashMap<String, String>,
pub headers: &'a HeaderMap<HeaderValue>,
}
#[async_trait]
pub trait ObjectEncryptionResolver: Send + Sync {
async fn resolve_read_material(
&self,
request: ReadEncryptionRequest<'_>,
) -> Result<Option<ReadEncryptionMaterial>, EncryptionResolutionError>;
}
-5
View File
@@ -84,7 +84,6 @@ pub(crate) fn legacy_encrypted_range_seek_enabled() -> bool {
}
mod body_cache_hook;
mod encryption;
mod hook_slot;
mod object_mutation_hook;
mod readers;
@@ -99,10 +98,6 @@ pub use body_cache_hook::{
pub(crate) use body_cache_hook::{
get_object_body_cache_hook, get_object_body_cache_hook_suppressed, without_get_object_body_cache_hook,
};
pub use encryption::{
EncryptionResolutionError, EncryptionResolutionErrorKind, ObjectEncryptionResolver, ReadEncryptionMaterial,
ReadEncryptionMode, ReadEncryptionRequest,
};
pub(crate) use object_mutation_hook::notify_object_mutation;
pub use object_mutation_hook::{ObjectMutationHook, register_object_mutation_hook, unregister_object_mutation_hook};
pub use readers::*;
File diff suppressed because it is too large Load Diff
+46 -17
View File
@@ -273,9 +273,29 @@ impl ObjectInfo {
}
pub fn is_encrypted(&self) -> bool {
self.user_defined
.keys()
.any(|key| rustfs_utils::http::is_object_encryption_marker(key))
// Corresponding to the logic in rustfs/src/sse.rs/encryption_material_to_metadata function
use rustfs_utils::http::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
self.user_defined.keys().any(|key| {
let lower = key.to_ascii_lowercase();
lower.starts_with("x-minio-encryption-")
|| lower.starts_with("x-minio-internal-server-side-encryption-")
|| matches!(
lower.as_str(),
"x-minio-internal-encrypted-multipart"
| "x-rustfs-encryption-key"
| "x-rustfs-encryption-algorithm"
| "x-rustfs-encryption-iv"
| "x-rustfs-encryption-key-id"
| "x-rustfs-encryption-context"
| "x-rustfs-encryption-tag"
| "x-amz-server-side-encryption-aws-kms-key-id"
| SSEC_ALGORITHM_HEADER
| SSEC_KEY_HEADER
| SSEC_KEY_MD5_HEADER
| "x-amz-server-side-encryption"
)
})
}
/// Maximum inline size for non-versioned objects (128 KiB).
@@ -319,7 +339,26 @@ impl ObjectInfo {
}
pub fn encryption_original_size(&self) -> std::io::Result<Option<i64>> {
rustfs_utils::http::get_object_encryption_original_size(&self.user_defined)
let actual_size = rustfs_utils::http::get_str(&self.user_defined, rustfs_utils::http::SUFFIX_ACTUAL_SIZE);
if let Some(size_str) = self
.user_defined
.get("x-rustfs-encryption-original-size")
.map(String::as_str)
.or_else(|| {
self.user_defined
.get("x-amz-server-side-encryption-customer-original-size")
.map(String::as_str)
})
.or(actual_size.as_deref())
&& !size_str.is_empty()
{
let size = size_str
.parse::<i64>()
.map_err(|e| std::io::Error::other(format!("Failed to parse encryption original size: {e}")))?;
return Ok(Some(size));
}
Ok(None)
}
pub fn decrypted_size(&self) -> std::io::Result<i64> {
@@ -349,6 +388,9 @@ impl ObjectInfo {
return Ok(actual_size);
}
// Check if object is encrypted
// Managed SSE stores original size in x-rustfs-encryption-original-size metadata
// SSE-C stores original size in x-amz-server-side-encryption-customer-original-size
if let Some(size) = self.encryption_original_size()? {
return Ok(size);
}
@@ -839,19 +881,6 @@ mod tests {
assert!(!object.is_inline_fast_path_eligible(), "transitioned objects must fall back");
}
#[test]
fn minio_internal_encryption_metadata_is_not_treated_as_plaintext() {
let object = ObjectInfo {
user_defined: Arc::new(HashMap::from([(
"X-Minio-Internal-Server-Side-Encryption-Sealed-Key".to_string(),
"sealed".to_string(),
)])),
..Default::default()
};
assert!(object.is_encrypted());
}
#[test]
fn versions_after_marker_handles_null_version_marker() {
let first_version = Uuid::parse_str("11111111-2222-3333-4444-555555555555").unwrap();
-17
View File
@@ -46,7 +46,6 @@ use crate::bucket::metadata_sys::BucketMetadataSys;
use crate::bucket::replication::{DynReplicationPool, ReplicationStats};
use crate::disk::DiskStore;
use crate::layout::endpoints::{EndpointServerPools, SetupType};
use crate::object_api::ObjectEncryptionResolver;
use crate::services::event_notification::EventNotifier;
use crate::services::tier::tier::TierConfigMgr;
use rustfs_lock::{GlobalLockManager, get_global_lock_manager};
@@ -160,8 +159,6 @@ pub struct InstanceContext {
/// workers (scanner/heal/tier/lifecycle) without touching another instance.
/// Replaces the process-global cancel-token static.
background_cancel_token: OnceLock<CancellationToken>,
/// Resolves object-encryption material at the application boundary.
object_encryption_resolver: OnceLock<Arc<dyn ObjectEncryptionResolver>>,
tier_delete_journal_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
transition_transaction_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
#[cfg(test)]
@@ -200,7 +197,6 @@ impl InstanceContext {
local_disk_set_drives: Arc::new(RwLock::new(Vec::new())),
bucket_metadata_sys: std::sync::Mutex::new(None),
background_cancel_token: OnceLock::new(),
object_encryption_resolver: OnceLock::new(),
tier_delete_journal_recovery_stores: std::sync::Mutex::new(HashSet::new()),
transition_transaction_recovery_stores: std::sync::Mutex::new(HashSet::new()),
#[cfg(test)]
@@ -213,19 +209,6 @@ impl InstanceContext {
self.lock_manager.clone()
}
/// Install the application-owned object-encryption resolver once.
pub fn set_object_encryption_resolver(
&self,
resolver: Arc<dyn ObjectEncryptionResolver>,
) -> Result<(), Arc<dyn ObjectEncryptionResolver>> {
self.object_encryption_resolver.set(resolver)
}
/// Return the configured object-encryption resolver, if startup installed one.
pub fn object_encryption_resolver(&self) -> Option<&dyn ObjectEncryptionResolver> {
self.object_encryption_resolver.get().map(Arc::as_ref)
}
/// Set this instance's S3 region.
///
/// Write-once: panics on a second write, preserving the startup fail-fast
+12 -111
View File
@@ -27,7 +27,7 @@ use crate::{
bucket::replication::{DynReplicationPool, ReplicationStats},
config::{get_global_storage_class, get_global_storage_class_snapshot, set_global_storage_class, storageclass},
disk::{DiskAPI, DiskOption, DiskStore, new_disk},
error::{Error, Result},
error::Result,
layout::endpoints::{EndpointServerPools, SetupType},
runtime::global::{
GLOBAL_BOOT_TIME, GLOBAL_LIFECYCLE_SYS, GLOBAL_LOCAL_NODE_NAME_FALLBACK, GLOBAL_ROOT_DISK_THRESHOLD,
@@ -46,6 +46,7 @@ use crate::{
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
use rustfs_config::server_config::{Config, get_global_server_config, set_global_server_config};
use rustfs_io_metrics::internode_metrics::global_internode_metrics;
use rustfs_kms::{ObjectEncryptionService, get_global_encryption_service};
use rustfs_lock::client::LockClient;
use s3s::dto::BucketLifecycleConfiguration;
use s3s::region::Region;
@@ -104,6 +105,10 @@ pub(crate) fn record_erasure_write_quorum_failure(stage: &'static str, dominant_
global_internode_metrics().record_erasure_write_quorum_failure(stage, dominant_error);
}
pub(crate) async fn object_encryption_service() -> Option<Arc<ObjectEncryptionService>> {
get_global_encryption_service().await
}
pub fn object_store_handle() -> Option<Arc<ECStore>> {
resolve_object_store_handle()
}
@@ -412,6 +417,10 @@ pub(crate) async fn clear_local_disk_id_map_for_test() {
local_disk_id_map_handle().write().await.clear();
}
pub(crate) async fn record_local_disk_id(instance_ctx: &Arc<InstanceContext>, disk_id: Uuid, endpoint: String) {
instance_ctx.local_disk_id_map().write().await.insert(disk_id, endpoint);
}
pub(crate) async fn replace_local_disk_id(previous: Option<Uuid>, current: Option<Uuid>, endpoint: String) {
let id_map = local_disk_id_map_handle();
let mut disk_id_map = id_map.write().await;
@@ -427,53 +436,6 @@ pub(crate) async fn replace_local_disk_id(previous: Option<Uuid>, current: Optio
}
}
pub(crate) async fn reconcile_local_disk_ids(
instance_ctx: &InstanceContext,
pool_endpoints: &[String],
selected: &[(Uuid, String)],
) {
let pool_endpoints = pool_endpoints.iter().map(String::as_str).collect::<HashSet<_>>();
let disk_id_map = instance_ctx.local_disk_id_map();
let mut disk_ids = disk_id_map.write().await;
disk_ids.retain(|_, registered_endpoint| !pool_endpoints.contains(registered_endpoint.as_str()));
disk_ids.extend(selected.iter().cloned());
}
pub(crate) async fn quarantine_local_disks(instance_ctx: &InstanceContext, endpoints: &[Endpoint]) -> Result<()> {
let slots = endpoints
.iter()
.map(|endpoint| {
Ok((
usize::try_from(endpoint.pool_idx).map_err(|_| Error::CorruptedFormat)?,
usize::try_from(endpoint.set_idx).map_err(|_| Error::CorruptedFormat)?,
usize::try_from(endpoint.disk_idx).map_err(|_| Error::CorruptedFormat)?,
))
})
.collect::<Result<Vec<_>>>()?;
let local_disk_map = instance_ctx.local_disk_map();
let mut local_disks = local_disk_map.write().await;
for endpoint in endpoints {
local_disks.insert(endpoint.to_string(), None);
}
drop(local_disks);
let set_drives = instance_ctx.local_disk_set_drives();
let mut local_set_drives = set_drives.write().await;
if local_set_drives.is_empty() {
return Ok(());
}
for (pool_idx, set_idx, disk_idx) in slots {
let disk = local_set_drives
.get_mut(pool_idx)
.and_then(|sets| sets.get_mut(set_idx))
.and_then(|disks| disks.get_mut(disk_idx))
.ok_or(Error::CorruptedFormat)?;
*disk = None;
}
Ok(())
}
pub(crate) async fn record_local_disks(instance_ctx: &Arc<InstanceContext>, disks: Vec<DiskStore>) {
let map = instance_ctx.local_disk_map();
let mut global_local_disk_map = map.write().await;
@@ -601,8 +563,8 @@ pub(crate) async fn init_tier_config_mgr(store: Arc<ECStore>) -> Result<()> {
#[cfg(test)]
mod tests {
use super::{
LockRegistry, clear_local_disk_id_map_for_test, local_disk_path_by_id, local_node_name, reconcile_local_disk_ids,
replace_local_disk_id, set_local_node_name,
LockRegistry, clear_local_disk_id_map_for_test, local_disk_path_by_id, local_node_name, replace_local_disk_id,
set_local_node_name,
};
use crate::disk::endpoint::Endpoint;
use rustfs_lock::{LocalClient, LockClient};
@@ -666,65 +628,4 @@ mod tests {
assert_eq!(local_disk_path_by_id(&disk_id).await, Some("endpoint-a".to_string()));
clear_local_disk_id_map_for_test().await;
}
#[tokio::test]
#[serial_test::serial]
async fn reconciling_pool_disk_ids_preserves_other_endpoints() {
let instance_ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
let process_ctx = crate::runtime::global::current_ctx();
let bootstrap_ctx = crate::runtime::instance::bootstrap_ctx();
let retained_id = Uuid::new_v4();
let removed_id = Uuid::new_v4();
let selected_id = Uuid::new_v4();
let process_sentinel = Uuid::new_v4();
let bootstrap_sentinel = Uuid::new_v4();
instance_ctx.local_disk_id_map().write().await.extend([
(retained_id, "endpoint-a".to_string()),
(removed_id, "endpoint-b".to_string()),
]);
process_ctx
.local_disk_id_map()
.write()
.await
.insert(process_sentinel, "endpoint-b".to_string());
bootstrap_ctx
.local_disk_id_map()
.write()
.await
.insert(bootstrap_sentinel, "endpoint-b".to_string());
reconcile_local_disk_ids(
&instance_ctx,
&["endpoint-b".to_string(), "endpoint-c".to_string()],
&[(selected_id, "endpoint-c".to_string())],
)
.await;
let disk_ids = instance_ctx.local_disk_id_map();
let disk_ids = disk_ids.read().await;
assert_eq!(disk_ids.get(&retained_id).map(String::as_str), Some("endpoint-a"));
assert_eq!(disk_ids.get(&removed_id), None);
assert_eq!(disk_ids.get(&selected_id).map(String::as_str), Some("endpoint-c"));
drop(disk_ids);
assert_eq!(
process_ctx
.local_disk_id_map()
.read()
.await
.get(&process_sentinel)
.map(String::as_str),
Some("endpoint-b")
);
assert_eq!(
bootstrap_ctx
.local_disk_id_map()
.read()
.await
.get(&bootstrap_sentinel)
.map(String::as_str),
Some("endpoint-b")
);
process_ctx.local_disk_id_map().write().await.remove(&process_sentinel);
bootstrap_ctx.local_disk_id_map().write().await.remove(&bootstrap_sentinel);
}
}
+3 -1
View File
@@ -505,7 +505,9 @@ impl SetDisks {
}
fn file_info_has_encryption_metadata(meta: &FileInfo) -> bool {
meta.metadata.keys().any(|name| http::is_object_encryption_marker(name))
meta.metadata
.keys()
.any(|name| http::is_encryption_metadata_key(name) || http::is_sse_header(name))
}
fn starts_with_ignore_ascii_case(value: &str, prefix: &str) -> bool {
+8 -230
View File
@@ -147,17 +147,15 @@ use rustfs_object_capacity::capacity_scope::{
CapacityScope, CapacityScopeDisk, current_dirty_generation, record_capacity_scope, record_global_dirty_scope,
};
use rustfs_s3_types::EventName;
#[cfg(test)]
use rustfs_utils::http::SSEC_ALGORITHM_HEADER;
use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
use rustfs_utils::http::headers::AMZ_STORAGE_CLASS;
use rustfs_utils::http::headers::{
CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _,
};
use rustfs_utils::http::{
SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
SUFFIX_RESTORE_OPERATION_ID, contains_key_str, get_header_map, get_str, insert_str, is_object_encryption_marker,
remove_header_map,
SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE,
SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_RESTORE_OPERATION_ID, contains_key_str,
get_header_map, get_str, insert_str, is_encryption_metadata_key, remove_header_map,
};
use rustfs_utils::{
HashAlgorithm,
@@ -672,7 +670,10 @@ pub(crate) fn strip_internal_multipart_metadata(metadata: &mut HashMap<String, S
}
fn should_persist_encryption_original_size(metadata: &HashMap<String, String>) -> bool {
metadata.keys().any(|key| is_object_encryption_marker(key))
metadata.keys().any(|key| is_encryption_metadata_key(key))
|| metadata.contains_key(SSEC_ALGORITHM_HEADER)
|| metadata.contains_key(SSEC_KEY_HEADER)
|| metadata.contains_key(SSEC_KEY_MD5_HEADER)
}
/// Per-set memoized capacity dirty scope.
@@ -4714,7 +4715,6 @@ mod tests {
use crate::layout::endpoints::SetupType;
use crate::object_api::BLOCK_SIZE_V2;
use crate::object_api::ObjectInfo;
use crate::set_disk::core::io_primitives::rename_fanout_barrier;
use crate::storage_api_contracts::{
heal::HealOperations as _, lifecycle::TransitionedObject, list::ListOperations as _, multipart::CompletePart,
namespace::NamespaceLocking as _, object::ObjectIO as _, object::ObjectOperations as _,
@@ -9566,15 +9566,6 @@ mod tests {
make_local_bucket_test_set_disks_with_drive_count(2).await
}
fn assert_exclusive_object_lock_held(set_disks: &SetDisks, bucket: &str, object: &str) {
let lock = set_disks
.local_lock_manager_for_test()
.get_lock_info(&ObjectKey::new(bucket, object))
.expect("object lock should be visible while rename is paused");
assert!(matches!(lock.mode, rustfs_lock::LockMode::Exclusive));
assert_eq!(lock.owner.as_ref(), set_disks.locker_owner.as_str());
}
async fn make_local_bucket_test_set_disks_with_drive_count(drive_count: usize) -> Arc<SetDisks> {
let format = FormatV3::new(1, drive_count);
let mut endpoints = Vec::new();
@@ -9609,9 +9600,7 @@ mod tests {
disks.push(Some(disk));
}
let instance_ctx = Arc::new(InstanceContext::new());
instance_ctx.update_erasure_type(SetupType::Erasure).await;
let set_disks = SetDisks::new_with_instance_ctx(
let set_disks = SetDisks::new(
"test-owner".to_string(),
Arc::new(RwLock::new(disks)),
drive_count,
@@ -9621,7 +9610,6 @@ mod tests {
endpoints,
format,
Vec::new(),
instance_ctx,
)
.await;
set_disks.set_test_storage_class_config(
@@ -10275,216 +10263,6 @@ mod tests {
));
}
#[tokio::test]
async fn conditional_replace_holds_object_lock_through_rename() {
let set_disks = make_local_bucket_test_set_disks().await;
let bucket = "bucket-conditional-replace-fence";
let object = "config/conditional-replace.json";
set_disks
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
let mut initial_reader = PutObjReader::from_vec(b"initial config".to_vec());
let initial = set_disks
.put_object(
bucket,
object,
&mut initial_reader,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
.expect("initial config should be written");
let initial_etag = initial.etag.expect("initial config should have an ETag");
let barrier = rename_fanout_barrier::arm(object, 0, rename_fanout_barrier::PHASE_RENAME);
let writer_store = set_disks.clone();
let expected_etag = initial_etag.clone();
let writer = tokio::spawn(async move {
let mut reader = PutObjReader::from_vec(b"replacement config".to_vec());
writer_store
.put_object(
bucket,
object,
&mut reader,
&ObjectOptions {
preserve_etag: Some("replacement-etag".to_string()),
http_preconditions: Some(HTTPPreconditions {
if_match: Some(expected_etag),
..Default::default()
}),
..Default::default()
},
)
.await
});
tokio::time::timeout(std::time::Duration::from_secs(30), barrier.wait_until_paused())
.await
.expect("conditional replace should reach the rename barrier");
assert_exclusive_object_lock_held(&set_disks, bucket, object);
barrier.release();
writer
.await
.expect("conditional writer task should finish")
.expect("matching conditional replace should commit");
assert!(
set_disks
.local_lock_manager_for_test()
.get_lock_info(&ObjectKey::new(bucket, object))
.is_none(),
"conditional replace should release the object lock after commit"
);
let contender = set_disks
.new_ns_lock(bucket, object)
.await
.expect("contender namespace lock should be created");
let contender_guard = contender
.get_write_lock(std::time::Duration::from_secs(30))
.await
.expect("contender should acquire after conditional replace commits");
drop(contender_guard);
let mut stale_reader = PutObjReader::from_vec(b"stale config".to_vec());
let err = set_disks
.put_object(
bucket,
object,
&mut stale_reader,
&ObjectOptions {
http_preconditions: Some(HTTPPreconditions {
if_match: Some(initial_etag),
..Default::default()
}),
..Default::default()
},
)
.await
.expect_err("the old ETag must fail after the fenced replacement commits");
assert_eq!(err, StorageError::PreconditionFailed);
}
#[tokio::test]
async fn repeated_body_write_keeps_etag_but_changes_data_dir_generation() {
let set_disks = make_local_bucket_test_set_disks().await;
let bucket = "bucket-write-generation";
let object = "config/write-generation.json";
let body = b"identical config body".to_vec();
set_disks
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
let mut first_reader = PutObjReader::from_vec(body.clone());
let first = set_disks
.put_object(
bucket,
object,
&mut first_reader,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
.expect("first config body should be written");
let mut second_reader = PutObjReader::from_vec(body);
let second = set_disks
.put_object(
bucket,
object,
&mut second_reader,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
.expect("identical config body should be rewritten");
assert_eq!(first.etag, second.etag, "content ETag should expose the ABA collision");
assert_ne!(first.data_dir, second.data_dir, "each committed body write needs a unique generation");
assert!(first.data_dir.is_some() && second.data_dir.is_some());
}
#[tokio::test]
async fn conditional_create_holds_object_lock_through_rename() {
let set_disks = make_local_bucket_test_set_disks().await;
let bucket = "bucket-conditional-create-fence";
let object = "config/conditional-create.json";
set_disks
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
let barrier = rename_fanout_barrier::arm(object, 0, rename_fanout_barrier::PHASE_RENAME);
let writer_store = set_disks.clone();
let writer = tokio::spawn(async move {
let mut reader = PutObjReader::from_vec(b"created config".to_vec());
writer_store
.put_object(
bucket,
object,
&mut reader,
&ObjectOptions {
http_preconditions: Some(HTTPPreconditions {
if_none_match: Some("*".to_string()),
..Default::default()
}),
..Default::default()
},
)
.await
});
tokio::time::timeout(std::time::Duration::from_secs(30), barrier.wait_until_paused())
.await
.expect("conditional create should reach the rename barrier");
assert_exclusive_object_lock_held(&set_disks, bucket, object);
barrier.release();
writer
.await
.expect("conditional writer task should finish")
.expect("first conditional create should commit");
assert!(
set_disks
.local_lock_manager_for_test()
.get_lock_info(&ObjectKey::new(bucket, object))
.is_none(),
"conditional create should release the object lock after commit"
);
let contender = set_disks
.new_ns_lock(bucket, object)
.await
.expect("contender namespace lock should be created");
let contender_guard = contender
.get_write_lock(std::time::Duration::from_secs(30))
.await
.expect("contender should acquire after conditional create commits");
drop(contender_guard);
let mut duplicate_reader = PutObjReader::from_vec(b"duplicate config".to_vec());
let err = set_disks
.put_object(
bucket,
object,
&mut duplicate_reader,
&ObjectOptions {
http_preconditions: Some(HTTPPreconditions {
if_none_match: Some("*".to_string()),
..Default::default()
}),
..Default::default()
},
)
.await
.expect_err("a second create-only write must not replace the committed config");
assert_eq!(err, StorageError::PreconditionFailed);
}
#[tokio::test]
async fn set_level_if_none_match_fails_closed_without_read_quorum() {
let set_disks = make_local_bucket_test_set_disks_with_drive_count(4).await;
+2 -71
View File
@@ -42,7 +42,6 @@ use crate::object_api::{GetObjectBodySource, get_object_body_cache_hook_suppress
use crate::services::tier::tier::{TierConfigMgr, TierOperationLease};
use crate::store::ECStore;
use futures::FutureExt as _;
use http::HeaderValue;
use std::future::Future;
fn erasure_from_file_info(fi: &FileInfo, uses_legacy: bool) -> Result<coding::Erasure> {
@@ -50,17 +49,6 @@ fn erasure_from_file_info(fi: &FileInfo, uses_legacy: bool) -> Result<coding::Er
.map_err(Error::from)
}
async fn get_object_reader_with_context(
ctx: &InstanceContext,
reader: Box<dyn AsyncRead + Unpin + Send + Sync>,
range: Option<HTTPRangeSpec>,
object_info: &ObjectInfo,
opts: &ObjectOptions,
headers: &HeaderMap<HeaderValue>,
) -> Result<(GetObjectReader, usize, i64)> {
GetObjectReader::new_with_resolver(reader, range, object_info, opts, headers, ctx.object_encryption_resolver()).await
}
/// Length of the full plaintext body when — and only when — this read's output
/// is exactly the object's complete plaintext, so the app-layer body cache may
/// serve it in place of the erasure read.
@@ -725,8 +713,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
size_bucket,
);
record_get_object_reader_path_observation(GET_OBJECT_PATH_CODEC_STREAMING, object_class, size_bucket);
let (mut reader, _offset, _length) =
get_object_reader_with_context(&self.ctx, stream, range, &object_info, opts, &h).await?;
let (mut reader, _offset, _length) = GetObjectReader::new(stream, range, &object_info, opts, &h).await?;
// Carry the hook probe result so the app layer skips its
// now-redundant lookup on the streaming miss path (ODC-16).
reader.body_source = body_source;
@@ -758,8 +745,7 @@ impl crate::storage_api_contracts::object::ObjectIO for SetDisks {
let (rd, wd) = tokio::io::duplex(duplex_buffer_size);
debug!(bucket, object, duplex_buffer_size, "Created duplex pipe for object data transfer");
let (mut reader, offset, length) =
get_object_reader_with_context(&self.ctx, Box::new(rd), range, &object_info, opts, &h).await?;
let (mut reader, offset, length) = GetObjectReader::new(Box::new(rd), range, &object_info, opts, &h).await?;
// Carry the hook probe result so the app layer skips its now-redundant
// lookup on the streaming miss path (ODC-16).
reader.body_source = body_source;
@@ -4550,61 +4536,6 @@ mod erasure_construction_tests {
}
}
#[cfg(test)]
mod object_encryption_resolver_wiring_tests {
use super::*;
use crate::object_api::{EncryptionResolutionError, ObjectEncryptionResolver, ReadEncryptionMaterial, ReadEncryptionRequest};
use std::io::Cursor;
use std::sync::atomic::{AtomicUsize, Ordering};
struct CountingResolver {
calls: AtomicUsize,
}
#[async_trait::async_trait]
impl ObjectEncryptionResolver for CountingResolver {
async fn resolve_read_material(
&self,
_request: ReadEncryptionRequest<'_>,
) -> std::result::Result<Option<ReadEncryptionMaterial>, EncryptionResolutionError> {
self.calls.fetch_add(1, Ordering::Relaxed);
Ok(None)
}
}
#[tokio::test]
async fn get_object_reader_forwards_instance_resolver() {
let resolver = Arc::new(CountingResolver {
calls: AtomicUsize::new(0),
});
let ctx = InstanceContext::new();
assert!(
ctx.set_object_encryption_resolver(resolver.clone()).is_ok(),
"fresh context should accept resolver"
);
let object_info = ObjectInfo {
bucket: "bucket".to_string(),
name: "object".to_string(),
size: 1,
user_defined: Arc::new(HashMap::from([("x-amz-server-side-encryption".to_string(), "AES256".to_string())])),
..Default::default()
};
let result = get_object_reader_with_context(
&ctx,
Box::new(Cursor::new(Vec::<u8>::new())),
None,
&object_info,
&ObjectOptions::default(),
&HeaderMap::new(),
)
.await;
assert!(result.is_err(), "resolver returning no material must fail closed");
assert_eq!(resolver.calls.load(Ordering::Relaxed), 1);
}
}
#[cfg(test)]
pub(in crate::set_disk::ops) mod hermetic_set_disks_support {
//! Shared hermetic `SetDisks` construction for the ops tests below: the
+7 -7
View File
@@ -199,7 +199,7 @@ impl SetDisks {
);
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub async fn read_version_optimized(
&self,
bucket: &str,
@@ -238,7 +238,7 @@ impl SetDisks {
}
#[tracing::instrument(level = "debug", skip(self))]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn get_object_fileinfo(
&self,
bucket: &str,
@@ -410,7 +410,7 @@ impl SetDisks {
Ok((fi, parts_metadata, op_online_disks))
}
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn get_object_info_and_quorum(
&self,
bucket: &str,
@@ -605,7 +605,7 @@ impl SetDisks {
}
#[allow(clippy::too_many_arguments)]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn get_object_with_fileinfo<W>(
// &self,
bucket: &str,
@@ -1140,7 +1140,7 @@ impl SetDisks {
}
#[allow(clippy::too_many_arguments)]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn get_object_decode_reader_with_fileinfo(
bucket: &str,
object: &str,
@@ -1296,7 +1296,7 @@ impl SetDisks {
}
#[allow(clippy::too_many_arguments)]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
async fn build_codec_streaming_part_reader(
bucket: &str,
object: &str,
@@ -1469,7 +1469,7 @@ fn multipart_part_checksum_algo(fi: &FileInfo, part_number: usize) -> HashAlgori
/// `get_object_with_fileinfo` (backlog#870) so both report the same
/// stage-duration semantics.
#[allow(clippy::too_many_arguments)]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
async fn setup_multipart_part_readers(
files: &[FileInfo],
disks: &[Option<DiskStore>],
+1 -21
View File
@@ -313,8 +313,7 @@ impl ECStore {
let mut times = 0;
let mut interval = 1;
loop {
match init_format::connect_load_init_formats_with_instance_ctx(
&instance_ctx,
match init_format::connect_load_init_formats(
pool_first_is_local,
&mut disks,
pool_eps.set_count,
@@ -1260,16 +1259,6 @@ mod tests {
let registered: Vec<String> = instance_ctx.local_disk_map().read().await.keys().cloned().collect();
assert_eq!(registered.len(), 4, "the passed context must register all four local disks");
let registered_disk_ids = instance_ctx.local_disk_id_map();
let registered_disk_ids = registered_disk_ids.read().await;
assert_eq!(registered_disk_ids.len(), 4, "the passed context must publish all four disk IDs");
for endpoint in registered_disk_ids.values() {
assert!(
registered.contains(endpoint),
"every disk ID in the passed context must resolve to one of its registered endpoints"
);
}
drop(registered_disk_ids);
let bootstrap = crate::runtime::instance::bootstrap_ctx();
assert_ne!(
bootstrap.deployment_id(),
@@ -1284,15 +1273,6 @@ mod tests {
"the bootstrap context must not absorb the fresh store's disks"
);
}
drop(bootstrap_map);
let bootstrap_disk_ids = bootstrap.local_disk_id_map();
let bootstrap_disk_ids = bootstrap_disk_ids.read().await;
for endpoint in bootstrap_disk_ids.values() {
assert!(
!registered.contains(endpoint),
"the bootstrap context must not absorb the fresh store's disk IDs"
);
}
}
#[tokio::test]
+58 -227
View File
@@ -16,8 +16,6 @@ use crate::config::storageclass;
use crate::disk::error_reduce::{count_errs, reduce_write_quorum_errs};
use crate::disk::{self, DiskAPI};
use crate::error::{Error, Result};
use crate::runtime::instance::InstanceContext;
use crate::runtime::sources::{quarantine_local_disks, reconcile_local_disk_ids};
use crate::{
disk::{
DiskInfoOptions, DiskOption, DiskStore, FORMAT_CONFIG_FILE, MIGRATING_META_BUCKET, RUSTFS_META_BUCKET,
@@ -28,10 +26,7 @@ use crate::{
layout::endpoints::Endpoints,
};
use futures::{future::join_all, stream, stream::StreamExt};
use std::{
collections::{HashMap, HashSet},
sync::Arc,
};
use std::collections::{HashMap, HashSet};
use tokio::io::AsyncReadExt;
use tracing::{debug, error, info, warn};
use uuid::Uuid;
@@ -67,25 +62,12 @@ pub async fn init_disks(eps: &Endpoints, opt: &DiskOption) -> (Vec<Option<DiskSt
(res, errors)
}
#[cfg(test)]
pub async fn connect_load_init_formats(
first_disk: bool,
disks: &mut [Option<DiskStore>],
set_count: usize,
set_drive_count: usize,
deployment_id: Option<Uuid>,
) -> Result<FormatV3> {
let instance_ctx = crate::runtime::global::current_ctx();
connect_load_init_formats_with_instance_ctx(&instance_ctx, first_disk, disks, set_count, set_drive_count, deployment_id).await
}
pub(crate) async fn connect_load_init_formats_with_instance_ctx(
instance_ctx: &Arc<InstanceContext>,
first_disk: bool,
disks: &mut [Option<DiskStore>],
set_count: usize,
set_drive_count: usize,
deployment_id: Option<Uuid>,
) -> Result<FormatV3> {
let (formats, errs) = load_format_erasure_all(disks, false).await;
@@ -116,7 +98,7 @@ pub(crate) async fn connect_load_init_formats_with_instance_ctx(
match try_migrate_format(disks, &formats, set_count, set_drive_count).await {
Ok(LegacyFormatOutcome::Migrated { format, quorum_members }) => {
info!("Migrated format from MinIO config");
retain_format_quorum_members(instance_ctx, disks, &format, &quorum_members, set_drive_count).await?;
retain_format_quorum_members(disks, &format, &quorum_members, set_drive_count).await?;
return Ok(*format);
}
Ok(LegacyFormatOutcome::Incompatible) => {
@@ -133,7 +115,7 @@ pub(crate) async fn connect_load_init_formats_with_instance_ctx(
Err(e) => return Err(e),
}
if all_unformatted {
let fm = init_format_erasure(instance_ctx, disks, set_count, set_drive_count, deployment_id).await?;
let fm = init_format_erasure(disks, set_count, set_drive_count, deployment_id).await?;
return Ok(fm);
}
}
@@ -158,13 +140,12 @@ pub(crate) async fn connect_load_init_formats_with_instance_ctx(
None => select_format_erasure_in_quorum(&formats, 0)?,
};
check_format_erasure_value_for_topology(&fm, formats.len(), set_drive_count)?;
retain_format_quorum_members(instance_ctx, disks, &fm, &quorum_members, set_drive_count).await?;
retain_format_quorum_members(disks, &fm, &quorum_members, set_drive_count).await?;
Ok(fm)
}
async fn retain_format_quorum_members(
instance_ctx: &Arc<InstanceContext>,
disks: &mut [Option<DiskStore>],
format: &FormatV3,
quorum_members: &[bool],
@@ -173,73 +154,26 @@ async fn retain_format_quorum_members(
if set_drive_count == 0 || quorum_members.len() != disks.len() {
return Err(Error::CorruptedFormat);
}
let pool_idx = disks
.iter()
.flatten()
.map(|disk| disk.endpoint().pool_idx)
.find_map(|pool_idx| usize::try_from(pool_idx).ok());
let registered_endpoints = if let Some(pool_idx) = pool_idx {
instance_ctx
.local_disk_set_drives()
.read()
.await
.get(pool_idx)
.map(|sets| {
sets.iter()
.flat_map(|set| set.iter())
.map(|disk| disk.as_ref().map(|disk| disk.endpoint()))
.collect::<Vec<_>>()
})
.filter(|endpoints| endpoints.len() == disks.len())
} else {
None
};
let endpoints = disks
.iter()
.enumerate()
.map(|(index, disk)| {
disk.as_ref()
.map(|disk| disk.endpoint())
.or_else(|| registered_endpoints.as_ref()?.get(index)?.clone())
})
.collect::<Vec<_>>();
let mut member_disk_ids = vec![None; disks.len()];
let mut local_pool_endpoints = Vec::new();
let mut selected_local_disk_ids = Vec::new();
let mut quarantined_endpoints = Vec::new();
for (index, ((disk, endpoint), belongs_to_quorum)) in disks.iter().zip(&endpoints).zip(quorum_members).enumerate() {
if let Some(endpoint) = endpoint.as_ref().filter(|endpoint| endpoint.is_local) {
local_pool_endpoints.push(endpoint.to_string());
if !belongs_to_quorum {
quarantined_endpoints.push(endpoint.clone());
}
for (disk, belongs_to_quorum) in disks.iter().zip(quorum_members) {
if !belongs_to_quorum && let Some(disk) = disk {
disk.set_disk_id(None).await?;
}
}
for (index, (disk, belongs_to_quorum)) in disks.iter().zip(quorum_members).enumerate() {
if !belongs_to_quorum {
continue;
}
let disk = disk.as_ref().ok_or(Error::CorruptedFormat)?;
let disk_id = format
.erasure
.sets
.get(index / set_drive_count)
.and_then(|set| set.get(index % set_drive_count))
.copied()
.ok_or(Error::CorruptedFormat)?;
member_disk_ids[index] = Some(disk_id);
if disk.is_local() {
selected_local_disk_ids.push((disk_id, disk.endpoint().to_string()));
}
disk.as_ref()
.ok_or(Error::CorruptedFormat)?
.set_disk_id(Some(*disk_id))
.await?;
}
quarantine_local_disks(instance_ctx, &quarantined_endpoints).await?;
for (disk, disk_id) in disks.iter().zip(member_disk_ids) {
if let Some(disk) = disk {
disk.set_disk_id_state(disk_id).await?;
}
}
reconcile_local_disk_ids(instance_ctx, &local_pool_endpoints, &selected_local_disk_ids).await;
for (disk, belongs_to_quorum) in disks.iter_mut().zip(quorum_members) {
if !belongs_to_quorum {
*disk = None;
@@ -273,8 +207,7 @@ pub fn check_disk_fatal_errs(errs: &[Option<DiskError>]) -> disk::error::Result<
}
async fn init_format_erasure(
instance_ctx: &Arc<InstanceContext>,
disks: &mut [Option<DiskStore>],
disks: &[Option<DiskStore>],
set_count: usize,
set_drive_count: usize,
deployment_id: Option<Uuid>,
@@ -294,11 +227,9 @@ async fn init_format_erasure(
}
}
write_format_file_all(disks, &fms).await?;
let format = get_format_erasure_in_quorum(&fms, 0)?;
let quorum_members = vec![true; disks.len()];
retain_format_quorum_members(instance_ctx, disks, &format, &quorum_members, set_drive_count).await?;
Ok(format)
save_format_file_all(disks, &fms).await?;
get_format_erasure_in_quorum(&fms, 0)
}
/// Outcome of attempting to migrate an on-disk MinIO `format.json`.
@@ -417,59 +348,44 @@ async fn try_migrate_format(
.iter()
.zip(&formats_to_write)
.zip(rustfs_formats)
.filter_map(|((disk, format), existing)| {
if existing.is_some() {
return None;
}
Some(write_format_file(disk.as_ref()?, format.as_ref()?))
});
.filter(|&((disk, _), existing)| disk.is_some() && existing.is_none())
.map(|((disk, format), _)| save_format_file(disk, format));
let mut write_error = None;
for result in join_all(writes).await {
if let Err(error) = result {
write_error.get_or_insert(error);
}
}
let (persisted_formats, persisted_errors) = load_format_erasure_all(disks, false).await;
let Some((persisted_format, quorum_members)) =
select_persisted_migration_format(&persisted_formats, persisted_errors, &format, set_drive_count, write_error)?
else {
return Ok(LegacyFormatOutcome::Incompatible);
};
Ok(LegacyFormatOutcome::Migrated {
format: Box::new(persisted_format),
quorum_members,
})
}
fn select_persisted_migration_format(
persisted_formats: &[Option<FormatV3>],
persisted_errors: Vec<Option<DiskError>>,
reference: &FormatV3,
set_drive_count: usize,
write_error: Option<DiskError>,
) -> Result<Option<(FormatV3, Vec<bool>)>> {
if !formats_match_reference_slots(persisted_formats, reference, 0) {
return Ok(None);
for disk in disks.iter().flatten() {
disk.set_disk_id(None).await?;
}
let quorum_error = match select_format_erasure_in_quorum(persisted_formats, 0) {
Ok((format, quorum_members)) => {
check_format_erasure_value_for_topology(&format, persisted_formats.len(), set_drive_count)?;
return Ok(Some((format, quorum_members)));
}
Err(error) => error,
};
let (persisted_formats, persisted_errors) = load_format_erasure_all(disks, false).await;
if let Some(error) = persisted_errors
.into_iter()
.flatten()
.find(|error| !matches!(error, DiskError::UnformattedDisk | DiskError::DiskNotFound))
{
return match error {
DiskError::CorruptedFormat | DiskError::CorruptedBackend | DiskError::InconsistentDisk => Ok(None),
DiskError::CorruptedFormat | DiskError::CorruptedBackend | DiskError::InconsistentDisk => {
Ok(LegacyFormatOutcome::Incompatible)
}
error => Err(error.into()),
};
}
Err(write_error.map_or(quorum_error, Into::into))
if !formats_match_reference_slots(&persisted_formats, &format, 0) {
return Ok(LegacyFormatOutcome::Incompatible);
}
let (persisted_format, quorum_members) = match select_format_erasure_in_quorum(&persisted_formats, 0) {
Ok(selected) => selected,
Err(error) => return Err(write_error.map_or(error, Into::into)),
};
check_format_erasure_value_for_topology(&persisted_format, disks.len(), set_drive_count)?;
Ok(LegacyFormatOutcome::Migrated {
format: Box::new(persisted_format),
quorum_members,
})
}
fn legacy_format_max_bytes(disk_count: usize) -> Result<usize> {
@@ -760,12 +676,13 @@ pub async fn load_format_erasure(disk: &DiskStore, heal: bool) -> disk::error::R
Ok(fm)
}
async fn write_format_file_all(disks: &[Option<DiskStore>], formats: &[Option<FormatV3>]) -> disk::error::Result<()> {
let futures = disks.iter().zip(formats).map(|(disk, format)| async move {
let disk = disk.as_ref().ok_or(DiskError::DiskNotFound)?;
let format = format.as_ref().ok_or_else(|| DiskError::other("format is none"))?;
write_format_file(disk, format).await
});
async fn save_format_file_all(disks: &[Option<DiskStore>], formats: &[Option<FormatV3>]) -> disk::error::Result<()> {
let mut futures = Vec::with_capacity(disks.len());
for (i, disk) in disks.iter().enumerate() {
futures.push(save_format_file(disk, &formats[i]));
}
let mut errors = Vec::with_capacity(disks.len());
let results = join_all(futures).await;
@@ -796,13 +713,6 @@ pub async fn save_format_file(disk: &Option<DiskStore>, format: &Option<FormatV3
return Err(DiskError::other("format is none"));
};
write_format_file(disk, format).await?;
disk.set_disk_id(Some(format.erasure.this)).await?;
Ok(())
}
async fn write_format_file(disk: &DiskStore, format: &FormatV3) -> disk::error::Result<()> {
let json_data = format.to_json()?;
let tmpfile = Uuid::new_v4().to_string();
@@ -813,6 +723,8 @@ async fn write_format_file(disk: &DiskStore, format: &FormatV3) -> disk::error::
disk.rename_file(RUSTFS_META_BUCKET, tmpfile.as_str(), RUSTFS_META_BUCKET, FORMAT_CONFIG_FILE)
.await?;
disk.set_disk_id(Some(format.erasure.this)).await?;
Ok(())
}
@@ -826,9 +738,7 @@ pub fn ec_drives_no_config(set_drive_count: usize) -> Result<usize> {
mod tests {
use super::*;
use crate::layout::endpoint::Endpoint;
use crate::runtime::global::{current_ctx, reset_local_disk_test_state};
use crate::runtime::sources::{local_disk_path_by_id, record_local_disks};
use crate::store::peer::{find_local_disk_by_ref, prewarm_local_disk_id_map_with_instance_ctx};
use crate::runtime::sources::{clear_local_disk_id_map_for_test, local_disk_path_by_id};
use serial_test::serial;
async fn local_disks(count: usize) -> (tempfile::TempDir, Vec<Option<DiskStore>>) {
@@ -1135,7 +1045,7 @@ mod tests {
#[tokio::test]
#[serial]
async fn existing_format_load_publishes_only_validated_quorum_disk_ids() {
reset_local_disk_test_state().await;
clear_local_disk_id_map_for_test().await;
let (_temp_dir, mut disks) = local_disks(5).await;
let canonical = FormatV3::new(1, 5);
for (index, disk) in disks.iter().enumerate().take(3) {
@@ -1145,56 +1055,19 @@ mod tests {
.await
.expect("canonical format should be written");
}
let canonical_id = canonical.erasure.sets[0][0];
let mut wrong_slot = canonical.clone();
wrong_slot.erasure.this = wrong_slot.erasure.sets[0][0];
save_format_file(&disks[3], &Some(wrong_slot))
.await
.expect("wrong-slot format should be written");
let mut foreign = FormatV3::new(1, 5);
foreign.erasure.this = foreign.erasure.sets[0][4];
let foreign_id = foreign.erasure.this;
save_format_file(&disks[4], &Some(foreign))
.await
.expect("foreign format should be written");
let mut collision = FormatV3::new(1, 5);
collision.erasure.sets[0][3] = canonical_id;
collision.erasure.this = canonical_id;
save_format_file(&disks[3], &Some(collision))
.await
.expect("foreign collision format should be written");
let canonical_id = canonical.erasure.sets[0][0];
let canonical_endpoint = disks[0].as_ref().expect("canonical disk should exist").endpoint().to_string();
let collision_endpoint = disks[3].as_ref().expect("collision disk should exist").endpoint().to_string();
let foreign_endpoint = disks[4].as_ref().expect("foreign disk should exist").endpoint().to_string();
let prewarm_endpoints = Endpoints::from(
disks
.iter()
.map(|disk| disk.as_ref().expect("test disk should exist").endpoint())
.collect::<Vec<_>>(),
);
for disk in disks.iter().flatten() {
disk.set_disk_id(None).await.expect("test disk ID should be reset");
}
let (prewarm_disks, prewarm_errors) = init_disks(
&prewarm_endpoints,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await;
assert!(prewarm_errors.iter().all(Option::is_none));
let prewarm_disks = prewarm_disks
.into_iter()
.map(|disk| disk.expect("prewarm disk should exist"))
.collect::<Vec<_>>();
let instance_ctx = current_ctx();
record_local_disks(&instance_ctx, prewarm_disks.clone()).await;
*instance_ctx.local_disk_set_drives().write().await = vec![vec![prewarm_disks.into_iter().map(Some).collect()]];
prewarm_local_disk_id_map_with_instance_ctx(&instance_ctx).await;
instance_ctx
.local_disk_id_map()
.write()
.await
.insert(canonical_id, collision_endpoint.clone());
assert!(local_disk_path_by_id(&foreign_id).await.is_some(), "foreign ID should be prewarmed");
assert_eq!(local_disk_path_by_id(&canonical_id).await, Some(collision_endpoint));
disks[4] = None;
connect_load_init_formats(true, &mut disks, 1, 5, None)
.await
@@ -1202,51 +1075,9 @@ mod tests {
assert!(disks[..3].iter().all(Option::is_some));
assert!(disks[3..].iter().all(Option::is_none));
assert_eq!(local_disk_path_by_id(&canonical_id).await, Some(canonical_endpoint.clone()));
assert_eq!(local_disk_path_by_id(&canonical_id).await, Some(canonical_endpoint));
assert_eq!(local_disk_path_by_id(&foreign_id).await, None);
assert!(
instance_ctx
.local_disk_map()
.read()
.await
.get(&foreign_endpoint)
.is_some_and(Option::is_none)
);
assert!(instance_ctx.local_disk_set_drives().read().await[0][0][4].is_none());
assert!(find_local_disk_by_ref(&foreign_id.to_string()).await.is_none());
assert_eq!(
find_local_disk_by_ref(&canonical_id.to_string())
.await
.map(|disk| disk.endpoint().to_string()),
Some(canonical_endpoint)
);
reset_local_disk_test_state().await;
}
#[test]
fn persisted_migration_quorum_ignores_nonmember_read_errors() {
for drive_count in [3, 4] {
let reference = FormatV3::new(1, drive_count);
let required = drive_count / 2 + 1;
let mut formats = vec![None; drive_count];
let mut errors = (0..drive_count).map(|_| None).collect::<Vec<Option<DiskError>>>();
for (index, format) in formats.iter_mut().enumerate().take(required) {
let mut disk_format = reference.clone();
disk_format.erasure.this = reference.erasure.sets[0][index];
*format = Some(disk_format);
}
errors[drive_count - 1] = Some(DiskError::FaultyDisk);
let (selected, members) =
select_persisted_migration_format(&formats, errors, &reference, drive_count, Some(DiskError::FaultyDisk))
.expect("a persisted strict majority must outrank a nonmember read error")
.expect("the persisted formats should be compatible");
assert_eq!(selected.shared_identity(), reference.shared_identity());
assert_eq!(members.iter().filter(|member| **member).count(), required);
assert!(members[..required].iter().all(|member| *member));
assert!(members[required..].iter().all(|member| !*member));
}
clear_local_disk_id_map_for_test().await;
}
#[tokio::test]
+4 -23
View File
@@ -4861,7 +4861,10 @@ async fn poll_merge_head(rx: &CancellationToken, in_channels: &mut [Receiver<Met
async fn send_or_cancel(rx: &CancellationToken, out_channel: &Sender<MetaCacheEntry>, entry: MetaCacheEntry) -> Result<bool> {
tokio::select! {
result = out_channel.send(entry) => Ok(result.is_ok()),
result = out_channel.send(entry) => {
result.map_err(Error::other)?;
Ok(true)
}
_ = rx.cancelled() => Ok(false),
}
}
@@ -9855,28 +9858,6 @@ mod test {
assert_eq!(results, vec!["obj-a", "obj-b"]);
}
#[tokio::test]
async fn merge_entry_channels_treats_dropped_output_receiver_as_completion() {
let (tx_a, rx_a) = mpsc::channel(4);
let (tx_b, rx_b) = mpsc::channel(4);
let (out_tx, out_rx) = mpsc::channel(1);
tx_a.send(test_meta_entry("obj-a")).await.unwrap();
tx_b.send(test_meta_entry("obj-b")).await.unwrap();
drop(tx_a);
drop(tx_b);
drop(out_rx);
let result = timeout(
Duration::from_secs(1),
merge_entry_channels(CancellationToken::new(), vec![rx_a, rx_b], out_tx, 1),
)
.await
.expect("merge should stop promptly when its consumer disconnects");
assert!(result.is_ok(), "consumer disconnect must not surface as a merge worker error");
}
#[test]
fn walk_ascending_versions_contract_reverses_newest_first_metadata() {
// Documents the invariant the walk `versions_sort` handling relies on:
+1 -1
View File
@@ -238,7 +238,7 @@ impl ECStore {
}
#[instrument(skip(self, data))]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn handle_put_object_part(
&self,
bucket: &str,
+2 -2
View File
@@ -815,7 +815,7 @@ impl ECStore {
}
#[instrument(level = "debug", skip(self))]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn handle_get_object_reader(
&self,
bucket: &str,
@@ -849,7 +849,7 @@ impl ECStore {
}
#[instrument(level = "debug", skip(self, data))]
#[hotpath::measure]
#[cfg_attr(feature = "hotpath", hotpath::measure)]
pub(super) async fn handle_put_object(
&self,
bucket: &str,
+2 -73
View File
@@ -28,26 +28,8 @@ async fn remember_local_disk_id(disk: &DiskStore) -> Option<Uuid> {
async fn remember_local_disk_id_with_instance_ctx(instance_ctx: &Arc<InstanceContext>, disk: &DiskStore) -> Option<Uuid> {
let disk_id = disk.get_disk_id().await.ok().flatten()?;
record_local_disk_id_if_active(instance_ctx, disk, disk_id)
.await
.then_some(disk_id)
}
async fn record_local_disk_id_if_active(instance_ctx: &Arc<InstanceContext>, disk: &DiskStore, disk_id: Uuid) -> bool {
let endpoint = disk.endpoint().to_string();
let local_disk_map = instance_ctx.local_disk_map();
let local_disks = local_disk_map.read().await;
let Some(active_disk) = local_disks.get(&endpoint).and_then(Option::as_ref) else {
return false;
};
if !Arc::ptr_eq(active_disk, disk) {
return false;
}
// Lock order is local_disk_map -> local_disk_id_map so quarantine is the
// linearization point for rejecting an in-flight stale disk snapshot.
instance_ctx.local_disk_id_map().write().await.insert(disk_id, endpoint);
true
runtime_sources::record_local_disk_id(instance_ctx, disk_id, disk.endpoint().to_string()).await;
Some(disk_id)
}
pub async fn find_local_disk(disk_path: &str) -> Option<DiskStore> {
@@ -246,7 +228,6 @@ pub async fn get_disk_infos(disks: &[Option<DiskStore>]) -> Vec<Option<DiskInfo>
#[cfg(test)]
mod tests {
use super::*;
use crate::disk::new_disk;
use crate::layout::endpoints::{Endpoints, PoolEndpoints};
fn single_local_disk_pools(dir: &std::path::Path) -> EndpointServerPools {
@@ -333,56 +314,4 @@ mod tests {
);
}
}
#[tokio::test]
async fn stale_local_disk_snapshot_cannot_repopulate_the_id_registry() {
let temp_dir = tempfile::tempdir().expect("create temp disk dir");
let endpoint_pools = single_local_disk_pools(temp_dir.path());
let instance_ctx = Arc::new(InstanceContext::new());
init_local_disks_with_instance_ctx(&instance_ctx, endpoint_pools)
.await
.expect("local disk should be registered");
let disk = instance_ctx
.local_disk_map()
.read()
.await
.values()
.find_map(|disk| disk.clone())
.expect("registered local disk");
let endpoint = disk.endpoint().to_string();
let disk_id = Uuid::new_v4();
let local_disk_map = instance_ctx.local_disk_map();
let mut quarantine = local_disk_map.write().await;
let replacement = new_disk(
&disk.endpoint(),
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("replacement disk should initialize");
assert!(!Arc::ptr_eq(&disk, &replacement));
let task_ctx = instance_ctx.clone();
let task_disk = disk.clone();
let remember = tokio::spawn(async move { record_local_disk_id_if_active(&task_ctx, &task_disk, disk_id).await });
tokio::task::yield_now().await;
quarantine.insert(endpoint.clone(), Some(replacement.clone()));
drop(quarantine);
assert!(!remember.await.expect("stale lookup task should complete"));
assert!(!instance_ctx.local_disk_id_map().read().await.contains_key(&disk_id));
let active = instance_ctx
.local_disk_map()
.read()
.await
.get(&endpoint)
.cloned()
.flatten()
.expect("replacement disk should remain registered");
assert!(Arc::ptr_eq(&active, &replacement));
assert!(record_local_disk_id_if_active(&instance_ctx, &replacement, disk_id).await);
assert_eq!(instance_ctx.local_disk_id_map().read().await.get(&disk_id), Some(&endpoint));
}
}
+2 -2
View File
@@ -2,7 +2,7 @@
## MinIO-generated encrypted fixtures
`rustfs/src/storage/minio_generated_read_test.rs` validates the `bitrot -> GetObjectReader` path against raw MinIO backend data captured by
`minio_generated_read_test.rs` validates the `bitrot -> GetObjectReader` path against raw MinIO backend data captured by
`.\rustfs\scripts\minio_fixture_lab\lab.py`.
It currently covers multipart fixtures for:
@@ -20,5 +20,5 @@ Example:
```powershell
$env:RUSTFS_MINIO_FIXTURE_ROOT = '.\rustfs\tmp\minio-fixture-lab-local-key'
$env:RUSTFS_MINIO_STATIC_KMS_KEY_B64 = '<base64-32-byte-local-minio-kms-key>'
cargo +1.97.1 test -p rustfs --features rio-v2 storage::minio_generated_read_test --lib -- --ignored
cargo +1.97.1 test -p rustfs-ecstore --features rio-v2 --test minio_generated_read_test -- --ignored
```
@@ -4,13 +4,14 @@ use std::fs;
use std::io::Cursor;
use std::path::{Path, PathBuf};
use super::sse::SseObjectEncryptionResolver;
use super::storage_api::ecstore_test_support::{
DiskAPI as _, DiskOption, Endpoint, Erasure, GetObjectReader, ObjectInfo, ObjectOptions, create_bitrot_reader, new_disk,
};
mod storage_api;
use rustfs_filemeta::{FileInfo, FileInfoOpts, get_file_info};
use serde::Deserialize;
use sha2::{Digest, Sha256};
use storage_api::minio_generated_read::{
DiskAPI as _, DiskOption, Endpoint, Erasure, GetObjectReader, ObjectInfo, ObjectOptions, create_bitrot_reader, new_disk,
};
use temp_env::async_with_vars;
use tokio::io::{AsyncReadExt, AsyncWrite};
@@ -48,7 +49,7 @@ impl AsyncWrite for VecAsyncWriter {
fn fixture_root() -> PathBuf {
std::env::var_os("RUSTFS_MINIO_FIXTURE_ROOT")
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../crates/rio-v2/tests/fixtures/minio-generated"))
.unwrap_or_else(|| PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../rio-v2/tests/fixtures/minio-generated"))
}
fn case_dir(case_id: &str) -> PathBuf {
@@ -136,14 +137,12 @@ async fn read_fixture_plaintext(encrypted: Vec<u8>, object_info: ObjectInfo, kms
("RUSTFS_SSE_S3_MASTER_KEY", None::<String>),
],
async move {
let resolver = SseObjectEncryptionResolver;
let (mut reader, offset, length) = GetObjectReader::new_with_resolver(
let (mut reader, offset, length) = GetObjectReader::new(
Box::new(Cursor::new(encrypted)),
None,
&object_info,
&ObjectOptions::default(),
&http::HeaderMap::new(),
Some(&resolver),
)
.await
.map_err(|err| format!("construct GetObjectReader from MinIO raw fixture: {err:?}"))?;
@@ -220,12 +219,11 @@ async fn encrypted_fixture_bytes(case_dir: &Path, manifest: &ManifestRecord, fil
readers.push(reader);
}
let erasure = Erasure::try_new(
let erasure = Erasure::new(
file_info.erasure.data_blocks,
file_info.erasure.parity_blocks,
file_info.erasure.block_size,
)
.expect("fixture erasure geometry");
);
let mut writer = VecAsyncWriter::default();
let (written, err) = erasure.decode(&mut writer, readers, 0, part.size, part.size).await;
if let Some(err) = err {
-7
View File
@@ -27,14 +27,7 @@ categories = ["web-programming", "development-tools"]
[lib]
doctest = false
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
serde = { workspace = true, features = ["derive"] }
thiserror.workspace = true
+2 -4
View File
@@ -27,12 +27,10 @@ documentation = "https://docs.rs/rustfs-filemeta/latest/rustfs_filemeta/"
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-utils/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-utils/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-utils/hotpath-cpu"]
hotpath = ["dep:hotpath", "hotpath/hotpath"]
[dependencies]
hotpath.workspace = true
hotpath = { workspace = true, optional = true }
crc-fast = { workspace = true }
rmp.workspace = true
rmp-serde.workspace = true
+3 -3
View File
@@ -19,7 +19,7 @@ impl FileMeta {
!matches!(Self::check_xl2_v1(buf), Err(_e))
}
#[hotpath::measure(impl_type = "FileMeta")]
#[cfg_attr(feature = "hotpath", hotpath::measure(impl_type = "FileMeta"))]
pub fn load(buf: &[u8]) -> Result<FileMeta> {
let mut xl = FileMeta::default();
xl.unmarshal_msg(buf)?;
@@ -112,7 +112,7 @@ impl FileMeta {
Ok((bin_len, remaining))
}
#[hotpath::measure(impl_type = "FileMeta")]
#[cfg_attr(feature = "hotpath", hotpath::measure(impl_type = "FileMeta"))]
pub fn unmarshal_msg(&mut self, buf: &[u8]) -> Result<u64> {
let i = buf.len() as u64;
@@ -326,7 +326,7 @@ impl FileMeta {
}
}
#[hotpath::measure(impl_type = "FileMeta")]
#[cfg_attr(feature = "hotpath", hotpath::measure(impl_type = "FileMeta"))]
pub fn marshal_msg(&self) -> Result<Vec<u8>> {
let mut wr = Vec::new();
-41
View File
@@ -29,48 +29,7 @@ categories = ["web-programming", "development-tools", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"rustfs-common/hotpath",
"rustfs-concurrency/hotpath",
"rustfs-config/hotpath",
"rustfs-ecstore/hotpath",
"rustfs-madmin/hotpath",
"rustfs-storage-api/hotpath",
"rustfs-utils/hotpath",
"rustfs-test-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-common/hotpath-alloc",
"rustfs-concurrency/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-ecstore/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-storage-api/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
"rustfs-test-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-common/hotpath-cpu",
"rustfs-concurrency/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-ecstore/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-storage-api/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
"rustfs-test-utils/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
rustfs-config = { workspace = true }
rustfs-concurrency = { workspace = true }
rustfs-ecstore = { workspace = true }
-1
View File
@@ -159,7 +159,6 @@ impl ErasureSetHealer {
/// execute erasure set heal with resume
#[tracing::instrument(skip(self, buckets), fields(set_disk_id = %set_disk_id, bucket_count = buckets.len()))]
#[hotpath::measure]
pub async fn heal_erasure_set(&self, buckets: &[String], set_disk_id: &str) -> Result<()> {
debug!(
target: "rustfs::heal::erasure_healer",
-3
View File
@@ -584,7 +584,6 @@ impl HealTask {
}
#[tracing::instrument(skip(self), fields(task_id = %self.id, heal_type = ?self.heal_type))]
#[hotpath::measure]
pub async fn execute(&self) -> Result<()> {
// update status and timestamps atomically to avoid race conditions
let now = SystemTime::now();
@@ -760,7 +759,6 @@ impl HealTask {
// specific heal implementation method
#[tracing::instrument(skip(self), fields(bucket = %bucket, object = %object, version_id = ?version_id))]
#[hotpath::measure]
async fn heal_object(&self, bucket: &str, object: &str, version_id: Option<&str>) -> Result<()> {
debug!(
target: "rustfs::heal::task",
@@ -1406,7 +1404,6 @@ impl HealTask {
self.heal_bucket_objects(bucket, prefix).await
}
#[hotpath::measure]
async fn heal_bucket_objects(&self, bucket: &str, prefix: &str) -> Result<()> {
let mut continuation_token: Option<String> = None;
let mut scanned = 0u64;
-48
View File
@@ -28,55 +28,7 @@ documentation = "https://docs.rs/rustfs-iam/latest/rustfs_iam/"
[lints]
workspace = true
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"hotpath/reqwest-0-13",
"rustfs-config/hotpath",
"rustfs-credentials/hotpath",
"rustfs-crypto/hotpath",
"rustfs-ecstore/hotpath",
"rustfs-io-metrics/hotpath",
"rustfs-madmin/hotpath",
"rustfs-policy/hotpath",
"rustfs-storage-api/hotpath",
"rustfs-utils/hotpath",
"rustfs-test-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-credentials/hotpath-alloc",
"rustfs-crypto/hotpath-alloc",
"rustfs-ecstore/hotpath-alloc",
"rustfs-io-metrics/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-policy/hotpath-alloc",
"rustfs-storage-api/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
"rustfs-test-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-credentials/hotpath-cpu",
"rustfs-crypto/hotpath-cpu",
"rustfs-ecstore/hotpath-cpu",
"rustfs-io-metrics/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-policy/hotpath-cpu",
"rustfs-storage-api/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
"rustfs-test-utils/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
rustfs-credentials = { workspace = true }
rustfs-config = { workspace = true, features = ["server-config-model"] }
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
+27 -58
View File
@@ -1139,11 +1139,8 @@ impl OidcSys {
let mut policies = Vec::new();
let mut groups = Vec::new();
// Role-policy and claim-based authorization are separate OIDC modes. When a
// role policy is configured, group claims still provide group context but
// must not also become policy names.
let has_role_policy = !config.role_policy.trim().is_empty();
if has_role_policy {
// Add default role policy if configured
if !config.role_policy.is_empty() {
for policy in config.role_policy.split(',') {
let policy = policy.trim();
if !policy.is_empty() {
@@ -1152,20 +1149,21 @@ impl OidcSys {
}
}
// Map groups claim to policies
for group in &claims.groups {
groups.push(group.clone());
if !has_role_policy {
let policy_name = if config.claim_prefix.is_empty() {
group.clone()
} else {
format!("{}{}", config.claim_prefix, group)
};
policies.push(policy_name);
}
let policy_name = if config.claim_prefix.is_empty() {
group.clone()
} else {
format!("{}{}", config.claim_prefix, group)
};
policies.push(policy_name);
}
if !has_role_policy && config.claim_name != config.groups_claim {
for val in extract_groups_claim(&claims.raw, &config.claim_name) {
// Map primary claim (if different from groups)
if config.claim_name != config.groups_claim {
let claim_values = extract_groups_claim(&claims.raw, &config.claim_name);
for val in claim_values {
let policy_name = if config.claim_prefix.is_empty() {
val
} else {
@@ -3203,22 +3201,26 @@ mod tests {
}
#[test]
fn role_policy_does_not_map_groups_as_policies() {
let mut config = test_config("authentik");
config.role_policy = "consoleAdmin".to_string();
config.claim_name = "policy".to_string();
fn test_map_claims_to_policies_with_provider() {
let mut config = test_config("okta");
config.role_policy = "readwrite".to_string();
config.display_name = "Okta".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
groups: vec!["authentik Admins".to_string(), "users".to_string()],
raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly"]))]),
..Default::default()
sub: "user123".to_string(),
email: "user@example.com".to_string(),
username: "user".to_string(),
groups: vec!["admin".to_string(), "devs".to_string()],
raw: HashMap::new(),
};
let (policies, groups) = sys.map_claims_to_policies("authentik", &claims);
assert_eq!(groups, vec!["authentik Admins", "users"]);
assert_eq!(policies, vec!["consoleAdmin"]);
let (policies, groups) = sys.map_claims_to_policies("okta", &claims);
assert_eq!(groups, vec!["admin", "devs"]);
assert!(policies.contains(&"readwrite".to_string()));
assert!(policies.contains(&"admin".to_string()));
assert!(policies.contains(&"devs".to_string()));
}
#[test]
@@ -3243,39 +3245,6 @@ mod tests {
assert_eq!(policies.len(), 1);
}
#[test]
fn blank_role_policy_uses_claim_mapping() {
let mut config = test_config("keycloak");
config.role_policy = " ".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
groups: vec!["readonly".to_string()],
..Default::default()
};
let (policies, groups) = sys.map_claims_to_policies("keycloak", &claims);
assert_eq!(groups, vec!["readonly"]);
assert_eq!(policies, vec!["readonly"]);
}
#[test]
fn claim_mapping_keeps_groups_with_distinct_primary_claim() {
let mut config = test_config("keycloak");
config.claim_name = "policy".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
groups: vec!["developers".to_string()],
raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly"]))]),
..Default::default()
};
let (policies, groups) = sys.map_claims_to_policies("keycloak", &claims);
assert_eq!(groups, vec!["developers"]);
assert_eq!(policies, vec!["developers", "readonly"]);
}
#[test]
fn test_list_providers() {
let mut config = test_config("keycloak");
-2
View File
@@ -469,7 +469,6 @@ impl ObjectStore {
});
}
#[hotpath::measure]
async fn list_all_iamconfig_items(&self) -> Result<HashMap<String, Vec<String>>> {
let (tx, mut rx) = mpsc::channel::<StringOrErr>(100);
@@ -509,7 +508,6 @@ impl ObjectStore {
Ok(res)
}
#[hotpath::measure]
async fn load_policy_doc_concurrent(&self, names: &[String], mode: LoadMode) -> Result<Vec<PolicyDoc>> {
let mut futures = Vec::with_capacity(names.len());
+30 -233
View File
@@ -41,7 +41,7 @@ use rustfs_policy::policy::Args;
use rustfs_policy::policy::opa;
use rustfs_policy::policy::{Policy, PolicyDoc, iam_policy_claim_name_sa, policy_needs_existing_object_tag_for_args};
use serde_json::Value;
use std::collections::{HashMap, HashSet};
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::OnceLock;
use time::OffsetDateTime;
@@ -61,45 +61,10 @@ const VERIFIED_FEDERATED_POLICY_CLAIM: &str = "x-rustfs-internal-federated-polic
const FEDERATED_POLICY_BOUNDARY_CLAIM: &str = "x-rustfs-internal-federated-boundary";
pub(crate) const SITE_REPLICATOR_CLAIM: &str = "site-replicator";
#[derive(Clone)]
enum PolicyPluginState {
Disabled,
Initializing,
Ready(opa::AuthZPlugin),
Failed,
}
static POLICY_PLUGIN_CLIENT: OnceLock<Arc<RwLock<Option<rustfs_policy::policy::opa::AuthZPlugin>>>> = OnceLock::new();
static POLICY_PLUGIN_STATE: OnceLock<Arc<RwLock<PolicyPluginState>>> = OnceLock::new();
fn get_policy_plugin_state() -> Arc<RwLock<PolicyPluginState>> {
POLICY_PLUGIN_STATE
.get_or_init(|| {
let configured = opa::is_configured();
let state = Arc::new(RwLock::new(if configured {
PolicyPluginState::Initializing
} else {
PolicyPluginState::Disabled
}));
if configured {
let state = Arc::clone(&state);
tokio::spawn(async move {
let next_state = match opa::lookup_config().await {
Ok(conf) if conf.enable() => {
info!("OPA plugin enabled");
PolicyPluginState::Ready(opa::AuthZPlugin::new(conf))
}
Ok(_) => PolicyPluginState::Failed,
Err(e) => {
error!("Error loading OPA configuration err:{}", e);
PolicyPluginState::Failed
}
};
*state.write().await = next_state;
});
}
state
})
.clone()
fn get_policy_plugin_client() -> Arc<RwLock<Option<rustfs_policy::policy::opa::AuthZPlugin>>> {
POLICY_PLUGIN_CLIENT.get_or_init(|| Arc::new(RwLock::new(None))).clone()
}
pub struct IamSys<T> {
@@ -208,7 +173,19 @@ impl<T: Store> IamSys<T> {
/// # Returns
/// A new instance of IamSys
pub fn new(store: Arc<IamCache<T>>) -> Self {
get_policy_plugin_state();
tokio::spawn(async move {
match opa::lookup_config().await {
Ok(conf) => {
if conf.enable() {
Self::set_policy_plugin_client(opa::AuthZPlugin::new(conf)).await;
info!("OPA plugin enabled");
}
}
Err(e) => {
error!("Error loading OPA configuration err:{}", e);
}
};
});
Self {
store,
@@ -229,22 +206,15 @@ impl<T: Store> IamSys<T> {
}
pub async fn set_policy_plugin_client(client: rustfs_policy::policy::opa::AuthZPlugin) {
let policy_plugin_state = get_policy_plugin_state();
let mut guard = policy_plugin_state.write().await;
*guard = PolicyPluginState::Ready(client);
let policy_plugin_client = get_policy_plugin_client();
let mut guard = policy_plugin_client.write().await;
*guard = Some(client);
}
pub async fn get_policy_plugin_client() -> Option<rustfs_policy::policy::opa::AuthZPlugin> {
let policy_plugin_state = get_policy_plugin_state();
let guard = policy_plugin_state.read().await;
match &*guard {
PolicyPluginState::Ready(client) => Some(client.clone()),
PolicyPluginState::Disabled | PolicyPluginState::Initializing | PolicyPluginState::Failed => None,
}
}
async fn policy_plugin_state() -> PolicyPluginState {
get_policy_plugin_state().read().await.clone()
let policy_plugin_client = get_policy_plugin_client();
let guard = policy_plugin_client.read().await;
guard.clone()
}
pub async fn load_group(&self, name: &str) -> Result<()> {
@@ -1090,20 +1060,11 @@ impl<T: Store> IamSys<T> {
};
}
match Self::policy_plugin_state().await {
PolicyPluginState::Ready(_) => {
return PreparedIamAuth {
needs_existing_object_tag: false,
mode: PreparedIamMode::Opa,
};
}
PolicyPluginState::Initializing | PolicyPluginState::Failed => {
return PreparedIamAuth {
needs_existing_object_tag: false,
mode: PreparedIamMode::Deny,
};
}
PolicyPluginState::Disabled => {}
if Self::get_policy_plugin_client().await.is_some() {
return PreparedIamAuth {
needs_existing_object_tag: false,
mode: PreparedIamMode::Opa,
};
}
let Ok((is_svc, parent_user)) = self.is_service_account(args.account).await else {
@@ -1202,22 +1163,7 @@ impl<T: Store> IamSys<T> {
};
}
let (resolved_policies, combined_policy) = self.store.merge_policies(&policies.join(",")).await;
if args.deny_only {
let resolved_policy_names: HashSet<&str> = resolved_policies
.split(',')
.filter(|policy_name| !policy_name.trim().is_empty())
.collect();
if policies
.iter()
.any(|policy_name| !resolved_policy_names.contains(policy_name.as_str()))
{
return PreparedIamAuth {
needs_existing_object_tag: false,
mode: PreparedIamMode::Deny,
};
}
}
let combined_policy = self.get_combined_policy(&policies).await;
PreparedIamAuth {
needs_existing_object_tag: policy_needs_existing_object_tag_for_args(&combined_policy, args).await,
mode: PreparedIamMode::Regular { combined_policy },
@@ -1757,7 +1703,7 @@ mod tests {
fn deprecated_list_polices_api_is_available() {
let _ = IamSys::<StsTestMockStore>::list_polices;
}
use rustfs_policy::policy::action::{Action, AdminAction, S3Action, StsAction};
use rustfs_policy::policy::action::{Action, AdminAction, S3Action};
use rustfs_policy::policy::policy_uses_existing_object_tag_conditions;
use serde_json::Value;
use std::{
@@ -3363,155 +3309,6 @@ mod tests {
assert!(!iam_sys.eval_prepared(&prepared, &args).await);
}
#[tokio::test]
async fn regular_deny_only_rejects_unresolved_mapped_policy() {
let iam_sys = test_iam_sys().await;
let user = "regular-unresolved-policy";
let identity = UserIdentity::from(Credentials {
access_key: user.to_string(),
secret_key: "longenoughsecret".to_string(),
status: ACCOUNT_ON.to_string(),
..Default::default()
});
iam_sys.store.cache.with_write_lock(|cache| {
let now = OffsetDateTime::now_utc();
cache.add_or_update_user(user, &identity, now);
cache.add_or_update_user_policy(user, &MappedPolicy::new("readwrite,missing-policy"), now);
});
let groups = None;
let claims = HashMap::new();
let conditions = HashMap::new();
let args = Args {
account: user,
groups: &groups,
action: Action::StsAction(StsAction::AssumeRoleAction),
bucket: "",
conditions: &conditions,
is_owner: false,
object: "",
claims: &claims,
deny_only: true,
};
let prepared = iam_sys.prepare_regular_auth(&args).await;
assert!(matches!(prepared.mode, PreparedIamMode::Deny));
assert!(
!iam_sys.eval_prepared(&prepared, &args).await,
"deny-only evaluation must fail closed when a mapped policy document is unresolved"
);
}
#[tokio::test]
async fn regular_full_evaluation_keeps_resolved_part_of_partial_mapping() {
let iam_sys = test_iam_sys().await;
let user = "regular-partial-policy";
let identity = UserIdentity::from(Credentials {
access_key: user.to_string(),
secret_key: "longenoughsecret".to_string(),
status: ACCOUNT_ON.to_string(),
..Default::default()
});
iam_sys.store.cache.with_write_lock(|cache| {
let now = OffsetDateTime::now_utc();
cache.add_or_update_user(user, &identity, now);
cache.add_or_update_user_policy(user, &MappedPolicy::new("readwrite,missing-policy"), now);
});
let groups = None;
let claims = HashMap::new();
let conditions = HashMap::new();
let args = Args {
account: user,
groups: &groups,
action: Action::S3Action(S3Action::GetObjectAction),
bucket: "bucket",
conditions: &conditions,
is_owner: false,
object: "object",
claims: &claims,
deny_only: false,
};
let prepared = iam_sys.prepare_regular_auth(&args).await;
assert!(matches!(prepared.mode, PreparedIamMode::Regular { .. }));
assert!(
iam_sys.eval_prepared(&prepared, &args).await,
"full evaluation must preserve the historical partial-resolution behavior"
);
}
#[tokio::test]
async fn regular_deny_only_honors_group_derived_explicit_deny() {
let iam_sys = test_iam_sys().await;
let deny_policy =
Policy::parse_config(br#"{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Action":["sts:AssumeRole"]}]}"#)
.expect("group deny policy should parse");
let now = OffsetDateTime::now_utc();
iam_sys
.store
.cache
.add_or_update_policy_doc("deny-assume-role", &PolicyDoc::new(deny_policy), now);
iam_sys
.store
.cache
.add_or_update_group_policy("testgroup", &MappedPolicy::new("deny-assume-role"), now);
let groups = Some(vec!["testgroup".to_string()]);
let claims = HashMap::new();
let conditions = HashMap::new();
let args = Args {
account: "sts-fallback-test-parent",
groups: &groups,
action: Action::StsAction(StsAction::AssumeRoleAction),
bucket: "",
conditions: &conditions,
is_owner: false,
object: "",
claims: &claims,
deny_only: true,
};
let prepared = iam_sys.prepare_regular_auth(&args).await;
assert!(matches!(prepared.mode, PreparedIamMode::Regular { .. }));
assert!(
!iam_sys.eval_prepared(&prepared, &args).await,
"group-derived explicit Deny must remain authoritative"
);
}
#[tokio::test]
async fn regular_deny_only_rejects_unresolved_group_mapping() {
let iam_sys = test_iam_sys().await;
iam_sys.store.cache.add_or_update_group_policy(
"testgroup",
&MappedPolicy::new("readwrite,missing-policy"),
OffsetDateTime::now_utc(),
);
let groups = Some(vec!["testgroup".to_string()]);
let claims = HashMap::new();
let conditions = HashMap::new();
let args = Args {
account: "sts-fallback-test-parent",
groups: &groups,
action: Action::StsAction(StsAction::AssumeRoleAction),
bucket: "",
conditions: &conditions,
is_owner: false,
object: "",
claims: &claims,
deny_only: true,
};
let prepared = iam_sys.prepare_regular_auth(&args).await;
assert!(matches!(prepared.mode, PreparedIamMode::Deny));
assert!(
!iam_sys.eval_prepared(&prepared, &args).await,
"deny-only evaluation must fail closed for an unresolved group-derived mapping"
);
}
#[tokio::test]
async fn test_sts_claim_policy_ignores_unsafe_and_missing_policy_names() {
let store = StsTestMockStore::new(true);
-7
View File
@@ -27,14 +27,7 @@ categories = ["development-tools", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-io-metrics/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-io-metrics/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-io-metrics/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["io-util", "fs", "sync", "rt-multi-thread"] }
-27
View File
@@ -28,36 +28,9 @@ categories = ["development-tools", "filesystem"]
name = "metrics_pipeline"
harness = false
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"rustfs-common/hotpath",
"rustfs-s3-ops/hotpath",
"rustfs-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-common/hotpath-alloc",
"rustfs-s3-ops/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-common/hotpath-cpu",
"rustfs-s3-ops/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
metrics = { workspace = true }
rustfs-common = { workspace = true }
rustfs-s3-ops = { workspace = true }
rustfs-utils = { workspace = true, features = ["ip"] }
num_cpus = { workspace = true }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread"] }

Some files were not shown because too many files have changed in this diff Show More