mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-06 03:59:14 +00:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 overtrue
|
081de25eeb |
@@ -6,7 +6,7 @@ description: "Run the end-to-end RustFS console gate, version bump, preview vali
|
||||
|
||||
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
|
||||
|
||||
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. That Release is temporary: `build.yml` deletes it automatically once the final tag's Release is published, so the Releases page ends up carrying deliverables only while the `-preview.N` tags stay behind as the traceability record. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
|
||||
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
|
||||
|
||||
Pipeline shape:
|
||||
|
||||
@@ -19,7 +19,6 @@ check console main against its latest Release
|
||||
-> validate with latest rc client
|
||||
-> report preview acceptance results -> STOP for explicit human confirmation
|
||||
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
|
||||
-> CI deletes the <target>-preview.N Releases (tags kept)
|
||||
```
|
||||
|
||||
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
|
||||
@@ -52,16 +51,14 @@ Rules:
|
||||
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
|
||||
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
|
||||
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
|
||||
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
|
||||
|
||||
## Hard rules
|
||||
|
||||
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
|
||||
- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
|
||||
- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
|
||||
- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
|
||||
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
|
||||
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
|
||||
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
|
||||
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
|
||||
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
|
||||
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
|
||||
- Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
|
||||
@@ -233,7 +230,6 @@ git push origin "<target>"
|
||||
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
|
||||
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
|
||||
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
|
||||
- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
|
||||
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
|
||||
|
||||
## Output contract
|
||||
@@ -243,5 +239,5 @@ Always report:
|
||||
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
|
||||
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
|
||||
- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
|
||||
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
|
||||
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
|
||||
- Any deviation from this pipeline and why the user approved it.
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
sha256-darwin=d6aa36cfaae2c4d8590482c7e47138c5965b335b34a75f50d11ffc3366e9021e
|
||||
sha256-linux=e3eb4ab7fc72224abf58c546ac0706d6605d3bd26bac7d8ce338829fd3daecc2
|
||||
sha256-linux=c8315465f50c194faee36141cdbb1e15e59271e524d948564a69e2d5eb408f2a
|
||||
|
||||
@@ -1 +1 @@
|
||||
sha256=8d5517f5f2fc32d561782dfccd51b7f746f5e25b2835e37e100c883f7f18777d
|
||||
sha256=655a3f3c1d042e694339d15caba7580518320322d1bac0f09450b37e6c09e2e7
|
||||
|
||||
+10
-16
@@ -100,16 +100,6 @@ test-group = 'embedded-test-ports'
|
||||
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
|
||||
test-group = 'ecstore-serial-flaky'
|
||||
|
||||
# Serialize the transition matrix tests. They build a 4-disk hermetic erasure
|
||||
# set, populate the get_object_metadata_cache, and assert generation lifecycle
|
||||
# semantics. serial_test's #[serial] has no effect across nextest's process
|
||||
# boundary, so concurrent execution races the shared metadata-cache generation
|
||||
# counter and causes spurious "metadata read should publish the generation"
|
||||
# panics. Preventive serialization, no retries.
|
||||
[[profile.default.overrides]]
|
||||
filter = 'package(rustfs-ecstore) & test(set_disk::transition_matrix_tests::)'
|
||||
test-group = 'ecstore-serial-flaky'
|
||||
|
||||
# The durable ILM decommission regressions build isolated multi-pool stores and
|
||||
# deliberately take source or target disks offline while checking fencing.
|
||||
[[profile.default.overrides]]
|
||||
@@ -215,6 +205,16 @@ retries = 2
|
||||
filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
|
||||
test-group = 'ecstore-serial-flaky'
|
||||
|
||||
# QUARANTINE: OPEN rustfs#6711 — the multipart fencing test's epoch helper
|
||||
# reads xl.meta back from EVERY disk, but a multipart commit only guarantees
|
||||
# quorum-many disks have persisted; a lagging disk under CI load panics the
|
||||
# read-back with "file not found" (observed on the rio-v2 leg of a
|
||||
# nextest-config-only PR; same all-disk-materialization assumption as the
|
||||
# relocated-pool fixture fixed by #6707).
|
||||
[[profile.ci.overrides]]
|
||||
filter = 'package(rustfs-ecstore) & test(object_transaction_fencing_persists_epoch_on_multipart_commit)'
|
||||
retries = 2
|
||||
|
||||
# Serialize the 4-disk reliability / degraded-read e2e tests under the ci
|
||||
# profile too (see the e2e-reliability test-group note near the top). Not a
|
||||
# quarantine: no retries, just single-threaded so several 4-disk servers never
|
||||
@@ -242,12 +242,6 @@ test-group = 'embedded-test-ports'
|
||||
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
|
||||
test-group = 'ecstore-serial-flaky'
|
||||
|
||||
# Serialize the transition matrix tests under the ci profile too (see the
|
||||
# matching default-profile override near the top). No retries.
|
||||
[[profile.ci.overrides]]
|
||||
filter = 'package(rustfs-ecstore) & test(set_disk::transition_matrix_tests::)'
|
||||
test-group = 'ecstore-serial-flaky'
|
||||
|
||||
[[profile.ci.overrides]]
|
||||
filter = 'package(rustfs-ecstore) & (test(decommission_migrates_and_verifies_registered_durable_ilm_records) | test(decommission_durable_ilm_target_read_error_is_not_masked_by_peer_success) | test(decommission_durable_ilm_terminal_receipt_recovers_failed_source_cleanup) | test(decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page) | test(decommission_durable_ilm_recovery_keeps_multiple_active_sources))'
|
||||
test-group = 'ecstore-serial-flaky'
|
||||
|
||||
@@ -5,5 +5,3 @@ self-hosted-runner:
|
||||
- sm-standard-2
|
||||
- sm-standard-4
|
||||
- dind-sm-standard-2
|
||||
- smoke-testing
|
||||
- pf-testing
|
||||
|
||||
@@ -7,11 +7,6 @@
|
||||
{ "workflow": ".github/workflows/e2e-s3tests.yml", "max_age_hours": 192 },
|
||||
{ "workflow": ".github/workflows/fuzz.yml", "max_age_hours": 36 },
|
||||
{ "workflow": ".github/workflows/mint.yml", "max_age_hours": 192 },
|
||||
{
|
||||
"workflow": ".github/workflows/minio-interop.yml",
|
||||
"max_age_hours": 36,
|
||||
"never_ran_grace_until": "2026-09-08T00:00:00Z"
|
||||
},
|
||||
{ "workflow": ".github/workflows/nightly-gnu.yml", "max_age_hours": 36 },
|
||||
{ "workflow": ".github/workflows/performance-ab.yml", "max_age_hours": 36 },
|
||||
{
|
||||
|
||||
@@ -1033,55 +1033,6 @@ jobs:
|
||||
echo "🎉 Released $TAG successfully!"
|
||||
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
|
||||
|
||||
# Remove the internal preview releases once the deliverable release is live.
|
||||
# Only the Releases are deleted; the -preview.N tags stay so the validated
|
||||
# commit remains traceable.
|
||||
cleanup-preview-releases:
|
||||
name: Cleanup Preview Releases
|
||||
needs: [ build-check, publish-release ]
|
||||
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- name: Delete preview releases for this target
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
TAG="${{ needs.build-check.outputs.version }}"
|
||||
RELEASES_JSON="${RUNNER_TEMP}/releases.json"
|
||||
|
||||
# Fetch before filtering: a failed listing must abort here instead of
|
||||
# looking like "nothing to clean up".
|
||||
gh api --paginate "repos/${GITHUB_REPOSITORY}/releases?per_page=100" > "$RELEASES_JSON"
|
||||
|
||||
# Match only <target>-preview.<digits>. String operations, not a
|
||||
# regex over the tag, so dots in the version cannot widen the match.
|
||||
DELETED=0
|
||||
while IFS= read -r preview_tag; do
|
||||
[[ -n "$preview_tag" ]] || continue
|
||||
echo "🧹 Deleting preview release $preview_tag (tag kept)"
|
||||
gh release delete "$preview_tag" --yes
|
||||
DELETED=$((DELETED + 1))
|
||||
done < <(
|
||||
jq -r --arg tag "$TAG" '
|
||||
.[]
|
||||
| select(.tag_name | startswith($tag + "-preview."))
|
||||
| select(.tag_name | ltrimstr($tag + "-preview.") | test("^[0-9]+$"))
|
||||
| .tag_name
|
||||
' "$RELEASES_JSON"
|
||||
)
|
||||
|
||||
if [[ "$DELETED" -eq 0 ]]; then
|
||||
echo "ℹ️ No preview releases to clean up for $TAG"
|
||||
else
|
||||
echo "✅ Removed $DELETED preview release(s) for $TAG"
|
||||
fi
|
||||
|
||||
alert-on-failure:
|
||||
name: Alert on scheduled failure
|
||||
needs: [build-check, prepare-platform-matrix, build-rustfs, build-summary]
|
||||
|
||||
@@ -20,27 +20,27 @@
|
||||
# each run with Docker and then runs the `#[ignore]` reader tests in
|
||||
# rustfs/src/storage/minio_generated_read_test.rs.
|
||||
#
|
||||
# Scope: MinIO-to-RustFS SSE read interop is implemented behind the `rio-v2`
|
||||
# feature for MinIO's builtin static-KMS deployments — SSE-S3 and SSE-KMS
|
||||
# (single- and multipart) since rustfs/rustfs#6191, SSE-C detection since the
|
||||
# rustfs/backlog#1638 D2 close-out. This job is the standing evidence: it
|
||||
# regenerates real MinIO backend trees and proves byte-identical plaintext
|
||||
# reconstruction. KES/MinKMS-backed MinIO objects remain unreadable by design
|
||||
# (their envelopes are sealed by the KES service, not by a key RustFS can
|
||||
# hold), and default RustFS builds do not include the read path — it is a
|
||||
# special-purpose migration capability, not a default-build feature.
|
||||
# Scope: end-to-end MinIO-to-RustFS SSE interop is NOT implemented yet. Both
|
||||
# envelope parsers reject MinIO's own wrapped-DEK shape — see
|
||||
# `is_data_key_envelope` in crates/kms/src/encryption/dek.rs and the
|
||||
# `deny_unknown_fields` `LocalSseDekEnvelope` in rustfs/src/storage/sse.rs — and
|
||||
# closing that gap is tracked in rustfs/backlog#1638. Treat this job as the
|
||||
# harness for #1638, not as standing evidence that a MinIO migration reads back.
|
||||
#
|
||||
# Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
|
||||
# unlike the self-hosted fleet, whose pods drift in Docker/pip availability
|
||||
# (see the infra note in e2e-s3tests.yml). Nightly + manual only.
|
||||
# DISABLED. This workflow is switched off in the repository's Actions settings
|
||||
# (state: disabled_manually) and does not run on any trigger, including its cron
|
||||
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
|
||||
# reading this file, which has already misled at least one audit — hence this
|
||||
# banner. Re-enabling is a UI action; anyone doing so should first check that the
|
||||
# workflow still matches the current CI layout. See rustfs/backlog#1603.
|
||||
#
|
||||
# Enablement: this workflow was long disabled in the repository's Actions
|
||||
# settings (state: disabled_manually — a state that lives in GitHub's UI and is
|
||||
# invisible in this file). The change that updated this banner also re-added
|
||||
# the .github/scheduled-validations.json entry; both only make sense together
|
||||
# with re-enabling the workflow in the Actions settings. If it is ever disabled
|
||||
# again, remove the scheduled-validations entry in the same change — a disabled
|
||||
# workflow can never satisfy the freshness check. See rustfs/backlog#1603.
|
||||
# While disabled, this workflow is deliberately absent from
|
||||
# .github/scheduled-validations.json — a disabled workflow can never satisfy the
|
||||
# freshness check. Whoever re-enables it must re-add the entry in the same
|
||||
# change so the freshness gate covers it again.
|
||||
#
|
||||
name: minio-interop
|
||||
|
||||
|
||||
@@ -27,7 +27,6 @@ on:
|
||||
paths:
|
||||
- 'flake.nix'
|
||||
- 'flake.lock'
|
||||
- 'nix/**'
|
||||
- 'Cargo.toml'
|
||||
- 'Cargo.lock'
|
||||
- '.github/workflows/nix.yml'
|
||||
@@ -37,7 +36,6 @@ on:
|
||||
paths:
|
||||
- 'flake.nix'
|
||||
- 'flake.lock'
|
||||
- 'nix/**'
|
||||
- 'Cargo.toml'
|
||||
- 'Cargo.lock'
|
||||
- '.github/workflows/nix.yml'
|
||||
|
||||
@@ -224,7 +224,7 @@ jobs:
|
||||
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
|
||||
if [[ -z "$ZIP_FILE" ]]; then
|
||||
echo "❌ No binary artifact found"
|
||||
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
|
||||
ls -la ./binary-artifact/ || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -239,7 +239,7 @@ jobs:
|
||||
fi
|
||||
|
||||
chmod +x ./bin/rustfs
|
||||
stat --printf='%n %s bytes\n' ./bin/rustfs
|
||||
ls -lh ./bin/rustfs
|
||||
echo "✅ Binary extracted"
|
||||
|
||||
- name: Build DEB package
|
||||
@@ -336,7 +336,7 @@ jobs:
|
||||
fakeroot dpkg-deb --build "${PKG_DIR}"
|
||||
|
||||
DEB_FILE="${PKG_DIR}.deb"
|
||||
stat --printf='%n %s bytes\n' "$DEB_FILE"
|
||||
ls -lh "$DEB_FILE"
|
||||
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ DEB built: $DEB_FILE"
|
||||
|
||||
@@ -410,14 +410,13 @@ jobs:
|
||||
LICENSE=/usr/share/doc/rustfs/LICENSE \
|
||||
README.md=/usr/share/doc/rustfs/README.md
|
||||
|
||||
RPM_FILE=$(find . -maxdepth 1 -type f -name 'rustfs-*.rpm' -print | head -1)
|
||||
RPM_FILE="${RPM_FILE#./}"
|
||||
RPM_FILE=$(ls -1 rustfs-*.rpm 2>/dev/null | head -1)
|
||||
if [[ -z "$RPM_FILE" ]]; then
|
||||
echo "❌ RPM build failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
stat --printf='%n %s bytes\n' "$RPM_FILE"
|
||||
ls -lh "$RPM_FILE"
|
||||
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ RPM built: $RPM_FILE"
|
||||
|
||||
@@ -553,13 +552,11 @@ jobs:
|
||||
- name: Print summary
|
||||
shell: bash
|
||||
run: |
|
||||
{
|
||||
echo "## 📦 Package Summary"
|
||||
echo ""
|
||||
echo "| Item | Value |"
|
||||
echo "|------|-------|"
|
||||
echo "| Version | \`${{ needs.resolve.outputs.version }}\` |"
|
||||
echo "| Build Type | ${{ needs.resolve.outputs.build_type }} |"
|
||||
echo "| Build Run | #${{ needs.resolve.outputs.build_run_id }} |"
|
||||
echo "| Package Status | ${{ needs.package.result }} |"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "## 📦 Package Summary" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Item | Value |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "|------|-------|" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Version | \`${{ needs.resolve.outputs.version }}\` |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Build Type | ${{ needs.resolve.outputs.build_type }} |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Build Run | #${{ needs.resolve.outputs.build_run_id }} |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Package Status | ${{ needs.package.result }} |" >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
@@ -1,128 +0,0 @@
|
||||
name: RustFS Heal Test
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
package_url:
|
||||
description: 'Direct .deb URL (nightly/R2). Defaults to the latest nightly deb.'
|
||||
required: false
|
||||
type: string
|
||||
stop_node_gb:
|
||||
description: 'Stop the outage node when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '15'
|
||||
warp_stop_gb:
|
||||
description: 'Stop warp when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '40'
|
||||
cleanup_before:
|
||||
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
cleanup_after:
|
||||
description: 'Reset the nodes after the test (DESTROYS test data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# Only one test at a time: both this and the pool-expansion workflow mutate
|
||||
# the same test environment, so they share one concurrency group.
|
||||
concurrency:
|
||||
group: rustfs-shared-functional-tests
|
||||
cancel-in-progress: false
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
env:
|
||||
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
|
||||
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
|
||||
RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
|
||||
jobs:
|
||||
heal-test:
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 480
|
||||
# Manual-only standalone run. Nightly chain already runs heal in
|
||||
# rustfs-pool-expand-test.yml to avoid duplicate heal executions.
|
||||
if: ${{ github.event_name == 'workflow_dispatch' }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
uname -a
|
||||
jq --version
|
||||
openssl version
|
||||
warp --version || true
|
||||
df -h /data | tail -1
|
||||
|
||||
- name: Reset test environment (before)
|
||||
if: ${{ inputs.cleanup_before != 'false' }}
|
||||
run: |
|
||||
chmod +x auto-testing/rustfs_heal_test.sh
|
||||
./auto-testing/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Install RustFS package & start cluster
|
||||
run: |
|
||||
ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Preflight checks
|
||||
run: |
|
||||
ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Run heal test (write -> outage -> heal -> verify)
|
||||
run: |
|
||||
./auto-testing/rustfs_heal_test.sh \
|
||||
--steps "3,4,5,6,7" -y \
|
||||
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
|
||||
--stop-node-gb "${{ inputs.stop_node_gb }}" \
|
||||
--warp-stop-gb "${{ inputs.warp_stop_gb }}" \
|
||||
--log-file /tmp/rustfs-heal-test.log
|
||||
|
||||
- name: Upload test logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-heal-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-heal-test*.log
|
||||
/tmp/rustfs-warp.*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
if: ${{ always() && inputs.cleanup_after != 'false' }}
|
||||
run: |
|
||||
./auto-testing/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS heal test failed"
|
||||
echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
|
||||
echo "See the uploaded log artifact for details."
|
||||
@@ -1,347 +0,0 @@
|
||||
name: RustFS KMS Test
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
rustfs_version:
|
||||
description: 'RustFS release tag to test (e.g. 1.0.0-rc.4-preview.1)'
|
||||
required: false
|
||||
default: '1.0.0-rc.4-preview.1'
|
||||
package_url:
|
||||
description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.'
|
||||
required: false
|
||||
type: string
|
||||
workflow_run:
|
||||
# Strict shared-environment order: run after S3 compatibility test succeeds.
|
||||
workflows: ["RustFS S3 Compatibility Test"]
|
||||
types: [completed]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: rustfs-shared-functional-tests
|
||||
cancel-in-progress: false
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
env:
|
||||
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
|
||||
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
jobs:
|
||||
kms-test:
|
||||
runs-on: smoke-testing
|
||||
continue-on-error: true
|
||||
timeout-minutes: 420
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
uname -a
|
||||
jq --version
|
||||
openssl version
|
||||
docker --version || true
|
||||
|
||||
- name: Cleanup environment (before)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
|
||||
SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
|
||||
for node in "${NODES[@]}"; do
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${SSH_USER}@${node}" '
|
||||
set -euo pipefail
|
||||
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
|
||||
${SUDO} systemctl stop rustfs 2>/dev/null || true
|
||||
if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
|
||||
${SUDO} dpkg -P rustfs
|
||||
fi
|
||||
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
|
||||
${SUDO} rm -rf /var/log/rustfs /var/lib/rustfs/kms /var/lib/rustfs/kms-backup
|
||||
'
|
||||
done
|
||||
|
||||
- name: Ensure docker (Vault container)
|
||||
run: |
|
||||
if ! command -v docker >/dev/null 2>&1; then
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y docker.io
|
||||
fi
|
||||
sudo systemctl enable --now docker
|
||||
docker info >/dev/null 2>&1 || sudo docker info >/dev/null 2>&1
|
||||
|
||||
- name: Run KMS suite
|
||||
id: test
|
||||
continue-on-error: true
|
||||
env:
|
||||
LOG_FILE: /tmp/rustfs-kms.log
|
||||
run: |
|
||||
set -euo pipefail
|
||||
chmod +x auto-testing/rustfs-kms-test.sh
|
||||
PACKAGE_URL='${{ inputs.package_url }}'
|
||||
RUSTFS_VERSION='${{ inputs.rustfs_version }}'
|
||||
ARGS=(--all-topologies --backends "local,vault-kv2" -y --log-file "${LOG_FILE}")
|
||||
if [ -n "${PACKAGE_URL}" ]; then
|
||||
ARGS+=(--package-url "${PACKAGE_URL}")
|
||||
elif [ -n "${RUSTFS_VERSION}" ]; then
|
||||
ARGS+=(--version "${RUSTFS_VERSION}")
|
||||
else
|
||||
ARGS+=(--package-url "${RUSTFS_NIGHTLY_PACKAGE_URL}")
|
||||
fi
|
||||
./auto-testing/rustfs-kms-test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Generate report
|
||||
if: always()
|
||||
env:
|
||||
LOG_FILE: /tmp/rustfs-kms.log
|
||||
REPORT_FILE: /tmp/rustfs-kms-report.md
|
||||
run: |
|
||||
set -euo pipefail
|
||||
PACKAGE_URL='${{ inputs.package_url }}'
|
||||
RUSTFS_VERSION='${{ inputs.rustfs_version }}'
|
||||
if [ -n "${PACKAGE_URL}" ]; then
|
||||
PACKAGE_SOURCE="${PACKAGE_URL}"
|
||||
elif [ -n "${RUSTFS_VERSION}" ]; then
|
||||
PACKAGE_SOURCE="version ${RUSTFS_VERSION}"
|
||||
else
|
||||
PACKAGE_SOURCE="${RUSTFS_NIGHTLY_PACKAGE_URL}"
|
||||
fi
|
||||
CASE_TABLE="/tmp/rustfs-kms-cases.md"
|
||||
python3 - "${LOG_FILE}" "${CASE_TABLE}" <<'PY'
|
||||
import re
|
||||
import sys
|
||||
|
||||
log_file, out_file = sys.argv[1], sys.argv[2]
|
||||
ansi = re.compile(r'\x1b\[[0-9;]*m')
|
||||
start_re = re.compile(r'^---\s+([A-Z]+-[0-9]+)\s+(.+?)\s+---$')
|
||||
done_re = re.compile(r'^\[(PASS|FAIL|UNSUPPORTED)\]\s+([A-Z]+-[0-9]+)\b')
|
||||
|
||||
rows = []
|
||||
index = {}
|
||||
try:
|
||||
with open(log_file, 'r', encoding='utf-8', errors='replace') as fh:
|
||||
for raw in fh:
|
||||
line = ansi.sub('', raw).strip()
|
||||
m = start_re.match(line)
|
||||
if m:
|
||||
case_id, name = m.group(1), m.group(2)
|
||||
if case_id not in index:
|
||||
index[case_id] = len(rows)
|
||||
rows.append([case_id, name, 'RUNNING'])
|
||||
continue
|
||||
m = done_re.match(line)
|
||||
if m:
|
||||
status, case_id = m.group(1), m.group(2)
|
||||
if case_id in index:
|
||||
rows[index[case_id]][2] = status
|
||||
else:
|
||||
rows.append([case_id, case_id, status])
|
||||
index[case_id] = len(rows) - 1
|
||||
except FileNotFoundError:
|
||||
rows = []
|
||||
|
||||
counts = {'PASS': 0, 'FAIL': 0, 'UNSUPPORTED': 0, 'RUNNING': 0}
|
||||
for _, _, status in rows:
|
||||
counts[status] = counts.get(status, 0) + 1
|
||||
|
||||
with open(out_file, 'w', encoding='utf-8') as out:
|
||||
out.write('## Case Summary\n\n')
|
||||
out.write(f"- Total: {len(rows)}\\n")
|
||||
out.write(f"- PASS: {counts.get('PASS', 0)}\\n")
|
||||
out.write(f"- FAIL: {counts.get('FAIL', 0)}\\n")
|
||||
out.write(f"- UNSUPPORTED: {counts.get('UNSUPPORTED', 0)}\\n")
|
||||
out.write('\\n')
|
||||
out.write('| Case | Name | Status |\\n')
|
||||
out.write('| --- | --- | --- |\\n')
|
||||
for case_id, name, status in rows:
|
||||
out.write(f'| {case_id} | {name} | {status} |\\n')
|
||||
PY
|
||||
{
|
||||
echo "# RustFS KMS test report"
|
||||
echo ""
|
||||
echo "- Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||
echo "- Trigger: ${{ github.event_name }}"
|
||||
echo "- Package: ${PACKAGE_SOURCE}"
|
||||
echo "- Test Step Outcome: ${{ steps.test.outcome }}"
|
||||
echo ""
|
||||
cat "${CASE_TABLE}" || true
|
||||
echo ""
|
||||
echo "## Log tail"
|
||||
echo '```text'
|
||||
tail -n 200 "${LOG_FILE}" || true
|
||||
echo '```'
|
||||
} | tee "${REPORT_FILE}"
|
||||
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
|
||||
|
||||
- name: Upload functional report to dashboard
|
||||
if: always()
|
||||
continue-on-error: true
|
||||
env:
|
||||
GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }}
|
||||
REPORT_FILE: /tmp/rustfs-kms-report.md
|
||||
SUITE: kms
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${GH_TOKEN:-}" ]; then
|
||||
echo "PF_TESTING_GH_TOKEN is not configured; skipping dashboard upload"
|
||||
exit 0
|
||||
fi
|
||||
DATE="$(date -u +%Y-%m-%d)"
|
||||
REPORT_PATH="functional-reports/${SUITE}/${DATE}.md"
|
||||
CONTENT="$(python3 -c 'import base64,sys;print(base64.b64encode(open(sys.argv[1],"rb").read()).decode())' "${REPORT_FILE}")"
|
||||
SHA="$(gh api "repos/rustfs/dashboard/contents/${REPORT_PATH}" -q '.sha' 2>/dev/null || true)"
|
||||
if [ -n "${SHA}" ]; then
|
||||
jq -n --arg msg "report(${SUITE}): ${DATE}" --arg content "${CONTENT}" --arg sha "${SHA}" \
|
||||
'{message:$msg, content:$content, sha:$sha}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
|
||||
else
|
||||
jq -n --arg msg "report(${SUITE}): ${DATE}" --arg content "${CONTENT}" \
|
||||
'{message:$msg, content:$content}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
|
||||
fi
|
||||
|
||||
cat > /tmp/rustfs-functional-index.html <<'EOF'
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>RustFS Functional Test Reports</title>
|
||||
<style>
|
||||
:root { --bg:#f4f6fb; --card:#fff; --text:#1f2937; --muted:#6b7280; --line:#e5e7eb; --accent:#0f766e; }
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; font-family: ui-sans-serif, -apple-system, Segoe UI, Helvetica, Arial, sans-serif; background: var(--bg); color: var(--text); }
|
||||
.wrap { max-width: 980px; margin: 32px auto; padding: 0 16px; }
|
||||
.card { background: var(--card); border: 1px solid var(--line); border-radius: 14px; padding: 20px; }
|
||||
h1 { margin: 0 0 8px; font-size: 26px; }
|
||||
p { margin: 0 0 14px; color: var(--muted); }
|
||||
.tabs { display: flex; gap: 10px; margin: 14px 0 18px; flex-wrap: wrap; }
|
||||
button { border: 1px solid var(--line); background: #fff; color: var(--text); border-radius: 10px; padding: 8px 14px; cursor: pointer; }
|
||||
button.active { background: var(--accent); color: #fff; border-color: var(--accent); }
|
||||
ul { list-style: none; margin: 0; padding: 0; }
|
||||
li { padding: 10px 0; border-bottom: 1px solid var(--line); }
|
||||
a { color: var(--accent); text-decoration: none; }
|
||||
a:hover { text-decoration: underline; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
<div class="card">
|
||||
<h1>RustFS Functional Test Reports</h1>
|
||||
<p>S3, KMS, Tier report tabs. Each tab lists reports by date.</p>
|
||||
<div class="tabs" id="tabs"></div>
|
||||
<ul id="list"></ul>
|
||||
</div>
|
||||
</div>
|
||||
<script>
|
||||
const suites = [
|
||||
{ key: 's3', label: 'S3 Compatibility' },
|
||||
{ key: 'kms', label: 'KMS' },
|
||||
{ key: 'tier', label: 'Tier' },
|
||||
];
|
||||
const tabs = document.getElementById('tabs');
|
||||
const list = document.getElementById('list');
|
||||
|
||||
async function loadSuite(suite) {
|
||||
list.innerHTML = '<li>Loading...</li>';
|
||||
const api = `https://api.github.com/repos/rustfs/dashboard/contents/functional-reports/${suite}`;
|
||||
try {
|
||||
const res = await fetch(api);
|
||||
if (!res.ok) {
|
||||
list.innerHTML = '<li>No reports yet.</li>';
|
||||
return;
|
||||
}
|
||||
const data = await res.json();
|
||||
const files = data.filter(f => f.type === 'file' && f.name.endsWith('.md')).sort((a,b) => b.name.localeCompare(a.name));
|
||||
if (!files.length) {
|
||||
list.innerHTML = '<li>No reports yet.</li>';
|
||||
return;
|
||||
}
|
||||
list.innerHTML = files.map(f => `<li><a href="${f.html_url}" target="_blank" rel="noreferrer">${f.name.replace('.md','')}</a></li>`).join('');
|
||||
} catch (_e) {
|
||||
list.innerHTML = '<li>Failed to load reports.</li>';
|
||||
}
|
||||
}
|
||||
|
||||
function setActive(key) {
|
||||
for (const btn of tabs.querySelectorAll('button')) {
|
||||
btn.classList.toggle('active', btn.dataset.key === key);
|
||||
}
|
||||
loadSuite(key);
|
||||
}
|
||||
|
||||
for (const suite of suites) {
|
||||
const btn = document.createElement('button');
|
||||
btn.textContent = suite.label;
|
||||
btn.dataset.key = suite.key;
|
||||
btn.addEventListener('click', () => setActive(suite.key));
|
||||
tabs.appendChild(btn);
|
||||
}
|
||||
setActive('s3');
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
EOF
|
||||
|
||||
INDEX_PATH="functional/index.html"
|
||||
INDEX_CONTENT="$(python3 -c 'import base64;print(base64.b64encode(open("/tmp/rustfs-functional-index.html","rb").read()).decode())')"
|
||||
INDEX_SHA="$(gh api "repos/rustfs/dashboard/contents/${INDEX_PATH}" -q '.sha' 2>/dev/null || true)"
|
||||
if [ -n "${INDEX_SHA}" ]; then
|
||||
jq -n --arg msg "functional ui update" --arg content "${INDEX_CONTENT}" --arg sha "${INDEX_SHA}" \
|
||||
'{message:$msg, content:$content, sha:$sha}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${INDEX_PATH}" --input - >/dev/null
|
||||
else
|
||||
jq -n --arg msg "functional ui init" --arg content "${INDEX_CONTENT}" \
|
||||
'{message:$msg, content:$content}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${INDEX_PATH}" --input - >/dev/null
|
||||
fi
|
||||
|
||||
- name: Upload report and logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-kms-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-kms.log
|
||||
/tmp/rustfs-kms-report.md
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Cleanup environment (after)
|
||||
if: always()
|
||||
run: |
|
||||
set -euo pipefail
|
||||
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
|
||||
SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
|
||||
for node in "${NODES[@]}"; do
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${SSH_USER}@${node}" '
|
||||
set -euo pipefail
|
||||
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
|
||||
${SUDO} systemctl stop rustfs 2>/dev/null || true
|
||||
if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
|
||||
${SUDO} dpkg -P rustfs
|
||||
fi
|
||||
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
|
||||
${SUDO} rm -rf /var/log/rustfs /var/lib/rustfs/kms /var/lib/rustfs/kms-backup
|
||||
'
|
||||
done
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS KMS suite failed"
|
||||
echo "See the uploaded report and log artifacts for details."
|
||||
@@ -1,236 +0,0 @@
|
||||
name: RustFS Performance Test
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
package_url:
|
||||
description: 'Direct .deb URL (nightly/R2). Defaults to the latest nightly deb.'
|
||||
required: false
|
||||
type: string
|
||||
test_method:
|
||||
description: 'Benchmark method(s) to run (manual runs only; "all" = GET+PUT+MIXED)'
|
||||
type: choice
|
||||
options:
|
||||
- all
|
||||
- get
|
||||
- put
|
||||
- mixed
|
||||
default: 'all'
|
||||
object_size:
|
||||
description: 'Object size(s) to test (manual runs only; "all" = all 10 sizes)'
|
||||
type: choice
|
||||
options:
|
||||
- all
|
||||
- 1KiB
|
||||
- 4KiB
|
||||
- 16KiB
|
||||
- 128KiB
|
||||
- 1MiB
|
||||
- 4MiB
|
||||
- 8MiB
|
||||
- 16MiB
|
||||
- 32MiB
|
||||
- 64MiB
|
||||
default: 'all'
|
||||
warp_duration:
|
||||
description: 'warp duration per round (e.g. 5m, 30s)'
|
||||
required: false
|
||||
default: '5m'
|
||||
warp_concurrency:
|
||||
description: 'warp concurrency'
|
||||
required: false
|
||||
default: '64'
|
||||
cleanup_before:
|
||||
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
cleanup_after:
|
||||
description: 'Reset the nodes after the test (DESTROYS test data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
workflow_run:
|
||||
# Run after the nightly build completes; the nightly deb is what the test installs.
|
||||
workflows: ["Nightly GNU Build"]
|
||||
types: [completed]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# Dedicated pf-testing runner/environment: own concurrency group so perf runs
|
||||
# never block (or are blocked by) the pool-expansion / heal tests.
|
||||
concurrency:
|
||||
group: rustfs-performance-test
|
||||
cancel-in-progress: false
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
env:
|
||||
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
|
||||
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
|
||||
# Performance test uses its own node list (4 nodes); the shared
|
||||
# RUSTFS_NODES secret is used by the 3-node pool-expansion / heal tests.
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_PERF_NODES || vars.RUSTFS_PERF_NODES || 'vm000 vm001 vm002 vm003' }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
# Package used by the nightly run (workflow_dispatch inputs are empty for
|
||||
# workflow_run events), i.e. the latest nightly deb published by nightly-gnu.yml.
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
# Fixed benchmark result directory so later steps can read summary.md
|
||||
RUSTFS_RESULT_DIR: /tmp/rustfs-perf-results
|
||||
# Cross-repo token for uploading reports to rustfs/dashboard (set in repo settings)
|
||||
PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
jobs:
|
||||
performance-test:
|
||||
runs-on: pf-testing
|
||||
timeout-minutes: 900
|
||||
# Run on manual dispatch, or when the nightly build completed successfully.
|
||||
# Skipped when nightly failed.
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
uname -a
|
||||
jq --version
|
||||
warp --version || true
|
||||
df -h /data | tail -1
|
||||
|
||||
- name: Reset test environment (before)
|
||||
if: ${{ inputs.cleanup_before != 'false' }}
|
||||
run: |
|
||||
chmod +x auto-testing/rustfs_performance_test.sh
|
||||
./auto-testing/rustfs_performance_test.sh --step 1 -y
|
||||
|
||||
- name: Install RustFS package & start cluster (4x4)
|
||||
run: |
|
||||
ARGS=(--steps "2,3,4" -y)
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_performance_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Preflight checks
|
||||
run: |
|
||||
ARGS=(--preflight)
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_performance_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Run benchmark (GET/PUT/MIXED)
|
||||
id: benchmark
|
||||
run: |
|
||||
# Empty on automatic (workflow_run) runs -> full 30 rounds.
|
||||
# Manual dispatch can restrict method(s)/size(s).
|
||||
export WARP_METHODS="${{ inputs.test_method }}"
|
||||
export WARP_SIZES="${{ inputs.object_size }}"
|
||||
./auto-testing/rustfs_performance_test.sh \
|
||||
--step 5 -y \
|
||||
--warp-duration "${{ inputs.warp_duration || '5m' }}" \
|
||||
--warp-concurrency "${{ inputs.warp_concurrency || '64' }}" \
|
||||
--log-file /tmp/rustfs-perf-test.log
|
||||
|
||||
- name: Analyze results
|
||||
if: ${{ steps.benchmark.conclusion == 'success' }}
|
||||
run: |
|
||||
./auto-testing/rustfs_performance_test.sh --step 6 -y
|
||||
|
||||
- name: Collect RustFS version info
|
||||
if: ${{ steps.benchmark.conclusion == 'success' }}
|
||||
env:
|
||||
VERSION_FILE: /tmp/rustfs-version.txt
|
||||
run: |
|
||||
set -euo pipefail
|
||||
read -r -a NODES <<< "${RUSTFS_NODES}"
|
||||
[ "${#NODES[@]}" -gt 0 ] || { echo "RUSTFS_NODES is empty"; exit 1; }
|
||||
NODE="${NODES[0]}"
|
||||
SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
|
||||
{
|
||||
echo "Node: ${NODE}"
|
||||
echo "Command: rustfs --version"
|
||||
echo ""
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new \
|
||||
"${SSH_USER}@${NODE}" 'rustfs --version'
|
||||
} > "${VERSION_FILE}"
|
||||
|
||||
- name: Upload report to dashboard (reports/YYYY-MM-DD.md)
|
||||
if: ${{ steps.benchmark.conclusion == 'success' }}
|
||||
env:
|
||||
GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }}
|
||||
RESULT_DIR: ${{ env.RUSTFS_RESULT_DIR }}
|
||||
VERSION_FILE: /tmp/rustfs-version.txt
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${GH_TOKEN:-}" ]; then
|
||||
echo "PF_TESTING_GH_TOKEN is not configured; skipping report upload"
|
||||
exit 0
|
||||
fi
|
||||
SUMMARY="${RESULT_DIR}/summary.md"
|
||||
[ -f "${SUMMARY}" ] || { echo "summary.md not found at ${SUMMARY}"; exit 1; }
|
||||
DATE="$(date -u +%Y-%m-%d)"
|
||||
REPORT_PATH="reports/${DATE}.md"
|
||||
{
|
||||
echo "# RustFS nightly build performance testing report"
|
||||
echo ""
|
||||
echo "- **Date**: ${DATE}"
|
||||
echo "- **Run**: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||
echo "- **Trigger**: ${{ github.event_name }}"
|
||||
echo "- **Package**: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
|
||||
echo ""
|
||||
cat "${SUMMARY}"
|
||||
echo ""
|
||||
echo "## RustFS version"
|
||||
echo '```text'
|
||||
cat "${VERSION_FILE}"
|
||||
echo '```'
|
||||
} > /tmp/rustfs-perf-report.md
|
||||
CONTENT="$(python3 -c 'import base64; print(base64.b64encode(open("/tmp/rustfs-perf-report.md","rb").read()).decode())')"
|
||||
SHA="$(gh api "repos/rustfs/dashboard/contents/${REPORT_PATH}" -q '.sha' 2>/dev/null || true)"
|
||||
if [ -n "${SHA}" ]; then
|
||||
jq -n --arg msg "report: ${DATE}" --arg content "${CONTENT}" --arg sha "${SHA}" \
|
||||
'{message:$msg, content:$content, sha:$sha}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
|
||||
echo "updated ${REPORT_PATH} in rustfs/dashboard"
|
||||
else
|
||||
jq -n --arg msg "report: ${DATE}" --arg content "${CONTENT}" \
|
||||
'{message:$msg, content:$content}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
|
||||
echo "created ${REPORT_PATH} in rustfs/dashboard"
|
||||
fi
|
||||
|
||||
- name: Upload test logs & results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-perf-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-perf-test*.log
|
||||
/tmp/rustfs-perf-results/**
|
||||
/tmp/rustfs-version.txt
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
if: ${{ always() && inputs.cleanup_after != 'false' }}
|
||||
run: |
|
||||
./auto-testing/rustfs_performance_test.sh --step 7 -y
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS performance test failed"
|
||||
echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
|
||||
echo "See the uploaded log artifact for details."
|
||||
@@ -1,11 +1,12 @@
|
||||
name: RustFS Pool Expansion / Heal Test
|
||||
name: RustFS Pool Expansion / Decommission Test
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
rustfs_version:
|
||||
description: 'RustFS release tag to test (leave empty to use the latest nightly deb)'
|
||||
description: 'RustFS release tag to test (e.g. 1.0.0-rc.3)'
|
||||
required: false
|
||||
default: '1.0.0-rc.3'
|
||||
package_url:
|
||||
description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.'
|
||||
required: false
|
||||
@@ -29,14 +30,6 @@ on:
|
||||
description: 'Run the pool decommission step (3-pool topology only)'
|
||||
type: boolean
|
||||
default: true
|
||||
stop_node_gb:
|
||||
description: 'Heal: stop the outage node when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '15'
|
||||
warp_stop_gb:
|
||||
description: 'Heal: stop warp when surviving nodes reach N GiB'
|
||||
required: false
|
||||
default: '40'
|
||||
cleanup_before:
|
||||
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
|
||||
type: boolean
|
||||
@@ -45,19 +38,17 @@ on:
|
||||
description: 'Reset the nodes after the test (DESTROYS test data/config)'
|
||||
type: boolean
|
||||
default: true
|
||||
workflow_run:
|
||||
# Strict shared-environment order: run after tier test succeeds.
|
||||
workflows: ["RustFS Tier Test"]
|
||||
types: [completed]
|
||||
schedule:
|
||||
# Nightly regression run; remove if you do not want a schedule.
|
||||
- cron: '0 21 * * *'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# Only one test run at a time: every job mutates the same shared test
|
||||
# environment (vm000/vm001/vm002), so concurrent runs must not clobber each
|
||||
# other. Jobs inside a run are chained with needs to serialize them.
|
||||
# Only one pool-expansion test at a time: the workflow mutates a shared
|
||||
# test environment, so concurrent runs must not clobber each other.
|
||||
concurrency:
|
||||
group: rustfs-shared-functional-tests
|
||||
group: rustfs-pool-expansion-test
|
||||
cancel-in-progress: false
|
||||
|
||||
defaults:
|
||||
@@ -70,25 +61,19 @@ env:
|
||||
RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }}
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
# Package used by the nightly run (workflow_dispatch inputs are empty for
|
||||
# workflow_run events), i.e. the latest nightly deb published by nightly-gnu.yml.
|
||||
# Package used by the scheduled run (workflow_dispatch inputs are empty for
|
||||
# schedule events), i.e. the latest nightly deb published by nightly-gnu.yml.
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
|
||||
jobs:
|
||||
pool-expansion-test:
|
||||
name: Pool expansion / decommission test
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 360
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
- name: Checkout
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
@@ -101,12 +86,12 @@ jobs:
|
||||
- name: Reset test environment (before)
|
||||
if: ${{ inputs.cleanup_before != 'false' }}
|
||||
run: |
|
||||
chmod +x auto-testing/rustfs_pool_expand.sh
|
||||
./auto-testing/rustfs_pool_expand.sh --reset -y
|
||||
chmod +x scripts/test/rustfs_pool_expand.sh
|
||||
./scripts/test/rustfs_pool_expand.sh --reset -y
|
||||
|
||||
- name: Install RustFS package & start first pool
|
||||
run: |
|
||||
ARGS=(--steps "1,2,3" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
ARGS=(--steps 1,2,3 -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
elif [ -n "${{ inputs.rustfs_version }}" ]; then
|
||||
@@ -114,7 +99,7 @@ jobs:
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_pool_expand.sh "${ARGS[@]}"
|
||||
./scripts/test/rustfs_pool_expand.sh "${ARGS[@]}"
|
||||
|
||||
- name: Preflight checks
|
||||
run: |
|
||||
@@ -126,7 +111,7 @@ jobs:
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_pool_expand.sh "${ARGS[@]}"
|
||||
./scripts/test/rustfs_pool_expand.sh "${ARGS[@]}"
|
||||
|
||||
- name: Run pool expansion & decommission test
|
||||
id: pool_test
|
||||
@@ -139,19 +124,12 @@ jobs:
|
||||
STEPS="$STEPS,9"
|
||||
fi
|
||||
fi
|
||||
ARGS=(--steps "$STEPS" --with-warp -y \
|
||||
./scripts/test/rustfs_pool_expand.sh \
|
||||
--steps "$STEPS" --with-warp -y \
|
||||
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
|
||||
--storage-threshold "${{ inputs.storage_threshold || '50' }}" \
|
||||
--warp-duration "${{ inputs.warp_duration || '10m' }}" \
|
||||
--log-file /tmp/rustfs-pool-test.log)
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
elif [ -n "${{ inputs.rustfs_version }}" ]; then
|
||||
ARGS+=(--version "${{ inputs.rustfs_version }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_pool_expand.sh "${ARGS[@]}"
|
||||
--log-file /tmp/rustfs-pool-test.log
|
||||
|
||||
- name: Upload test logs
|
||||
if: always()
|
||||
@@ -159,14 +137,14 @@ jobs:
|
||||
with:
|
||||
name: rustfs-pool-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-pool-test*.log
|
||||
/tmp/rustfs-warp.*.log
|
||||
/tmp/rustfs-pool-test.log
|
||||
/tmp/rustfs-warp.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
if: ${{ always() && inputs.cleanup_after != 'false' }}
|
||||
run: |
|
||||
./auto-testing/rustfs_pool_expand.sh --reset -y
|
||||
./scripts/test/rustfs_pool_expand.sh --reset -y
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
@@ -174,82 +152,3 @@ jobs:
|
||||
echo "RustFS pool expansion test failed"
|
||||
echo "Package source: ${{ inputs.package_url || inputs.rustfs_version || 'nightly (R2 latest)' }}"
|
||||
echo "See the uploaded log artifact for details."
|
||||
|
||||
# Heal regression runs after the pool test regardless of its outcome: a pool
|
||||
# failure must be reported (it makes the run red) but must not block heal.
|
||||
heal-test:
|
||||
name: Heal test (after pool test)
|
||||
runs-on: smoke-testing
|
||||
timeout-minutes: 480
|
||||
needs: pool-expansion-test
|
||||
if: ${{ always() && (github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success') }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Reset test environment (before)
|
||||
if: ${{ inputs.cleanup_before != 'false' }}
|
||||
run: |
|
||||
chmod +x auto-testing/rustfs_heal_test.sh
|
||||
./auto-testing/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Install RustFS package & start cluster
|
||||
run: |
|
||||
ARGS=(--steps "1,2" -y --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Preflight checks
|
||||
run: |
|
||||
ARGS=(--preflight --endpoint "${{ env.RUSTFS_API_ENDPOINT }}")
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Run heal test (write -> outage -> heal -> verify)
|
||||
run: |
|
||||
ARGS=(--steps "3,4,5,6,7" -y \
|
||||
--endpoint "${{ env.RUSTFS_API_ENDPOINT }}" \
|
||||
--stop-node-gb "${{ inputs.stop_node_gb || '15' }}" \
|
||||
--warp-stop-gb "${{ inputs.warp_stop_gb || '40' }}" \
|
||||
--log-file /tmp/rustfs-heal-test.log)
|
||||
if [ -n "${{ inputs.package_url }}" ]; then
|
||||
ARGS+=(--package-url "${{ inputs.package_url }}")
|
||||
else
|
||||
ARGS+=(--package-url "${{ env.RUSTFS_NIGHTLY_PACKAGE_URL }}")
|
||||
fi
|
||||
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Upload test logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-heal-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-heal-test.log
|
||||
/tmp/rustfs-warp.*.log
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Reset test environment (after)
|
||||
if: ${{ always() && inputs.cleanup_after != 'false' }}
|
||||
run: |
|
||||
./auto-testing/rustfs_heal_test.sh --reset -y
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS heal test failed"
|
||||
echo "See the uploaded log artifact for details."
|
||||
|
||||
@@ -1,341 +0,0 @@
|
||||
name: RustFS S3 Compatibility Test
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
rustfs_version:
|
||||
description: 'RustFS release tag to test (e.g. 1.0.0-rc.4-preview.1)'
|
||||
required: false
|
||||
default: '1.0.0-rc.4-preview.1'
|
||||
package_url:
|
||||
description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.'
|
||||
required: false
|
||||
type: string
|
||||
workflow_run:
|
||||
# Run after the nightly build completes; the nightly deb is what the test installs.
|
||||
workflows: ["Nightly GNU Build"]
|
||||
types: [completed]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: rustfs-shared-functional-tests
|
||||
cancel-in-progress: false
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
env:
|
||||
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
|
||||
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
jobs:
|
||||
s3-compat-test:
|
||||
runs-on: smoke-testing
|
||||
continue-on-error: true
|
||||
timeout-minutes: 360
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
uname -a
|
||||
jq --version
|
||||
openssl version
|
||||
df -h /data | tail -1
|
||||
|
||||
- name: Cleanup environment (before)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
|
||||
SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
|
||||
for node in "${NODES[@]}"; do
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${SSH_USER}@${node}" '
|
||||
set -euo pipefail
|
||||
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
|
||||
${SUDO} systemctl stop rustfs 2>/dev/null || true
|
||||
if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
|
||||
${SUDO} dpkg -P rustfs
|
||||
fi
|
||||
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
|
||||
${SUDO} rm -rf /var/log/rustfs
|
||||
'
|
||||
done
|
||||
|
||||
- name: Run S3 compatibility suite
|
||||
id: test
|
||||
continue-on-error: true
|
||||
env:
|
||||
LOG_FILE: /tmp/rustfs-s3-compat.log
|
||||
run: |
|
||||
set -euo pipefail
|
||||
chmod +x auto-testing/rustfs-s3-compat-test.sh
|
||||
PACKAGE_URL='${{ inputs.package_url }}'
|
||||
RUSTFS_VERSION='${{ inputs.rustfs_version }}'
|
||||
ARGS=(--all-topologies -y --log-file "${LOG_FILE}")
|
||||
if [ -n "${PACKAGE_URL}" ]; then
|
||||
ARGS+=(--package-url "${PACKAGE_URL}")
|
||||
elif [ -n "${RUSTFS_VERSION}" ]; then
|
||||
ARGS+=(--version "${RUSTFS_VERSION}")
|
||||
else
|
||||
ARGS+=(--package-url "${RUSTFS_NIGHTLY_PACKAGE_URL}")
|
||||
fi
|
||||
./auto-testing/rustfs-s3-compat-test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Generate report
|
||||
if: always()
|
||||
env:
|
||||
LOG_FILE: /tmp/rustfs-s3-compat.log
|
||||
REPORT_FILE: /tmp/rustfs-s3-compat-report.md
|
||||
run: |
|
||||
set -euo pipefail
|
||||
PACKAGE_URL='${{ inputs.package_url }}'
|
||||
RUSTFS_VERSION='${{ inputs.rustfs_version }}'
|
||||
if [ -n "${PACKAGE_URL}" ]; then
|
||||
PACKAGE_SOURCE="${PACKAGE_URL}"
|
||||
elif [ -n "${RUSTFS_VERSION}" ]; then
|
||||
PACKAGE_SOURCE="version ${RUSTFS_VERSION}"
|
||||
else
|
||||
PACKAGE_SOURCE="${RUSTFS_NIGHTLY_PACKAGE_URL}"
|
||||
fi
|
||||
CASE_TABLE="/tmp/rustfs-s3-compat-cases.md"
|
||||
python3 - "${LOG_FILE}" "${CASE_TABLE}" <<'PY'
|
||||
import re
|
||||
import sys
|
||||
|
||||
log_file, out_file = sys.argv[1], sys.argv[2]
|
||||
ansi = re.compile(r'\x1b\[[0-9;]*m')
|
||||
start_re = re.compile(r'^---\s+([A-Z]+-[0-9]+)\s+(.+?)\s+---$')
|
||||
done_re = re.compile(r'^\[(PASS|FAIL|UNSUPPORTED)\]\s+([A-Z]+-[0-9]+)\b')
|
||||
|
||||
rows = []
|
||||
index = {}
|
||||
current = None
|
||||
try:
|
||||
with open(log_file, 'r', encoding='utf-8', errors='replace') as fh:
|
||||
for raw in fh:
|
||||
line = ansi.sub('', raw).strip()
|
||||
m = start_re.match(line)
|
||||
if m:
|
||||
case_id, name = m.group(1), m.group(2)
|
||||
current = case_id
|
||||
if case_id not in index:
|
||||
index[case_id] = len(rows)
|
||||
rows.append([case_id, name, 'RUNNING'])
|
||||
continue
|
||||
m = done_re.match(line)
|
||||
if m:
|
||||
status, case_id = m.group(1), m.group(2)
|
||||
if case_id in index:
|
||||
rows[index[case_id]][2] = status
|
||||
else:
|
||||
rows.append([case_id, case_id, status])
|
||||
index[case_id] = len(rows) - 1
|
||||
current = None
|
||||
except FileNotFoundError:
|
||||
rows = []
|
||||
|
||||
counts = {'PASS': 0, 'FAIL': 0, 'UNSUPPORTED': 0, 'RUNNING': 0}
|
||||
for _, _, status in rows:
|
||||
counts[status] = counts.get(status, 0) + 1
|
||||
|
||||
with open(out_file, 'w', encoding='utf-8') as out:
|
||||
out.write('## Case Summary\n\n')
|
||||
out.write(f"- Total: {len(rows)}\\n")
|
||||
out.write(f"- PASS: {counts.get('PASS', 0)}\\n")
|
||||
out.write(f"- FAIL: {counts.get('FAIL', 0)}\\n")
|
||||
out.write(f"- UNSUPPORTED: {counts.get('UNSUPPORTED', 0)}\\n")
|
||||
out.write('\\n')
|
||||
out.write('| Case | Name | Status |\\n')
|
||||
out.write('| --- | --- | --- |\\n')
|
||||
for case_id, name, status in rows:
|
||||
out.write(f'| {case_id} | {name} | {status} |\\n')
|
||||
PY
|
||||
{
|
||||
echo "# RustFS S3 compatibility test report"
|
||||
echo ""
|
||||
echo "- Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||
echo "- Trigger: ${{ github.event_name }}"
|
||||
echo "- Package: ${PACKAGE_SOURCE}"
|
||||
echo "- Test Step Outcome: ${{ steps.test.outcome }}"
|
||||
echo ""
|
||||
cat "${CASE_TABLE}" || true
|
||||
echo ""
|
||||
echo "## Log tail"
|
||||
echo '```text'
|
||||
tail -n 200 "${LOG_FILE}" || true
|
||||
echo '```'
|
||||
} | tee "${REPORT_FILE}"
|
||||
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
|
||||
|
||||
- name: Upload functional report to dashboard
|
||||
if: always()
|
||||
continue-on-error: true
|
||||
env:
|
||||
GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }}
|
||||
REPORT_FILE: /tmp/rustfs-s3-compat-report.md
|
||||
SUITE: s3
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${GH_TOKEN:-}" ]; then
|
||||
echo "PF_TESTING_GH_TOKEN is not configured; skipping dashboard upload"
|
||||
exit 0
|
||||
fi
|
||||
DATE="$(date -u +%Y-%m-%d)"
|
||||
REPORT_PATH="functional-reports/${SUITE}/${DATE}.md"
|
||||
CONTENT="$(python3 -c 'import base64,sys;print(base64.b64encode(open(sys.argv[1],"rb").read()).decode())' "${REPORT_FILE}")"
|
||||
SHA="$(gh api "repos/rustfs/dashboard/contents/${REPORT_PATH}" -q '.sha' 2>/dev/null || true)"
|
||||
if [ -n "${SHA}" ]; then
|
||||
jq -n --arg msg "report(${SUITE}): ${DATE}" --arg content "${CONTENT}" --arg sha "${SHA}" \
|
||||
'{message:$msg, content:$content, sha:$sha}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
|
||||
else
|
||||
jq -n --arg msg "report(${SUITE}): ${DATE}" --arg content "${CONTENT}" \
|
||||
'{message:$msg, content:$content}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
|
||||
fi
|
||||
|
||||
cat > /tmp/rustfs-functional-index.html <<'EOF'
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>RustFS Functional Test Reports</title>
|
||||
<style>
|
||||
:root { --bg:#f4f6fb; --card:#fff; --text:#1f2937; --muted:#6b7280; --line:#e5e7eb; --accent:#0f766e; }
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; font-family: ui-sans-serif, -apple-system, Segoe UI, Helvetica, Arial, sans-serif; background: var(--bg); color: var(--text); }
|
||||
.wrap { max-width: 980px; margin: 32px auto; padding: 0 16px; }
|
||||
.card { background: var(--card); border: 1px solid var(--line); border-radius: 14px; padding: 20px; }
|
||||
h1 { margin: 0 0 8px; font-size: 26px; }
|
||||
p { margin: 0 0 14px; color: var(--muted); }
|
||||
.tabs { display: flex; gap: 10px; margin: 14px 0 18px; flex-wrap: wrap; }
|
||||
button { border: 1px solid var(--line); background: #fff; color: var(--text); border-radius: 10px; padding: 8px 14px; cursor: pointer; }
|
||||
button.active { background: var(--accent); color: #fff; border-color: var(--accent); }
|
||||
ul { list-style: none; margin: 0; padding: 0; }
|
||||
li { padding: 10px 0; border-bottom: 1px solid var(--line); }
|
||||
a { color: var(--accent); text-decoration: none; }
|
||||
a:hover { text-decoration: underline; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
<div class="card">
|
||||
<h1>RustFS Functional Test Reports</h1>
|
||||
<p>S3, KMS, Tier report tabs. Each tab lists reports by date.</p>
|
||||
<div class="tabs" id="tabs"></div>
|
||||
<ul id="list"></ul>
|
||||
</div>
|
||||
</div>
|
||||
<script>
|
||||
const suites = [
|
||||
{ key: 's3', label: 'S3 Compatibility' },
|
||||
{ key: 'kms', label: 'KMS' },
|
||||
{ key: 'tier', label: 'Tier' },
|
||||
];
|
||||
const tabs = document.getElementById('tabs');
|
||||
const list = document.getElementById('list');
|
||||
|
||||
async function loadSuite(suite) {
|
||||
list.innerHTML = '<li>Loading...</li>';
|
||||
const api = `https://api.github.com/repos/rustfs/dashboard/contents/functional-reports/${suite}`;
|
||||
try {
|
||||
const res = await fetch(api);
|
||||
if (!res.ok) {
|
||||
list.innerHTML = '<li>No reports yet.</li>';
|
||||
return;
|
||||
}
|
||||
const data = await res.json();
|
||||
const files = data.filter(f => f.type === 'file' && f.name.endsWith('.md')).sort((a,b) => b.name.localeCompare(a.name));
|
||||
if (!files.length) {
|
||||
list.innerHTML = '<li>No reports yet.</li>';
|
||||
return;
|
||||
}
|
||||
list.innerHTML = files.map(f => `<li><a href="${f.html_url}" target="_blank" rel="noreferrer">${f.name.replace('.md','')}</a></li>`).join('');
|
||||
} catch (_e) {
|
||||
list.innerHTML = '<li>Failed to load reports.</li>';
|
||||
}
|
||||
}
|
||||
|
||||
function setActive(key) {
|
||||
for (const btn of tabs.querySelectorAll('button')) {
|
||||
btn.classList.toggle('active', btn.dataset.key === key);
|
||||
}
|
||||
loadSuite(key);
|
||||
}
|
||||
|
||||
for (const suite of suites) {
|
||||
const btn = document.createElement('button');
|
||||
btn.textContent = suite.label;
|
||||
btn.dataset.key = suite.key;
|
||||
btn.addEventListener('click', () => setActive(suite.key));
|
||||
tabs.appendChild(btn);
|
||||
}
|
||||
setActive('s3');
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
EOF
|
||||
|
||||
INDEX_PATH="functional/index.html"
|
||||
INDEX_CONTENT="$(python3 -c 'import base64;print(base64.b64encode(open("/tmp/rustfs-functional-index.html","rb").read()).decode())')"
|
||||
INDEX_SHA="$(gh api "repos/rustfs/dashboard/contents/${INDEX_PATH}" -q '.sha' 2>/dev/null || true)"
|
||||
if [ -n "${INDEX_SHA}" ]; then
|
||||
jq -n --arg msg "functional ui update" --arg content "${INDEX_CONTENT}" --arg sha "${INDEX_SHA}" \
|
||||
'{message:$msg, content:$content, sha:$sha}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${INDEX_PATH}" --input - >/dev/null
|
||||
else
|
||||
jq -n --arg msg "functional ui init" --arg content "${INDEX_CONTENT}" \
|
||||
'{message:$msg, content:$content}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${INDEX_PATH}" --input - >/dev/null
|
||||
fi
|
||||
|
||||
- name: Upload report and logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-s3-compat-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-s3-compat.log
|
||||
/tmp/rustfs-s3-compat-report.md
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Cleanup environment (after)
|
||||
if: always()
|
||||
run: |
|
||||
set -euo pipefail
|
||||
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
|
||||
SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
|
||||
for node in "${NODES[@]}"; do
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${SSH_USER}@${node}" '
|
||||
set -euo pipefail
|
||||
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
|
||||
${SUDO} systemctl stop rustfs 2>/dev/null || true
|
||||
if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
|
||||
${SUDO} dpkg -P rustfs
|
||||
fi
|
||||
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
|
||||
${SUDO} rm -rf /var/log/rustfs
|
||||
'
|
||||
done
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS S3 compatibility suite failed"
|
||||
echo "See the uploaded report and log artifacts for details."
|
||||
@@ -1,370 +0,0 @@
|
||||
name: RustFS Tier Test
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
rustfs_version:
|
||||
description: 'RustFS release tag to test (e.g. 1.0.0-rc.4-preview.1)'
|
||||
required: false
|
||||
default: '1.0.0-rc.4-preview.1'
|
||||
package_url:
|
||||
description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.'
|
||||
required: false
|
||||
type: string
|
||||
workflow_run:
|
||||
# Strict shared-environment order: run after KMS test succeeds.
|
||||
workflows: ["RustFS KMS Test"]
|
||||
types: [completed]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: rustfs-shared-functional-tests
|
||||
cancel-in-progress: false
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
env:
|
||||
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
|
||||
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
|
||||
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
|
||||
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
|
||||
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
|
||||
PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
jobs:
|
||||
tier-test:
|
||||
runs-on: smoke-testing
|
||||
continue-on-error: true
|
||||
timeout-minutes: 420
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
|
||||
steps:
|
||||
- name: Checkout auto-testing scripts
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: main
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
uname -a
|
||||
jq --version
|
||||
openssl version
|
||||
df -h /data | tail -1
|
||||
|
||||
- name: Cleanup environment (before)
|
||||
run: |
|
||||
set -euo pipefail
|
||||
sudo docker rm -f rustfs-test-mqtt >/dev/null 2>&1 || true
|
||||
sudo rm -f /tmp/rustfs-mosquitto.conf
|
||||
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
|
||||
SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
|
||||
for node in "${NODES[@]}"; do
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${SSH_USER}@${node}" '
|
||||
set -euo pipefail
|
||||
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
|
||||
${SUDO} systemctl stop rustfs 2>/dev/null || true
|
||||
if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
|
||||
${SUDO} dpkg -P rustfs
|
||||
fi
|
||||
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
|
||||
${SUDO} rm -rf /var/log/rustfs /var/lib/rustfs/kms
|
||||
'
|
||||
done
|
||||
|
||||
- name: Ensure MQTT broker + clients
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if ! command -v mosquitto_sub >/dev/null 2>&1; then
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y mosquitto-clients
|
||||
fi
|
||||
command -v docker >/dev/null 2>&1 || { echo 'docker not found on runner'; exit 1; }
|
||||
sudo docker rm -f rustfs-test-mqtt >/dev/null 2>&1 || true
|
||||
cat <<'EOF' | sudo tee /tmp/rustfs-mosquitto.conf >/dev/null
|
||||
listener 1883 0.0.0.0
|
||||
allow_anonymous true
|
||||
EOF
|
||||
sudo docker run -d --name rustfs-test-mqtt -p 1883:1883 \
|
||||
-v /tmp/rustfs-mosquitto.conf:/mosquitto/config/mosquitto.conf:ro \
|
||||
eclipse-mosquitto:2 >/dev/null
|
||||
for _ in {1..10}; do
|
||||
if ss -tln 2>/dev/null | grep -q ':1883'; then
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
ss -tln 2>/dev/null | grep -q ':1883' || {
|
||||
echo 'mosquitto container is not listening on 1883'
|
||||
sudo docker logs rustfs-test-mqtt || true
|
||||
exit 1
|
||||
}
|
||||
|
||||
- name: Run tier suite
|
||||
id: test
|
||||
continue-on-error: true
|
||||
env:
|
||||
LOG_FILE: /tmp/rustfs-tier.log
|
||||
run: |
|
||||
set -euo pipefail
|
||||
chmod +x auto-testing/rustfs-tier-test.sh
|
||||
PACKAGE_URL='${{ inputs.package_url }}'
|
||||
RUSTFS_VERSION='${{ inputs.rustfs_version }}'
|
||||
ARGS=(--all-topologies -y --log-file "${LOG_FILE}")
|
||||
if [ -n "${PACKAGE_URL}" ]; then
|
||||
ARGS+=(--package-url "${PACKAGE_URL}")
|
||||
elif [ -n "${RUSTFS_VERSION}" ]; then
|
||||
ARGS+=(--version "${RUSTFS_VERSION}")
|
||||
else
|
||||
ARGS+=(--package-url "${RUSTFS_NIGHTLY_PACKAGE_URL}")
|
||||
fi
|
||||
./auto-testing/rustfs-tier-test.sh "${ARGS[@]}"
|
||||
|
||||
- name: Generate report
|
||||
if: always()
|
||||
env:
|
||||
LOG_FILE: /tmp/rustfs-tier.log
|
||||
REPORT_FILE: /tmp/rustfs-tier-report.md
|
||||
run: |
|
||||
set -euo pipefail
|
||||
PACKAGE_URL='${{ inputs.package_url }}'
|
||||
RUSTFS_VERSION='${{ inputs.rustfs_version }}'
|
||||
if [ -n "${PACKAGE_URL}" ]; then
|
||||
PACKAGE_SOURCE="${PACKAGE_URL}"
|
||||
elif [ -n "${RUSTFS_VERSION}" ]; then
|
||||
PACKAGE_SOURCE="version ${RUSTFS_VERSION}"
|
||||
else
|
||||
PACKAGE_SOURCE="${RUSTFS_NIGHTLY_PACKAGE_URL}"
|
||||
fi
|
||||
CASE_TABLE="/tmp/rustfs-tier-cases.md"
|
||||
python3 - "${LOG_FILE}" "${CASE_TABLE}" <<'PY'
|
||||
import re
|
||||
import sys
|
||||
|
||||
log_file, out_file = sys.argv[1], sys.argv[2]
|
||||
ansi = re.compile(r'\x1b\[[0-9;]*m')
|
||||
start_re = re.compile(r'^---\s+([A-Z]+-[0-9]+)\s+(.+?)\s+---$')
|
||||
done_re = re.compile(r'^\[(PASS|FAIL|UNSUPPORTED)\]\s+([A-Z]+-[0-9]+)\b')
|
||||
|
||||
rows = []
|
||||
index = {}
|
||||
try:
|
||||
with open(log_file, 'r', encoding='utf-8', errors='replace') as fh:
|
||||
for raw in fh:
|
||||
line = ansi.sub('', raw).strip()
|
||||
m = start_re.match(line)
|
||||
if m:
|
||||
case_id, name = m.group(1), m.group(2)
|
||||
if case_id not in index:
|
||||
index[case_id] = len(rows)
|
||||
rows.append([case_id, name, 'RUNNING'])
|
||||
continue
|
||||
m = done_re.match(line)
|
||||
if m:
|
||||
status, case_id = m.group(1), m.group(2)
|
||||
if case_id in index:
|
||||
rows[index[case_id]][2] = status
|
||||
else:
|
||||
rows.append([case_id, case_id, status])
|
||||
index[case_id] = len(rows) - 1
|
||||
except FileNotFoundError:
|
||||
rows = []
|
||||
|
||||
counts = {'PASS': 0, 'FAIL': 0, 'UNSUPPORTED': 0, 'RUNNING': 0}
|
||||
for _, _, status in rows:
|
||||
counts[status] = counts.get(status, 0) + 1
|
||||
|
||||
with open(out_file, 'w', encoding='utf-8') as out:
|
||||
out.write('## Case Summary\n\n')
|
||||
out.write(f"- Total: {len(rows)}\\n")
|
||||
out.write(f"- PASS: {counts.get('PASS', 0)}\\n")
|
||||
out.write(f"- FAIL: {counts.get('FAIL', 0)}\\n")
|
||||
out.write(f"- UNSUPPORTED: {counts.get('UNSUPPORTED', 0)}\\n")
|
||||
out.write('\\n')
|
||||
out.write('| Case | Name | Status |\\n')
|
||||
out.write('| --- | --- | --- |\\n')
|
||||
for case_id, name, status in rows:
|
||||
out.write(f'| {case_id} | {name} | {status} |\\n')
|
||||
PY
|
||||
{
|
||||
echo "# RustFS tier test report"
|
||||
echo ""
|
||||
echo "- Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||||
echo "- Trigger: ${{ github.event_name }}"
|
||||
echo "- Package: ${PACKAGE_SOURCE}"
|
||||
echo "- Test Step Outcome: ${{ steps.test.outcome }}"
|
||||
echo ""
|
||||
cat "${CASE_TABLE}" || true
|
||||
echo ""
|
||||
echo "## Log tail"
|
||||
echo '```text'
|
||||
tail -n 200 "${LOG_FILE}" || true
|
||||
echo '```'
|
||||
} | tee "${REPORT_FILE}"
|
||||
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
|
||||
|
||||
- name: Upload functional report to dashboard
|
||||
if: always()
|
||||
continue-on-error: true
|
||||
env:
|
||||
GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }}
|
||||
REPORT_FILE: /tmp/rustfs-tier-report.md
|
||||
SUITE: tier
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${GH_TOKEN:-}" ]; then
|
||||
echo "PF_TESTING_GH_TOKEN is not configured; skipping dashboard upload"
|
||||
exit 0
|
||||
fi
|
||||
DATE="$(date -u +%Y-%m-%d)"
|
||||
REPORT_PATH="functional-reports/${SUITE}/${DATE}.md"
|
||||
CONTENT="$(python3 -c 'import base64,sys;print(base64.b64encode(open(sys.argv[1],"rb").read()).decode())' "${REPORT_FILE}")"
|
||||
SHA="$(gh api "repos/rustfs/dashboard/contents/${REPORT_PATH}" -q '.sha' 2>/dev/null || true)"
|
||||
if [ -n "${SHA}" ]; then
|
||||
jq -n --arg msg "report(${SUITE}): ${DATE}" --arg content "${CONTENT}" --arg sha "${SHA}" \
|
||||
'{message:$msg, content:$content, sha:$sha}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
|
||||
else
|
||||
jq -n --arg msg "report(${SUITE}): ${DATE}" --arg content "${CONTENT}" \
|
||||
'{message:$msg, content:$content}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
|
||||
fi
|
||||
|
||||
cat > /tmp/rustfs-functional-index.html <<'EOF'
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>RustFS Functional Test Reports</title>
|
||||
<style>
|
||||
:root { --bg:#f4f6fb; --card:#fff; --text:#1f2937; --muted:#6b7280; --line:#e5e7eb; --accent:#0f766e; }
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; font-family: ui-sans-serif, -apple-system, Segoe UI, Helvetica, Arial, sans-serif; background: var(--bg); color: var(--text); }
|
||||
.wrap { max-width: 980px; margin: 32px auto; padding: 0 16px; }
|
||||
.card { background: var(--card); border: 1px solid var(--line); border-radius: 14px; padding: 20px; }
|
||||
h1 { margin: 0 0 8px; font-size: 26px; }
|
||||
p { margin: 0 0 14px; color: var(--muted); }
|
||||
.tabs { display: flex; gap: 10px; margin: 14px 0 18px; flex-wrap: wrap; }
|
||||
button { border: 1px solid var(--line); background: #fff; color: var(--text); border-radius: 10px; padding: 8px 14px; cursor: pointer; }
|
||||
button.active { background: var(--accent); color: #fff; border-color: var(--accent); }
|
||||
ul { list-style: none; margin: 0; padding: 0; }
|
||||
li { padding: 10px 0; border-bottom: 1px solid var(--line); }
|
||||
a { color: var(--accent); text-decoration: none; }
|
||||
a:hover { text-decoration: underline; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
<div class="card">
|
||||
<h1>RustFS Functional Test Reports</h1>
|
||||
<p>S3, KMS, Tier report tabs. Each tab lists reports by date.</p>
|
||||
<div class="tabs" id="tabs"></div>
|
||||
<ul id="list"></ul>
|
||||
</div>
|
||||
</div>
|
||||
<script>
|
||||
const suites = [
|
||||
{ key: 's3', label: 'S3 Compatibility' },
|
||||
{ key: 'kms', label: 'KMS' },
|
||||
{ key: 'tier', label: 'Tier' },
|
||||
];
|
||||
const tabs = document.getElementById('tabs');
|
||||
const list = document.getElementById('list');
|
||||
|
||||
async function loadSuite(suite) {
|
||||
list.innerHTML = '<li>Loading...</li>';
|
||||
const api = `https://api.github.com/repos/rustfs/dashboard/contents/functional-reports/${suite}`;
|
||||
try {
|
||||
const res = await fetch(api);
|
||||
if (!res.ok) {
|
||||
list.innerHTML = '<li>No reports yet.</li>';
|
||||
return;
|
||||
}
|
||||
const data = await res.json();
|
||||
const files = data.filter(f => f.type === 'file' && f.name.endsWith('.md')).sort((a,b) => b.name.localeCompare(a.name));
|
||||
if (!files.length) {
|
||||
list.innerHTML = '<li>No reports yet.</li>';
|
||||
return;
|
||||
}
|
||||
list.innerHTML = files.map(f => `<li><a href="${f.html_url}" target="_blank" rel="noreferrer">${f.name.replace('.md','')}</a></li>`).join('');
|
||||
} catch (_e) {
|
||||
list.innerHTML = '<li>Failed to load reports.</li>';
|
||||
}
|
||||
}
|
||||
|
||||
function setActive(key) {
|
||||
for (const btn of tabs.querySelectorAll('button')) {
|
||||
btn.classList.toggle('active', btn.dataset.key === key);
|
||||
}
|
||||
loadSuite(key);
|
||||
}
|
||||
|
||||
for (const suite of suites) {
|
||||
const btn = document.createElement('button');
|
||||
btn.textContent = suite.label;
|
||||
btn.dataset.key = suite.key;
|
||||
btn.addEventListener('click', () => setActive(suite.key));
|
||||
tabs.appendChild(btn);
|
||||
}
|
||||
setActive('s3');
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
EOF
|
||||
|
||||
INDEX_PATH="functional/index.html"
|
||||
INDEX_CONTENT="$(python3 -c 'import base64;print(base64.b64encode(open("/tmp/rustfs-functional-index.html","rb").read()).decode())')"
|
||||
INDEX_SHA="$(gh api "repos/rustfs/dashboard/contents/${INDEX_PATH}" -q '.sha' 2>/dev/null || true)"
|
||||
if [ -n "${INDEX_SHA}" ]; then
|
||||
jq -n --arg msg "functional ui update" --arg content "${INDEX_CONTENT}" --arg sha "${INDEX_SHA}" \
|
||||
'{message:$msg, content:$content, sha:$sha}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${INDEX_PATH}" --input - >/dev/null
|
||||
else
|
||||
jq -n --arg msg "functional ui init" --arg content "${INDEX_CONTENT}" \
|
||||
'{message:$msg, content:$content}' \
|
||||
| gh api --method PUT "repos/rustfs/dashboard/contents/${INDEX_PATH}" --input - >/dev/null
|
||||
fi
|
||||
|
||||
- name: Upload report and logs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
|
||||
with:
|
||||
name: rustfs-tier-test-${{ github.run_id }}
|
||||
path: |
|
||||
/tmp/rustfs-tier.log
|
||||
/tmp/rustfs-tier-report.md
|
||||
if-no-files-found: warn
|
||||
|
||||
- name: Cleanup environment (after)
|
||||
if: always()
|
||||
run: |
|
||||
set -euo pipefail
|
||||
sudo docker rm -f rustfs-test-mqtt >/dev/null 2>&1 || true
|
||||
sudo rm -f /tmp/rustfs-mosquitto.conf
|
||||
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
|
||||
SSH_USER="${RUSTFS_SSH_USER:-azureuser}"
|
||||
for node in "${NODES[@]}"; do
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new "${SSH_USER}@${node}" '
|
||||
set -euo pipefail
|
||||
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
|
||||
${SUDO} systemctl stop rustfs 2>/dev/null || true
|
||||
if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
|
||||
${SUDO} dpkg -P rustfs
|
||||
fi
|
||||
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
|
||||
${SUDO} rm -rf /var/log/rustfs /var/lib/rustfs/kms
|
||||
'
|
||||
done
|
||||
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
run: |
|
||||
echo "RustFS tier suite failed"
|
||||
echo "See the uploaded report and log artifacts for details."
|
||||
@@ -31,13 +31,8 @@ This file contains repository-wide rules. Use the nearest subdirectory
|
||||
- An existing clean, isolated task worktree is sufficient. Create another
|
||||
worktree only when the current checkout is shared, dirty with unrelated work,
|
||||
or belongs to another task.
|
||||
- Never commit from a shared checkout.
|
||||
- Use a task-specific branch named `<type>/<topic>`, such as `fix/...`,
|
||||
`feat/...`, `test/...`, or `docs/...`, unless the user specifies a name.
|
||||
- Do not include agent, tool, contributor, account, or organization names in
|
||||
branch names.
|
||||
- Push to the user-requested remote or the repository's configured push remote.
|
||||
Do not hard-code or infer a remote from an account name.
|
||||
- Never commit from a shared checkout. Use an `overtrue/` feature branch unless
|
||||
the user requests another name.
|
||||
- Check free space before artifact-heavy builds, tests, coverage, or downloads.
|
||||
Re-check before a broad gate when space is tight.
|
||||
- Remove only task-owned temporary/build artifacts. Never delete another task's
|
||||
|
||||
Generated
+171
-298
File diff suppressed because it is too large
Load Diff
+11
-11
@@ -155,7 +155,7 @@ futures-util = "0.3.34"
|
||||
pollster = "1.0.1"
|
||||
pulsar = { default-features = false, version = "6.9.0" }
|
||||
lapin = { default-features = false, version = "4.10.0" }
|
||||
hyper = { version = "1.11.1" }
|
||||
hyper = { version = "1.11.0" }
|
||||
hyper-rustls = { default-features = false, version = "0.27.9" }
|
||||
hyper-util = { version = "0.1.20" }
|
||||
http = "1.5.0"
|
||||
@@ -198,7 +198,7 @@ serde_urlencoded = "0.7.1"
|
||||
# have incompatible APIs. Keep them exact-pinned and monitor upstream for stable
|
||||
# releases.
|
||||
aes-gcm = { version = "=0.11.1" }
|
||||
argon2 = { version = "=0.6.0" }
|
||||
argon2 = { version = "=0.6.0-rc.8" }
|
||||
blake2 = "=0.11.0"
|
||||
chacha20poly1305 = { version = "=0.11.0" }
|
||||
crc-fast = "1.10.0"
|
||||
@@ -232,7 +232,7 @@ tokio-postgres-rustls = "0.14.0"
|
||||
# Utilities and Tools
|
||||
anyhow = "1.0.104"
|
||||
arc-swap = "1.9.2"
|
||||
astral-tokio-tar = "0.7.0"
|
||||
astral-tokio-tar = "0.6.4"
|
||||
atoi = "3.1.0"
|
||||
atomic_enum = "0.3.0"
|
||||
aws-config = { version = "1.11.0" }
|
||||
@@ -247,7 +247,7 @@ base64-simd = "0.8.0"
|
||||
brotli = "8.0.4"
|
||||
clap = { version = "4.6.6" }
|
||||
const-str = { version = "1.1.0" }
|
||||
convert_case = "0.12.0"
|
||||
convert_case = "0.11.0"
|
||||
criterion = { version = "0.8" }
|
||||
crossbeam-queue = "0.3.13"
|
||||
crossbeam-channel = "0.5.16"
|
||||
@@ -257,10 +257,10 @@ datafusion = { default-features = false, version = "55.0.0" }
|
||||
derive_builder = "0.20.2"
|
||||
enumset = "1.1.14"
|
||||
faster-hex = "0.10.0"
|
||||
flate2 = "1.1.10"
|
||||
flate2 = "1.1.9"
|
||||
glob = "0.3.4"
|
||||
google-cloud-storage = "1.18.0"
|
||||
google-cloud-auth = "1.16.0"
|
||||
google-cloud-storage = "1.17.0"
|
||||
google-cloud-auth = "1.15.0"
|
||||
hashbrown = { version = "0.17.1" }
|
||||
# Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
|
||||
# every authenticator app expects). Already in the graph transitively.
|
||||
@@ -304,7 +304,7 @@ rustify = { version = "0.7", default-features = false }
|
||||
rustix = { version = "1.1.4" }
|
||||
rust-embed = { version = "8.12.0" }
|
||||
rustc-hash = { version = "2.1.3" }
|
||||
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "9c4690d8e73fc8d184031a19b2c4539ebc77d180", version = "0.15.0", features = ["minio"] }
|
||||
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "f4dedc905ec621fa85a4686df6304190b55375f6", version = "0.15.0", features = ["minio"] }
|
||||
serial_test = "4.0.1"
|
||||
shadow-rs = { default-features = false, version = "2.0.0" }
|
||||
siphasher = "1.0.3"
|
||||
@@ -327,7 +327,7 @@ tracing-subscriber = { version = "0.3.23" }
|
||||
transform-stream = "0.3.1"
|
||||
url = "2.5.8"
|
||||
urlencoding = "2.1.3"
|
||||
uuid = { version = "1.26.0" }
|
||||
uuid = { version = "1.25.0" }
|
||||
vaultrs = { version = "0.8.0" }
|
||||
tar = "0.4.46"
|
||||
walkdir = "2.5.0"
|
||||
@@ -341,7 +341,7 @@ zstd = "0.13.3"
|
||||
# Observability and Metrics
|
||||
metrics = "0.24.6"
|
||||
metrics-util = "0.20"
|
||||
dial9-tokio-telemetry = "0.5.0"
|
||||
dial9-tokio-telemetry = "0.3"
|
||||
opentelemetry = { version = "0.32.0" }
|
||||
opentelemetry-appender-tracing = { version = "0.32.0" }
|
||||
opentelemetry-otlp = { version = "0.32.0" }
|
||||
@@ -355,7 +355,7 @@ pyroscope = { version = "2.1.1" }
|
||||
libunftp = { version = "0.23.0" }
|
||||
unftp-core = "0.1.0"
|
||||
suppaftp = { version = "10.0.2" }
|
||||
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
|
||||
rcgen = { version = "0.14.9", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
|
||||
russh = { version = "0.63.1" }
|
||||
russh-sftp = "2.4.0"
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://docs.rustfs.com/en/installation">Getting Started</a>
|
||||
<a href="https://docs.rustfs.com/installation/">Getting Started</a>
|
||||
· <a href="https://docs.rustfs.com/">Docs</a>
|
||||
· <a href="https://github.com/rustfs/rustfs/issues">Bug reports</a>
|
||||
· <a href="https://github.com/rustfs/rustfs/discussions">Discussions</a>
|
||||
@@ -245,26 +245,6 @@ nix build
|
||||
nix run
|
||||
```
|
||||
|
||||
The flake also exports a NixOS module and the RustFS `rc` client. Add the
|
||||
module to your system and provide credentials through runtime files (for
|
||||
example, sops-nix or agenix) so secrets are never stored in the Nix store:
|
||||
|
||||
```nix
|
||||
imports = [ inputs.rustfs.nixosModules.rustfs ];
|
||||
|
||||
services.rustfs = {
|
||||
enable = true;
|
||||
accessKeyFile = "/run/secrets/rustfs-access-key";
|
||||
secretKeyFile = "/run/secrets/rustfs-secret-key";
|
||||
volumes = [ "/var/lib/rustfs" ];
|
||||
};
|
||||
```
|
||||
|
||||
Install the S3-compatible client with
|
||||
`nix profile install github:rustfs/rustfs#rustfs-client` (the executable is named
|
||||
`rc`), or use `inputs.rustfs.packages.${pkgs.system}.rustfs-client` in a system
|
||||
configuration.
|
||||
|
||||
### 6\. X-CMD (Option 6)
|
||||
|
||||
If you are an [x-cmd](https://www.x-cmd.com/install/rustfs) user:
|
||||
|
||||
+1
-7
@@ -16,7 +16,7 @@
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://docs.rustfs.com/zh/installation">快速开始</a>
|
||||
<a href="https://docs.rustfs.com/installation/">快速开始</a>
|
||||
· <a href="https://docs.rustfs.com/">文档</a>
|
||||
· <a href="https://github.com/rustfs/rustfs/issues">报告 Bug</a>
|
||||
· <a href="https://github.com/rustfs/rustfs/discussions">社区讨论</a>
|
||||
@@ -191,12 +191,6 @@ nix build
|
||||
nix run
|
||||
```
|
||||
|
||||
该 Flake 同时提供 NixOS 模块和 RustFS `rc` 客户端。将
|
||||
`inputs.rustfs.nixosModules.rustfs` 加入 `imports`，并通过运行时密钥文件
|
||||
（例如 sops-nix 或 agenix）配置 `accessKeyFile` 与 `secretKeyFile`，避免密钥
|
||||
进入 Nix store。客户端包为
|
||||
`inputs.rustfs.packages.${pkgs.system}.rustfs-client`，安装后的命令名为 `rc`。
|
||||
|
||||
### 6\. X-CMD (Option 6)
|
||||
|
||||
如果你是 [x-cmd](https://www.x-cmd.com/install/rustfs) 用户：
|
||||
|
||||
@@ -68,6 +68,7 @@ tracing = { workspace = true, features = ["std", "attributes"] }
|
||||
|
||||
[dev-dependencies]
|
||||
rustfs-targets = { workspace = true, features = ["test-support"] }
|
||||
async-trait = { workspace = true }
|
||||
temp-env = { workspace = true }
|
||||
url = { workspace = true }
|
||||
|
||||
|
||||
@@ -178,76 +178,6 @@ pub trait WorkloadAdmissionSnapshotProvider {
|
||||
fn workload_admission_snapshot(&self) -> WorkloadAdmissionRegistrySnapshot;
|
||||
}
|
||||
|
||||
/// Foreground workload pressure observed against a configured utilization threshold.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct ForegroundPressure {
|
||||
/// Foreground workload class whose utilization reached its threshold.
|
||||
pub class: WorkloadClass,
|
||||
/// Observed utilization percentage for the class.
|
||||
pub usage_pct: usize,
|
||||
/// Configured threshold percentage that the observed utilization reached.
|
||||
pub threshold_pct: usize,
|
||||
}
|
||||
|
||||
impl ForegroundPressure {
|
||||
/// Return a stable reason label for logs and metrics.
|
||||
pub const fn reason(self) -> &'static str {
|
||||
match self.class {
|
||||
WorkloadClass::ForegroundRead => "foreground_read_pressure",
|
||||
WorkloadClass::ForegroundWrite => "foreground_write_pressure",
|
||||
_ => "foreground_pressure",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Return the strongest foreground pressure in `snapshot`, if any.
|
||||
///
|
||||
/// A zero threshold disables its class. `Saturated` counts as full utilization
|
||||
/// regardless of the reported limit; otherwise a class contributes only when it
|
||||
/// reports a non-zero limit, with a missing active count read as zero. When both
|
||||
/// classes are above their threshold the higher utilization wins.
|
||||
///
|
||||
/// Callers own the enable switch: this function evaluates thresholds only.
|
||||
pub fn foreground_pressure(
|
||||
snapshot: &WorkloadAdmissionRegistrySnapshot,
|
||||
read_threshold_pct: usize,
|
||||
write_threshold_pct: usize,
|
||||
) -> Option<ForegroundPressure> {
|
||||
[
|
||||
(WorkloadClass::ForegroundRead, read_threshold_pct),
|
||||
(WorkloadClass::ForegroundWrite, write_threshold_pct),
|
||||
]
|
||||
.into_iter()
|
||||
.filter_map(|(class, threshold_pct)| {
|
||||
if threshold_pct == 0 {
|
||||
return None;
|
||||
}
|
||||
|
||||
let entry = snapshot.get(class)?;
|
||||
let usage_pct = if matches!(entry.state, AdmissionState::Saturated) {
|
||||
100
|
||||
} else {
|
||||
let limit = entry.limit?;
|
||||
if limit == 0 {
|
||||
return None;
|
||||
}
|
||||
entry
|
||||
.active
|
||||
.unwrap_or(0)
|
||||
.saturating_mul(100)
|
||||
.checked_div(limit)
|
||||
.unwrap_or(100)
|
||||
};
|
||||
|
||||
(usage_pct >= threshold_pct).then_some(ForegroundPressure {
|
||||
class,
|
||||
usage_pct,
|
||||
threshold_pct,
|
||||
})
|
||||
})
|
||||
.max_by_key(|pressure| pressure.usage_pct)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -384,205 +314,4 @@ mod tests {
|
||||
|
||||
assert!(err.to_string().contains("unexpected"));
|
||||
}
|
||||
|
||||
fn counted(
|
||||
class: WorkloadClass,
|
||||
state: AdmissionState,
|
||||
active: Option<usize>,
|
||||
limit: Option<usize>,
|
||||
) -> WorkloadAdmissionSnapshot {
|
||||
WorkloadAdmissionSnapshot::new(class, state).with_counts(active, None, limit)
|
||||
}
|
||||
|
||||
fn registry(entries: Vec<WorkloadAdmissionSnapshot>) -> WorkloadAdmissionRegistrySnapshot {
|
||||
WorkloadAdmissionRegistrySnapshot::new(entries)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_reason_labels_cover_non_foreground_classes() {
|
||||
let read = ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundRead,
|
||||
usage_pct: 90,
|
||||
threshold_pct: 80,
|
||||
};
|
||||
let write = ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundWrite,
|
||||
usage_pct: 90,
|
||||
threshold_pct: 80,
|
||||
};
|
||||
let repair = ForegroundPressure {
|
||||
class: WorkloadClass::Repair,
|
||||
usage_pct: 90,
|
||||
threshold_pct: 80,
|
||||
};
|
||||
|
||||
assert_eq!(read.reason(), "foreground_read_pressure");
|
||||
assert_eq!(write.reason(), "foreground_write_pressure");
|
||||
assert_eq!(repair.reason(), "foreground_pressure");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_is_disabled_when_both_thresholds_are_zero() {
|
||||
let snapshot = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Saturated, Some(8), Some(8)),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Saturated, Some(8), Some(8)),
|
||||
]);
|
||||
|
||||
assert_eq!(foreground_pressure(&snapshot, 0, 0), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_skips_only_the_class_whose_threshold_is_zero() {
|
||||
let snapshot = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(10), Some(10)),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(9), Some(10)),
|
||||
]);
|
||||
|
||||
assert_eq!(
|
||||
foreground_pressure(&snapshot, 0, 80),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundWrite,
|
||||
usage_pct: 90,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
assert_eq!(
|
||||
foreground_pressure(&snapshot, 80, 0),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundRead,
|
||||
usage_pct: 100,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_ignores_missing_entries() {
|
||||
let snapshot = registry(vec![counted(WorkloadClass::Scanner, AdmissionState::Saturated, Some(8), Some(8))]);
|
||||
|
||||
assert_eq!(foreground_pressure(&snapshot, 1, 1), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_ignores_missing_and_zero_limits() {
|
||||
let missing_limit = registry(vec![counted(
|
||||
WorkloadClass::ForegroundRead,
|
||||
AdmissionState::Throttled,
|
||||
Some(8),
|
||||
None,
|
||||
)]);
|
||||
let zero_limit = registry(vec![counted(
|
||||
WorkloadClass::ForegroundWrite,
|
||||
AdmissionState::Throttled,
|
||||
Some(8),
|
||||
Some(0),
|
||||
)]);
|
||||
|
||||
assert_eq!(foreground_pressure(&missing_limit, 1, 1), None);
|
||||
assert_eq!(foreground_pressure(&zero_limit, 1, 1), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_treats_saturated_as_full_without_reading_limit() {
|
||||
let snapshot = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Saturated, None, None),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Saturated, Some(0), Some(0)),
|
||||
]);
|
||||
|
||||
assert_eq!(
|
||||
foreground_pressure(&snapshot, 100, 0),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundRead,
|
||||
usage_pct: 100,
|
||||
threshold_pct: 100,
|
||||
})
|
||||
);
|
||||
assert_eq!(
|
||||
foreground_pressure(&snapshot, 0, 100),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundWrite,
|
||||
usage_pct: 100,
|
||||
threshold_pct: 100,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_reads_missing_active_as_zero() {
|
||||
let snapshot = registry(vec![counted(WorkloadClass::ForegroundRead, AdmissionState::Open, None, Some(8))]);
|
||||
|
||||
assert_eq!(foreground_pressure(&snapshot, 1, 1), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_returns_the_higher_utilization_when_both_classes_exceed() {
|
||||
let read_higher = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(19), Some(20)),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(17), Some(20)),
|
||||
]);
|
||||
let write_higher = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(17), Some(20)),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(19), Some(20)),
|
||||
]);
|
||||
|
||||
assert_eq!(
|
||||
foreground_pressure(&read_higher, 80, 80),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundRead,
|
||||
usage_pct: 95,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
assert_eq!(
|
||||
foreground_pressure(&write_higher, 80, 80),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundWrite,
|
||||
usage_pct: 95,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_breaks_utilization_ties_toward_the_write_class() {
|
||||
let snapshot = registry(vec![
|
||||
counted(WorkloadClass::ForegroundRead, AdmissionState::Open, Some(18), Some(20)),
|
||||
counted(WorkloadClass::ForegroundWrite, AdmissionState::Open, Some(18), Some(20)),
|
||||
]);
|
||||
|
||||
assert_eq!(
|
||||
foreground_pressure(&snapshot, 80, 80),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundWrite,
|
||||
usage_pct: 90,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_pressure_triggers_exactly_at_the_threshold_and_not_below() {
|
||||
let at_threshold = registry(vec![counted(
|
||||
WorkloadClass::ForegroundRead,
|
||||
AdmissionState::Open,
|
||||
Some(8),
|
||||
Some(10),
|
||||
)]);
|
||||
let below_threshold = registry(vec![counted(
|
||||
WorkloadClass::ForegroundRead,
|
||||
AdmissionState::Open,
|
||||
Some(7),
|
||||
Some(10),
|
||||
)]);
|
||||
|
||||
assert_eq!(
|
||||
foreground_pressure(&at_threshold, 80, 80),
|
||||
Some(ForegroundPressure {
|
||||
class: WorkloadClass::ForegroundRead,
|
||||
usage_pct: 80,
|
||||
threshold_pct: 80,
|
||||
})
|
||||
);
|
||||
assert_eq!(foreground_pressure(&below_threshold, 80, 80), None);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -40,14 +40,6 @@ pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
|
||||
pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
|
||||
pub const DEFAULT_HEALTH_CLUSTER_TIMEOUT_MS: u64 = 2000;
|
||||
|
||||
/// Timeout for one remote lock-client online check used by readiness (milliseconds).
|
||||
///
|
||||
/// This is intentionally shorter than the generic lock RPC timeout so
|
||||
/// `/health/ready` can report degradation instead of riding a dead peer's
|
||||
/// connect or HTTP/2 keepalive budget.
|
||||
pub const ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS: &str = "RUSTFS_HEALTH_LOCK_ONLINE_TIMEOUT_MS";
|
||||
pub const DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS: u64 = 1000;
|
||||
|
||||
/// Maximum time to wait for local node runtime readiness (storage / IAM / lock
|
||||
/// quorum) during startup before failing fast (seconds).
|
||||
///
|
||||
|
||||
@@ -288,49 +288,6 @@ pub const DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 0;
|
||||
|
||||
const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
|
||||
|
||||
/// Enable automatic foreground admission for large or unknown-size PutObject requests.
|
||||
///
|
||||
/// Unlike the strict experimental gate above, this default-on path only applies
|
||||
/// to requests that are large enough to create sustained erasure/RPC pressure.
|
||||
/// Small PUTs continue on the legacy path unless the strict gate is explicitly
|
||||
/// enabled.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: bool = true;
|
||||
|
||||
/// Maximum automatic foreground write requests admitted concurrently per process.
|
||||
///
|
||||
/// `0` derives a conservative default from the local disk-read scheduler cap,
|
||||
/// currently clamped to protect the commit path without making ordinary high
|
||||
/// throughput uploads single-file.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: usize = 0;
|
||||
|
||||
/// Minimum direct PutObject size that enters automatic foreground write admission.
|
||||
///
|
||||
/// Requests with an unknown size are treated as large because the write pressure
|
||||
/// cannot be bounded from headers.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 1024 * 1024;
|
||||
|
||||
/// Minimum UploadPart size that enters automatic foreground write admission.
|
||||
///
|
||||
/// Multipart pressure is often many moderate-sized parts rather than one very
|
||||
/// large request. The default gates every multipart part through the same permit
|
||||
/// pool as large/unknown-size PutObject while keeping small direct PUTs on the
|
||||
/// legacy path.
|
||||
pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str =
|
||||
"RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
|
||||
pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 0;
|
||||
|
||||
/// Time in milliseconds an automatic foreground write waits for a permit.
|
||||
///
|
||||
/// A short wait smooths transient bursts while still returning S3
|
||||
/// `SlowDown`/503 before body ingest when the node is already saturated.
|
||||
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
|
||||
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
|
||||
|
||||
const _: () = assert!(DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE);
|
||||
|
||||
/// Environment variable for minimum GetObject timeout in seconds.
|
||||
///
|
||||
/// When dynamic timeout calculation is enabled, this is the minimum timeout
|
||||
|
||||
@@ -31,9 +31,14 @@
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use crate::common::{RustFSTestEnvironment, init_logging, rustfs_binary_path};
|
||||
use aws_sdk_s3::Client;
|
||||
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client, rustfs_binary_path};
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::{Client, Config};
|
||||
use http::header::HOST;
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use s3s::Body;
|
||||
use std::error::Error;
|
||||
use std::io::Read;
|
||||
use std::process::{Command, Stdio};
|
||||
@@ -82,10 +87,10 @@ mod tests {
|
||||
}
|
||||
|
||||
/// Send a SigV4-signed request to `path` (optionally with a JSON `body`) and
|
||||
/// return `(status, body)`.
|
||||
///
|
||||
/// Thin wrapper over [`crate::common::admin_request`], kept local so the
|
||||
/// call sites below keep their `Option<&str>` body shape.
|
||||
/// return `(status, body)`. Uses the `UNSIGNED_PAYLOAD` content hash so a
|
||||
/// request body can be attached without the caller pre-hashing it — the
|
||||
/// server verifies the signature against the same sentinel, exactly as the
|
||||
/// AWS SDKs / MinIO client do for streaming/unsigned payloads.
|
||||
async fn signed_request(
|
||||
base_url: &str,
|
||||
method: http::Method,
|
||||
@@ -94,13 +99,47 @@ mod tests {
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(reqwest::StatusCode, String), Box<dyn Error + Send + Sync>> {
|
||||
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
|
||||
let url = format!("{base_url}{path}");
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("missing authority")?.to_string();
|
||||
let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
|
||||
|
||||
// The signature is computed over `UNSIGNED_PAYLOAD`, so the body bytes do
|
||||
// not participate in the SigV4 hash — sign over an empty body and attach
|
||||
// the real payload to the wire request below.
|
||||
let request = http::Request::builder()
|
||||
.method(method.clone())
|
||||
.uri(uri)
|
||||
.header(HOST, authority)
|
||||
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
|
||||
|
||||
let client = local_http_client();
|
||||
let mut rb = client.request(method, url.as_str());
|
||||
for (name, value) in signed.headers() {
|
||||
rb = rb.header(name, value);
|
||||
}
|
||||
if !body_bytes.is_empty() {
|
||||
rb = rb.body(body_bytes);
|
||||
}
|
||||
let resp = rb.send().await?;
|
||||
let status = resp.status();
|
||||
let text = resp.text().await?;
|
||||
Ok((status, text))
|
||||
}
|
||||
|
||||
/// Build an S3 client bound to explicit credentials (used to exercise the S3
|
||||
/// data plane with rotated / stale root credentials).
|
||||
fn s3_client_with(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
|
||||
env.create_s3_client_with_credentials(access_key, secret_key)
|
||||
let credentials = Credentials::new(access_key, secret_key, None, None, "sec4-admin-auth");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
Client::from_conf(config)
|
||||
}
|
||||
|
||||
/// Create a non-admin IAM user via the admin `add-user` API using the root
|
||||
@@ -112,7 +151,12 @@ mod tests {
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
crate::common::admin_create_user(env, access_key, secret_key).await
|
||||
let path = format!("/rustfs/admin/v3/add-user?accessKey={access_key}");
|
||||
let body = serde_json::json!({ "secretKey": secret_key, "status": "enabled" }).to_string();
|
||||
let (status, resp) =
|
||||
signed_request(&env.url, http::Method::PUT, &path, Some(&body), &env.access_key, &env.secret_key).await?;
|
||||
assert!(status.is_success(), "add-user should succeed (status={status}, body={resp})");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// A fully authenticated but non-admin credential must be rejected with
|
||||
|
||||
@@ -59,8 +59,8 @@ mod tests {
|
||||
|
||||
/// One signed admin request, returning the status and the raw body.
|
||||
///
|
||||
/// Thin wrapper over [`crate::common::admin_request`], kept local so the
|
||||
/// call sites below keep their `Option<&str>` body shape.
|
||||
/// Signs with `UNSIGNED_PAYLOAD` so the body does not participate in the
|
||||
/// hash, matching how the other admin e2e tests drive these routes.
|
||||
async fn signed_request(
|
||||
base_url: &str,
|
||||
method: http::Method,
|
||||
@@ -69,7 +69,30 @@ mod tests {
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(reqwest::StatusCode, String), Box<dyn Error + Send + Sync>> {
|
||||
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
|
||||
let url = format!("{base_url}{path}");
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("missing authority")?.to_string();
|
||||
let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
|
||||
|
||||
let request = http::Request::builder()
|
||||
.method(method.clone())
|
||||
.uri(uri)
|
||||
.header(HOST, authority)
|
||||
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
|
||||
|
||||
let client = local_http_client();
|
||||
let mut builder = client.request(method, url.as_str());
|
||||
for (name, value) in signed.headers() {
|
||||
builder = builder.header(name, value);
|
||||
}
|
||||
if !body_bytes.is_empty() {
|
||||
builder = builder.body(body_bytes);
|
||||
}
|
||||
let response = builder.send().await?;
|
||||
let status = response.status();
|
||||
let text = response.text().await?;
|
||||
Ok((status, text))
|
||||
}
|
||||
|
||||
/// A SigV4-signed `AssumeRole` form POST, optionally carrying a second factor.
|
||||
|
||||
@@ -15,23 +15,39 @@
|
||||
//! Regression test for Issue #1423
|
||||
//! Verifies that Bucket Policies are honored for Authenticated Users.
|
||||
|
||||
use crate::common::{AdminTransport, RustFSTestEnvironment, admin_create_user_via, init_logging};
|
||||
use aws_sdk_s3::Client;
|
||||
use crate::common::{RustFSTestEnvironment, init_logging};
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::{Client, Config};
|
||||
use tracing::info;
|
||||
|
||||
/// This suite deliberately drives the admin API through the external `awscurl`
|
||||
/// binary, so user creation pins `AdminTransport::Awscurl`.
|
||||
async fn create_user(
|
||||
env: &RustFSTestEnvironment,
|
||||
username: &str,
|
||||
password: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
admin_create_user_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, username, password).await
|
||||
let create_user_body = serde_json::json!({
|
||||
"secretKey": password,
|
||||
"status": "enabled"
|
||||
})
|
||||
.to_string();
|
||||
|
||||
let create_user_url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
|
||||
crate::common::awscurl_put(&create_user_url, &create_user_body, &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn create_user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
|
||||
env.create_s3_client_with_credentials(access_key, secret_key)
|
||||
let credentials = Credentials::new(access_key, secret_key, None, None, "test-user");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
|
||||
Client::from_conf(config)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
+28
-169
@@ -217,37 +217,7 @@ pub(crate) async fn signed_s3_request(
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
|
||||
signed_s3_request_with_headers(method, url, body, content_type, access_key, secret_key, &http::HeaderMap::new()).await
|
||||
}
|
||||
|
||||
pub(crate) async fn signed_s3_request_with_headers(
|
||||
method: http::Method,
|
||||
url: &str,
|
||||
body: Option<String>,
|
||||
content_type: Option<&str>,
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
extra_headers: &http::HeaderMap,
|
||||
) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
|
||||
signed_s3_request_with_session_token(
|
||||
method,
|
||||
url,
|
||||
body,
|
||||
content_type,
|
||||
SigningCredentials {
|
||||
access_key,
|
||||
secret_key,
|
||||
session_token: None,
|
||||
},
|
||||
extra_headers,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
struct SigningCredentials<'a> {
|
||||
access_key: &'a str,
|
||||
secret_key: &'a str,
|
||||
session_token: Option<&'a str>,
|
||||
signed_s3_request_with_session_token(method, url, body, content_type, access_key, secret_key, None).await
|
||||
}
|
||||
|
||||
async fn signed_s3_request_with_session_token(
|
||||
@@ -255,8 +225,9 @@ async fn signed_s3_request_with_session_token(
|
||||
url: &str,
|
||||
body: Option<String>,
|
||||
content_type: Option<&str>,
|
||||
credentials: SigningCredentials<'_>,
|
||||
extra_headers: &http::HeaderMap,
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
session_token: Option<&str>,
|
||||
) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("S3 URL missing authority")?.to_string();
|
||||
@@ -268,17 +239,14 @@ async fn signed_s3_request_with_session_token(
|
||||
if let Some(content_type) = content_type {
|
||||
request = request.header(CONTENT_TYPE, content_type);
|
||||
}
|
||||
for (name, value) in extra_headers {
|
||||
request = request.header(name, value);
|
||||
}
|
||||
|
||||
let content_length = i64::try_from(body.as_ref().map_or(0, String::len)).map_err(|_| "S3 request body is too large")?;
|
||||
let signed = sign_v4(
|
||||
request.body(Body::empty())?,
|
||||
content_length,
|
||||
credentials.access_key,
|
||||
credentials.secret_key,
|
||||
credentials.session_token.unwrap_or_default(),
|
||||
access_key,
|
||||
secret_key,
|
||||
session_token.unwrap_or_default(),
|
||||
"us-east-1",
|
||||
);
|
||||
|
||||
@@ -315,19 +283,8 @@ pub(crate) async fn admin_request_with_session_token(
|
||||
) -> Result<(StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let url = format!("{base_url}{path_and_query}");
|
||||
let content_type = body.as_ref().map(|_| "application/json");
|
||||
let response = signed_s3_request_with_session_token(
|
||||
method,
|
||||
&url,
|
||||
body,
|
||||
content_type,
|
||||
SigningCredentials {
|
||||
access_key,
|
||||
secret_key,
|
||||
session_token,
|
||||
},
|
||||
&http::HeaderMap::new(),
|
||||
)
|
||||
.await?;
|
||||
let response =
|
||||
signed_s3_request_with_session_token(method, &url, body, content_type, access_key, secret_key, session_token).await?;
|
||||
let status = response.status();
|
||||
let body = response.text().await?;
|
||||
Ok((status, body))
|
||||
@@ -1787,128 +1744,30 @@ pub(crate) async fn admin_create_user(
|
||||
username: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
admin_create_user_via(AdminTransport::Signed, &env.url, &env.access_key, &env.secret_key, username, secret_key).await
|
||||
}
|
||||
let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
|
||||
let body = serde_json::json!({
|
||||
"secretKey": secret_key,
|
||||
"status": "enabled"
|
||||
});
|
||||
let response = signed_request(
|
||||
http::Method::PUT,
|
||||
&url,
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
Some(body.to_string().into_bytes()),
|
||||
Some("application/json"),
|
||||
)
|
||||
.await?;
|
||||
|
||||
/// Transport used by the shared admin-API helpers: in-process SigV4 signing
|
||||
/// via [`signed_request`], or the external `awscurl` binary (an independent
|
||||
/// SigV4 implementation exercised by the awscurl-gated suites).
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum AdminTransport {
|
||||
Signed,
|
||||
Awscurl,
|
||||
}
|
||||
|
||||
/// Execute an admin-API request against `base_url` with admin credentials over
|
||||
/// the chosen transport, failing on any non-success response.
|
||||
pub(crate) async fn admin_execute_at(
|
||||
transport: AdminTransport,
|
||||
method: http::Method,
|
||||
base_url: &str,
|
||||
admin_access_key: &str,
|
||||
admin_secret_key: &str,
|
||||
path_and_query: &str,
|
||||
body: Option<&str>,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let url = format!("{base_url}{path_and_query}");
|
||||
match transport {
|
||||
AdminTransport::Signed => {
|
||||
let content_type = match body {
|
||||
Some(body) if !body.is_empty() => Some("application/json"),
|
||||
_ => None,
|
||||
};
|
||||
let response = signed_request(
|
||||
method.clone(),
|
||||
&url,
|
||||
admin_access_key,
|
||||
admin_secret_key,
|
||||
body.map(|body| body.as_bytes().to_vec()),
|
||||
content_type,
|
||||
)
|
||||
.await?;
|
||||
if !response.status().is_success() {
|
||||
let status = response.status();
|
||||
let text = response.text().await.unwrap_or_default();
|
||||
return Err(format!("{method} {path_and_query} failed: {status} {text}").into());
|
||||
}
|
||||
}
|
||||
AdminTransport::Awscurl => {
|
||||
execute_awscurl(&url, method.as_str(), body, admin_access_key, admin_secret_key).await?;
|
||||
}
|
||||
if response.status() != reqwest::StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
return Err(format!("create user failed: {status} {body}").into());
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Create a new IAM user via the admin API over the chosen transport.
|
||||
pub(crate) async fn admin_create_user_via(
|
||||
transport: AdminTransport,
|
||||
base_url: &str,
|
||||
admin_access_key: &str,
|
||||
admin_secret_key: &str,
|
||||
username: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let path = format!("/rustfs/admin/v3/add-user?accessKey={username}");
|
||||
let body = serde_json::json!({"secretKey": secret_key, "status": "enabled"}).to_string();
|
||||
admin_execute_at(
|
||||
transport,
|
||||
http::Method::PUT,
|
||||
base_url,
|
||||
admin_access_key,
|
||||
admin_secret_key,
|
||||
&path,
|
||||
Some(&body),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Install a canned policy via the admin API over the chosen transport.
|
||||
pub(crate) async fn admin_add_canned_policy_via(
|
||||
transport: AdminTransport,
|
||||
base_url: &str,
|
||||
admin_access_key: &str,
|
||||
admin_secret_key: &str,
|
||||
policy_name: &str,
|
||||
policy_json: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let path = format!("/rustfs/admin/v3/add-canned-policy?name={policy_name}");
|
||||
admin_execute_at(
|
||||
transport,
|
||||
http::Method::PUT,
|
||||
base_url,
|
||||
admin_access_key,
|
||||
admin_secret_key,
|
||||
&path,
|
||||
Some(policy_json),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Attach a canned policy to a user via the admin API over the chosen transport.
|
||||
pub(crate) async fn admin_attach_user_policy_via(
|
||||
transport: AdminTransport,
|
||||
base_url: &str,
|
||||
admin_access_key: &str,
|
||||
admin_secret_key: &str,
|
||||
policy_name: &str,
|
||||
username: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
let path = format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={username}&isGroup=false");
|
||||
// `Some("")` preserves the historical wire shape on both transports: awscurl
|
||||
// keeps sending `-d ''` and the signed path attaches an empty body with no
|
||||
// content type.
|
||||
admin_execute_at(
|
||||
transport,
|
||||
http::Method::PUT,
|
||||
base_url,
|
||||
admin_access_key,
|
||||
admin_secret_key,
|
||||
&path,
|
||||
Some(""),
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
@@ -16,29 +16,37 @@
|
||||
//! session policy** (`Policy` parameter) via `awscurl --service sts` with explicit
|
||||
//! `Content-Type: application/x-www-form-urlencoded` on `POST /`.
|
||||
|
||||
use crate::common::{
|
||||
AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via,
|
||||
awscurl_delete, awscurl_post_sts_form_urlencoded, build_test_s3_config, init_logging,
|
||||
};
|
||||
use aws_sdk_s3::Client;
|
||||
use crate::common::{RustFSTestEnvironment, awscurl_delete, awscurl_post_sts_form_urlencoded, awscurl_put, init_logging};
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::{Delete, ObjectIdentifier, Tag, Tagging};
|
||||
use aws_sdk_s3::{Client, Config};
|
||||
use tracing::info;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
|
||||
env.create_s3_client_with_credentials(access_key, secret_key)
|
||||
let credentials = Credentials::new(access_key, secret_key, None, None, "e2e-existing-tag");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
Client::from_conf(config)
|
||||
}
|
||||
|
||||
fn sts_session_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str, session_token: &str) -> Client {
|
||||
Client::from_conf(build_test_s3_config(
|
||||
&env.url,
|
||||
access_key,
|
||||
secret_key,
|
||||
Some(session_token),
|
||||
"e2e-sts-session",
|
||||
))
|
||||
let credentials = Credentials::new(access_key, secret_key, Some(session_token.into()), None, "e2e-sts-session");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
Client::from_conf(config)
|
||||
}
|
||||
|
||||
fn extract_xml_tag(xml: &str, tag: &str) -> Option<String> {
|
||||
@@ -69,16 +77,15 @@ async fn assume_role_with_session_policy(
|
||||
parse_assume_role_credentials(&xml)
|
||||
}
|
||||
|
||||
// This suite deliberately drives the admin API through the external `awscurl`
|
||||
// binary (an independent SigV4 implementation), so the wrappers below pin
|
||||
// `AdminTransport::Awscurl`.
|
||||
|
||||
async fn admin_create_user(
|
||||
env: &RustFSTestEnvironment,
|
||||
username: &str,
|
||||
password: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
admin_create_user_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, username, password).await
|
||||
let body = serde_json::json!({ "secretKey": password, "status": "enabled" }).to_string();
|
||||
let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
|
||||
awscurl_put(&url, &body, &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn admin_add_canned_policy(
|
||||
@@ -86,15 +93,9 @@ async fn admin_add_canned_policy(
|
||||
policy_name: &str,
|
||||
policy_json: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
admin_add_canned_policy_via(
|
||||
AdminTransport::Awscurl,
|
||||
&env.url,
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
policy_name,
|
||||
policy_json,
|
||||
)
|
||||
.await
|
||||
let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", env.url, policy_name);
|
||||
awscurl_put(&url, policy_json, &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn admin_attach_policy_to_user(
|
||||
@@ -102,7 +103,12 @@ async fn admin_attach_policy_to_user(
|
||||
policy_name: &str,
|
||||
username: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
admin_attach_user_policy_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, policy_name, username).await
|
||||
let url = format!(
|
||||
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
|
||||
env.url, policy_name, username
|
||||
);
|
||||
awscurl_put(&url, "", &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn admin_remove_user(env: &RustFSTestEnvironment, username: &str) {
|
||||
|
||||
@@ -15,11 +15,20 @@
|
||||
//! E2E tests for group management (fixes #2028).
|
||||
|
||||
use crate::common::{RustFSTestEnvironment, admin_ok, admin_request, init_logging};
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use aws_sdk_s3::{Client, Config};
|
||||
use tracing::info;
|
||||
|
||||
fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
|
||||
env.create_s3_client_with_credentials(access_key, secret_key)
|
||||
let credentials = Credentials::new(access_key, secret_key, None, None, "e2e-group-test");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
Client::from_conf(config)
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
|
||||
@@ -15,13 +15,13 @@
|
||||
//! Four-node EC regression gate for inline storage and the inline GET reader.
|
||||
//!
|
||||
//! The storage decision is based on shard bytes (256 KiB / 32 KiB objects for
|
||||
//! the default EC 2+2 geometry), and the GET fast path follows the persisted
|
||||
//! inline marker. A local OTLP/HTTP collector observes the existing reader-path
|
||||
//! counter without adding a scrape endpoint or production logging.
|
||||
//! the default EC 2+2 geometry), while the GET fast path has its own object-size
|
||||
//! limits (128 KiB / 16 KiB). A local OTLP/HTTP collector observes the existing
|
||||
//! reader-path counter without adding a scrape endpoint or production logging.
|
||||
//! One S3 GET can select readers on multiple EC nodes, so the counter tracks
|
||||
//! distributed reader selection rather than HTTP request count.
|
||||
|
||||
use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging};
|
||||
use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging, local_http_client};
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::{
|
||||
@@ -30,7 +30,7 @@ use aws_sdk_s3::types::{
|
||||
};
|
||||
use bytes::Bytes;
|
||||
use flate2::read::GzDecoder;
|
||||
use http::header::CONTENT_ENCODING;
|
||||
use http::header::{CONTENT_ENCODING, HOST};
|
||||
use http::{Method, Request, Response, StatusCode};
|
||||
use http_body_util::{BodyExt, Full};
|
||||
use hyper::body::Incoming;
|
||||
@@ -42,6 +42,9 @@ use opentelemetry_proto::tonic::metrics::v1::{
|
||||
Metric, NumberDataPoint, ResourceMetrics, ScopeMetrics, Sum, metric, number_data_point,
|
||||
};
|
||||
use prost::Message;
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use s3s::Body;
|
||||
use std::collections::BTreeMap;
|
||||
use std::convert::Infallible;
|
||||
use std::error::Error;
|
||||
@@ -89,7 +92,6 @@ const MPU_PART_1_SIZE: usize = 5 * 1024 * 1024;
|
||||
const MPU_PART_2_SIZE: usize = 16 * KIB;
|
||||
const TIER_BUCKET: &str = "inline-fallback-cold-tier";
|
||||
const TIER_PREFIX: &str = "tiered";
|
||||
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: &str = "RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT";
|
||||
const MSGPACK_FALLBACK_CONTROL_SERIES: [(&str, &str); 4] = [
|
||||
(FALLBACK_REQUEST_DIRECTION, "ReadMultipleReq"),
|
||||
(FALLBACK_RESPONSE_DIRECTION, "ReadMultipleResp"),
|
||||
@@ -792,12 +794,12 @@ fn metric_attribute(key: &str, value: &str) -> KeyValue {
|
||||
}
|
||||
|
||||
fn boundary_cases(state: VersionState) -> Vec<BoundaryCase> {
|
||||
let storage_limit = match state {
|
||||
VersionState::Enabled => 32 * KIB,
|
||||
VersionState::Unversioned => 256 * KIB,
|
||||
let (fast_limit, storage_limit) = match state {
|
||||
VersionState::Enabled => (16 * KIB, 32 * KIB),
|
||||
VersionState::Unversioned => (128 * KIB, 256 * KIB),
|
||||
// A suspended bucket stores its null version using the unversioned
|
||||
// shard threshold, while ObjectInfo keeps version-aware GET semantics.
|
||||
VersionState::Suspended => 256 * KIB,
|
||||
VersionState::Suspended => (16 * KIB, 256 * KIB),
|
||||
};
|
||||
let mut sizes = vec![0, 16 * KIB - 1, 16 * KIB, 16 * KIB + 1, 32 * KIB - 1, 32 * KIB, 32 * KIB + 1];
|
||||
if !matches!(state, VersionState::Enabled) {
|
||||
@@ -818,7 +820,7 @@ fn boundary_cases(state: VersionState) -> Vec<BoundaryCase> {
|
||||
stored_inline: size <= storage_limit,
|
||||
expected_reader_path: if size == 0 {
|
||||
EMPTY
|
||||
} else if size <= storage_limit {
|
||||
} else if size <= fast_limit {
|
||||
INLINE_DIRECT
|
||||
} else {
|
||||
LEGACY_DUPLEX
|
||||
@@ -1260,8 +1262,6 @@ async fn put_two_part_multipart(client: &Client, bucket: &str, key: &str) -> Tes
|
||||
Ok((body, part2, complete.e_tag().map(str::to_owned)))
|
||||
}
|
||||
|
||||
/// Thin wrapper over [`crate::common::admin_request`], kept local so the call
|
||||
/// sites below keep their `Option<&str>` body shape.
|
||||
async fn signed_admin_request(
|
||||
base_url: &str,
|
||||
method: Method,
|
||||
@@ -1270,7 +1270,30 @@ async fn signed_admin_request(
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> TestResult<(reqwest::StatusCode, String)> {
|
||||
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
|
||||
let url = format!("{base_url}{path}");
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
|
||||
let body_bytes = body.map(|value| value.as_bytes().to_vec()).unwrap_or_default();
|
||||
|
||||
let request = http::Request::builder()
|
||||
.method(method.clone())
|
||||
.uri(uri)
|
||||
.header(HOST, authority)
|
||||
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
|
||||
|
||||
let client = local_http_client();
|
||||
let mut request_builder = client.request(method, url.as_str());
|
||||
for (name, value) in signed.headers() {
|
||||
request_builder = request_builder.header(name, value);
|
||||
}
|
||||
if !body_bytes.is_empty() {
|
||||
request_builder = request_builder.body(body_bytes);
|
||||
}
|
||||
let response = request_builder.send().await?;
|
||||
let status = response.status();
|
||||
let text = response.text().await?;
|
||||
Ok((status, text))
|
||||
}
|
||||
|
||||
fn unique_tier_name() -> String {
|
||||
@@ -2101,7 +2124,6 @@ async fn four_node_add_tier_converges() -> TestResult {
|
||||
cold.create_s3_client().create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
hot.start().await?;
|
||||
|
||||
let tier_name = unique_tier_name();
|
||||
@@ -2120,7 +2142,6 @@ async fn four_node_add_tier_converges_after_offline_node_restart_without_second_
|
||||
cold.create_s3_client().create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
hot.start().await?;
|
||||
|
||||
let tier_name = unique_tier_name();
|
||||
@@ -2217,7 +2238,6 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
hot.set_env("RUSTFS_SCANNER_ENABLED", "false");
|
||||
hot.set_env("RUSTFS_SCANNER_CYCLE", "3600");
|
||||
hot.set_env("RUSTFS_MAX_TRANSITION_WORKERS", "1");
|
||||
@@ -2360,7 +2380,6 @@ async fn four_node_manual_transition_rollout_non_empty_restart_readback() -> Tes
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
hot.set_env("RUSTFS_SCANNER_ENABLED", "false");
|
||||
hot.set_env("RUSTFS_SCANNER_CYCLE", "3600");
|
||||
hot.set_env("RUSTFS_MAX_TRANSITION_WORKERS", "2");
|
||||
@@ -2465,7 +2484,6 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls_during_
|
||||
|
||||
let collector = OtlpMetricCollector::start().await?;
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
configure_mixed_msgpack_cluster(&mut hot, &collector)?;
|
||||
hot.set_env("RUSTFS_SCANNER_CYCLE", "1");
|
||||
hot.set_env("RUSTFS_ILM_PROCESS_TIME", "1");
|
||||
@@ -2577,7 +2595,6 @@ async fn four_node_transitioned_inline_fallback() -> TestResult {
|
||||
|
||||
let collector = OtlpMetricCollector::start().await?;
|
||||
let mut hot = RustFSTestClusterEnvironment::new(4).await?;
|
||||
hot.set_env(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV, "true");
|
||||
configure_reader_metric_cluster(&mut hot, &collector);
|
||||
hot.set_env("RUSTFS_SCANNER_CYCLE", "1");
|
||||
hot.set_env("RUSTFS_ILM_PROCESS_TIME", "1");
|
||||
|
||||
@@ -17,7 +17,6 @@
|
||||
|
||||
use super::common::{
|
||||
LocalKMSTestEnvironment, VAULT_KEY_NAME, VaultTestEnvironment, configure_kms, get_kms_status, kms_admin_request, start_kms,
|
||||
test_sse_kms_encryption,
|
||||
};
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::{BucketVersioningStatus, ServerSideEncryption, VersioningConfiguration};
|
||||
@@ -432,38 +431,6 @@ async fn test_configured_local_kms_admin_and_versioned_cleanup() -> TestResult {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_admin_configured_local_kms_is_restored_after_restart() -> TestResult {
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
env.base_env.start_rustfs_server(Vec::new()).await?;
|
||||
|
||||
let default_key_id = env.configure_local_kms().await?;
|
||||
start_kms(&env.base_env.url, &env.base_env.access_key, &env.base_env.secret_key).await?;
|
||||
|
||||
env.base_env.restart_server_preserving_data(Vec::new(), &[]).await?;
|
||||
assert_configured_status(
|
||||
&env.base_env.url,
|
||||
&env.base_env.access_key,
|
||||
&env.base_env.secret_key,
|
||||
"local",
|
||||
&default_key_id,
|
||||
)
|
||||
.await?;
|
||||
|
||||
let bucket = format!("kms-restart-{}", Uuid::new_v4());
|
||||
env.base_env.create_test_bucket(&bucket).await?;
|
||||
let client = env.base_env.create_s3_client();
|
||||
test_sse_kms_encryption(&client, &bucket).await?;
|
||||
client
|
||||
.delete_object()
|
||||
.bucket(&bucket)
|
||||
.key("test-sse-kms-object")
|
||||
.send()
|
||||
.await?;
|
||||
env.base_env.delete_test_bucket(&bucket).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_configured_vault_kms_admin_and_versioned_cleanup() -> TestResult {
|
||||
let mut env = VaultTestEnvironment::new().await?;
|
||||
|
||||
@@ -1,220 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Anonymous access to SSE-KMS objects under per-key authorization.
|
||||
//!
|
||||
//! Locks both halves of the anonymous contract decided in backlog#2028 (D4):
|
||||
//!
|
||||
//! - **Enforcement on**: anonymous requests hold no `kms` grants, so a public
|
||||
//! bucket policy does not let them read SSE-KMS objects or write through an
|
||||
//! SSE-KMS default-encryption rule. Both fail with `AccessDenied`.
|
||||
//! - **Enforcement off** (the default): bucket policy alone governs anonymous
|
||||
//! access, matching the pre-enforcement behavior — public SSE-KMS objects are
|
||||
//! decrypted and served, and anonymous writes are encrypted under the default
|
||||
//! key.
|
||||
//!
|
||||
//! The denial today is emergent — an empty-account principal falling through to
|
||||
//! the IAM default deny — so without this file a refactor of principal
|
||||
//! construction or policy evaluation could silently flip it. Each test carries a
|
||||
//! plaintext-object positive control: a denial proves nothing while the bucket
|
||||
//! policy has not propagated.
|
||||
|
||||
use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
|
||||
use crate::common::{init_logging, local_http_client};
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::{
|
||||
ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
|
||||
};
|
||||
use std::time::Duration;
|
||||
|
||||
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
|
||||
|
||||
const DEFAULT_KEY: &str = "kms-anon-default-key";
|
||||
const BUCKET: &str = "kms-anon-enforcement";
|
||||
const PLAIN_OBJECT: &str = "plain.txt";
|
||||
const ENCRYPTED_OBJECT: &str = "encrypted.txt";
|
||||
const PAYLOAD: &[u8] = b"kms anonymous enforcement payload";
|
||||
|
||||
/// How long a bucket policy change may take to reach the request path.
|
||||
const POLICY_PROPAGATION: Duration = Duration::from_secs(20);
|
||||
|
||||
/// Start a local-KMS server and build the public-bucket fixture.
|
||||
///
|
||||
/// The bucket holds a plaintext object (the positive control), an SSE-KMS
|
||||
/// object, an SSE-KMS default-encryption rule, and a bucket policy opening
|
||||
/// `GetObject`/`PutObject` to everyone. The enforcement switch defaults to off,
|
||||
/// so the enforcing case has to set it explicitly.
|
||||
async fn start_public_sse_kms_bucket(env: &mut LocalKMSTestEnvironment, enforce: bool) -> TestResult {
|
||||
create_key_with_specific_id(&env.kms_keys_dir, DEFAULT_KEY).await?;
|
||||
|
||||
let key_dir = env.kms_keys_dir.clone();
|
||||
let args = vec![
|
||||
"--kms-enable",
|
||||
"--kms-backend",
|
||||
"local",
|
||||
"--kms-key-dir",
|
||||
key_dir.as_str(),
|
||||
"--kms-default-key-id",
|
||||
DEFAULT_KEY,
|
||||
];
|
||||
let mut envs = vec![("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")];
|
||||
if enforce {
|
||||
envs.push(("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"));
|
||||
}
|
||||
env.base_env.start_rustfs_server_with_env(args, &envs).await?;
|
||||
env.base_env.create_test_bucket(BUCKET).await?;
|
||||
|
||||
let owner = env.base_env.create_s3_client();
|
||||
|
||||
owner
|
||||
.put_object()
|
||||
.bucket(BUCKET)
|
||||
.key(PLAIN_OBJECT)
|
||||
.body(ByteStream::from_static(PAYLOAD))
|
||||
.send()
|
||||
.await?;
|
||||
owner
|
||||
.put_object()
|
||||
.bucket(BUCKET)
|
||||
.key(ENCRYPTED_OBJECT)
|
||||
.body(ByteStream::from_static(PAYLOAD))
|
||||
.server_side_encryption(ServerSideEncryption::AwsKms)
|
||||
.ssekms_key_id(DEFAULT_KEY)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
let encryption_config = ServerSideEncryptionConfiguration::builder()
|
||||
.rules(
|
||||
ServerSideEncryptionRule::builder()
|
||||
.apply_server_side_encryption_by_default(
|
||||
ServerSideEncryptionByDefault::builder()
|
||||
.sse_algorithm(ServerSideEncryption::AwsKms)
|
||||
.kms_master_key_id(DEFAULT_KEY)
|
||||
.build()?,
|
||||
)
|
||||
.build(),
|
||||
)
|
||||
.build()?;
|
||||
owner
|
||||
.put_bucket_encryption()
|
||||
.bucket(BUCKET)
|
||||
.server_side_encryption_configuration(encryption_config)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
let policy = serde_json::json!({
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [{
|
||||
"Sid": "PublicReadWrite",
|
||||
"Effect": "Allow",
|
||||
"Principal": "*",
|
||||
"Action": ["s3:GetObject", "s3:PutObject"],
|
||||
"Resource": [format!("arn:aws:s3:::{BUCKET}/*")]
|
||||
}]
|
||||
})
|
||||
.to_string();
|
||||
owner.put_bucket_policy().bucket(BUCKET).policy(&policy).send().await?;
|
||||
let _ = owner.delete_public_access_block().bucket(BUCKET).send().await;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn object_url(env: &LocalKMSTestEnvironment, key: &str) -> String {
|
||||
format!("{}/{BUCKET}/{key}", env.base_env.url)
|
||||
}
|
||||
|
||||
async fn anonymous_get(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
|
||||
local_http_client().get(object_url(env, key)).send().await
|
||||
}
|
||||
|
||||
async fn anonymous_put(env: &LocalKMSTestEnvironment, key: &str) -> Result<reqwest::Response, reqwest::Error> {
|
||||
local_http_client().put(object_url(env, key)).body(PAYLOAD).send().await
|
||||
}
|
||||
|
||||
/// Retry the plaintext read until the public bucket policy is live.
|
||||
async fn wait_for_public_read(env: &LocalKMSTestEnvironment) -> TestResult {
|
||||
let deadline = tokio::time::Instant::now() + POLICY_PROPAGATION;
|
||||
loop {
|
||||
let status = anonymous_get(env, PLAIN_OBJECT).await?.status();
|
||||
if status.as_u16() == 200 {
|
||||
return Ok(());
|
||||
}
|
||||
if tokio::time::Instant::now() >= deadline {
|
||||
return Err(format!("positive control never became readable: anonymous GET {PLAIN_OBJECT} -> {status}").into());
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn assert_anonymous_denied(response: reqwest::Response, what: &str) -> TestResult {
|
||||
let status = response.status().as_u16();
|
||||
let body = response.text().await?;
|
||||
assert_eq!(status, 403, "{what} must be denied, got {status}: {body}");
|
||||
assert!(body.contains("AccessDenied"), "{what} must carry AccessDenied: {body}");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Enforcement on: a public bucket policy does not exempt anonymous requests
|
||||
/// from per-key authorization, on either the read or the default-encryption
|
||||
/// write path.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn anonymous_sse_kms_denied_under_enforcement() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
start_public_sse_kms_bucket(&mut env, true).await?;
|
||||
wait_for_public_read(&env).await?;
|
||||
|
||||
let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
|
||||
assert_anonymous_denied(read, "anonymous GET of an SSE-KMS object").await?;
|
||||
|
||||
let write = anonymous_put(&env, "anon-write.txt").await?;
|
||||
assert_anonymous_denied(write, "anonymous PUT through an SSE-KMS default-encryption rule").await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Enforcement off (the default): bucket policy alone governs anonymous access,
|
||||
/// and the default-encryption rule still encrypts anonymous writes.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn anonymous_sse_kms_governed_by_bucket_policy_without_enforcement() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
start_public_sse_kms_bucket(&mut env, false).await?;
|
||||
wait_for_public_read(&env).await?;
|
||||
|
||||
let read = anonymous_get(&env, ENCRYPTED_OBJECT).await?;
|
||||
assert_eq!(read.status().as_u16(), 200, "anonymous GET of a public SSE-KMS object must succeed");
|
||||
assert_eq!(read.bytes().await?.as_ref(), PAYLOAD, "the object must be served decrypted");
|
||||
|
||||
let write = anonymous_put(&env, "anon-write.txt").await?;
|
||||
assert_eq!(write.status().as_u16(), 200, "anonymous PUT to a public bucket must succeed");
|
||||
|
||||
let stored = env
|
||||
.base_env
|
||||
.create_s3_client()
|
||||
.head_object()
|
||||
.bucket(BUCKET)
|
||||
.key("anon-write.txt")
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(
|
||||
stored.server_side_encryption(),
|
||||
Some(&ServerSideEncryption::AwsKms),
|
||||
"the anonymous write must be encrypted by the bucket default rule"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -66,7 +66,6 @@ const SURVIVOR_KEY: &str = "keep/object.bin";
|
||||
const TIER_NAME: &str = "KMSCOLD";
|
||||
const TIER_BUCKET: &str = "kms-ilm-cold-tier";
|
||||
const TIER_PREFIX: &str = "tiered";
|
||||
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: (&str, &str) = ("RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT", "true");
|
||||
const TRANSITION_BUCKET: &str = "kms-ilm-transition";
|
||||
const TRANSITION_KEY: &str = "tier/object.bin";
|
||||
|
||||
@@ -81,7 +80,7 @@ const ILM_DEADLINE: StdDuration = StdDuration::from_secs(90);
|
||||
/// `--kms-default-key-id`, insecure dev defaults). The lifecycle env matches
|
||||
/// `reliant/lifecycle.rs::fast_lifecycle_env` plus `RUSTFS_ILM_DEBUG_DAY_SECS=2`,
|
||||
/// so a `Days=1` rule is due about two seconds after the write.
|
||||
async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
|
||||
async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment) -> TestResult {
|
||||
create_key_with_specific_id(&env.kms_keys_dir, SSE_KEY).await?;
|
||||
|
||||
let key_dir = env.kms_keys_dir.clone();
|
||||
@@ -95,14 +94,13 @@ async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment, extra_env
|
||||
SSE_KEY,
|
||||
];
|
||||
|
||||
let mut envs = vec![
|
||||
let envs = [
|
||||
("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true"),
|
||||
("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true"),
|
||||
("RUSTFS_SCANNER_CYCLE", "1"),
|
||||
("RUSTFS_ILM_PROCESS_TIME", "1"),
|
||||
("RUSTFS_ILM_DEBUG_DAY_SECS", "2"),
|
||||
];
|
||||
envs.extend_from_slice(extra_env);
|
||||
|
||||
env.base_env.start_rustfs_server_with_env(args, &envs).await?;
|
||||
Ok(())
|
||||
@@ -429,7 +427,7 @@ async fn ilm_expiration_on_sse_kms_bucket_under_enforcement() -> TestResult {
|
||||
init_logging();
|
||||
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
start_enforcing_ilm_server(&mut env, &[]).await?;
|
||||
start_enforcing_ilm_server(&mut env).await?;
|
||||
env.base_env.create_test_bucket(EXPIRY_BUCKET).await?;
|
||||
|
||||
let client = env.base_env.create_s3_client();
|
||||
@@ -501,7 +499,7 @@ async fn ilm_transition_on_sse_kms_bucket_under_enforcement_reads_back() -> Test
|
||||
|
||||
// Hot server: Local KMS + enforcement + accelerated lifecycle clock.
|
||||
let mut env = LocalKMSTestEnvironment::new().await?;
|
||||
start_enforcing_ilm_server(&mut env, &[ALLOW_LOOPBACK_TIER_ENDPOINT_ENV]).await?;
|
||||
start_enforcing_ilm_server(&mut env).await?;
|
||||
let hot_client = env.base_env.create_s3_client();
|
||||
|
||||
add_rustfs_tier(&env.base_env, &cold.base_env).await?;
|
||||
|
||||
@@ -57,12 +57,6 @@ mod copy_object_version_restore_sse_test;
|
||||
#[cfg(test)]
|
||||
mod configured_roundtrip_test;
|
||||
|
||||
#[cfg(test)]
|
||||
mod select_sse_response_test;
|
||||
|
||||
#[cfg(test)]
|
||||
mod kms_anonymous_enforcement_test;
|
||||
|
||||
#[cfg(test)]
|
||||
mod kms_authorization_negative_matrix_test;
|
||||
|
||||
|
||||
@@ -1,241 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! SelectObjectContent SSE response-header compatibility (backlog#1625).
|
||||
|
||||
use super::common::{LocalKMSTestEnvironment, sse_customer_key_md5_base64, start_kms};
|
||||
use crate::common::signed_s3_request_with_headers;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use aws_sdk_s3::types::ServerSideEncryption;
|
||||
use base64_simd::STANDARD as BASE64;
|
||||
use http::{HeaderMap, Method};
|
||||
use std::error::Error;
|
||||
use uuid::Uuid;
|
||||
|
||||
type TestResult<T = ()> = Result<T, Box<dyn Error + Send + Sync>>;
|
||||
|
||||
const CSV_BODY: &[u8] = b"name\nalice\n";
|
||||
const SELECT_BODY: &str = r#"<SelectObjectContentRequest xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
||||
<Expression>SELECT * FROM S3Object</Expression>
|
||||
<ExpressionType>SQL</ExpressionType>
|
||||
<InputSerialization><CSV><FileHeaderInfo>USE</FileHeaderInfo></CSV></InputSerialization>
|
||||
<OutputSerialization><CSV/></OutputSerialization>
|
||||
</SelectObjectContentRequest>"#;
|
||||
const KMS_CONTEXT: &str = "eyJ0ZW5hbnQiOiJzMy1zZWxlY3QifQ==";
|
||||
const SSE_ALGORITHM: &str = "x-amz-server-side-encryption";
|
||||
const SSE_KMS_KEY_ID: &str = "x-amz-server-side-encryption-aws-kms-key-id";
|
||||
const SSE_KMS_CONTEXT: &str = "x-amz-server-side-encryption-context";
|
||||
const SSE_C_ALGORITHM: &str = "x-amz-server-side-encryption-customer-algorithm";
|
||||
const SSE_C_KEY: &str = "x-amz-server-side-encryption-customer-key";
|
||||
const SSE_C_KEY_MD5: &str = "x-amz-server-side-encryption-customer-key-md5";
|
||||
const LOG_FLUSH_SENTINEL: &str = "select-sse-log-flush-sentinel.csv";
|
||||
|
||||
async fn raw_select(
|
||||
env: &crate::common::RustFSTestEnvironment,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
request_headers: &HeaderMap,
|
||||
) -> TestResult<reqwest::Response> {
|
||||
let url = format!("{}/{bucket}/{object}?select&select-type=2", env.url);
|
||||
signed_s3_request_with_headers(
|
||||
Method::POST,
|
||||
&url,
|
||||
Some(SELECT_BODY.to_string()),
|
||||
Some("application/xml"),
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
request_headers,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn assert_success_headers(response: reqwest::Response, expected: &[(&str, &str)], absent: &[&str]) -> TestResult {
|
||||
if response.status() != reqwest::StatusCode::OK {
|
||||
let status = response.status();
|
||||
let url = response.url().clone();
|
||||
let body = response.text().await?;
|
||||
panic!("Select request to {url} failed with {status}: {body}");
|
||||
}
|
||||
for (name, value) in expected {
|
||||
assert_eq!(response.headers().get(*name).and_then(|header| header.to_str().ok()), Some(*value));
|
||||
}
|
||||
for name in absent {
|
||||
assert!(response.headers().get(*name).is_none(), "successful Select response must omit {name}");
|
||||
}
|
||||
let body = response.bytes().await?;
|
||||
assert!(
|
||||
body.windows(b"alice".len()).any(|window| window == b"alice"),
|
||||
"successful Select response must contain a Records event with the selected row"
|
||||
);
|
||||
assert!(
|
||||
body.windows(b"End".len()).any(|window| window == b"End"),
|
||||
"successful Select response must contain the terminal End event"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn assert_pre_stream_failure(response: reqwest::Response) -> TestResult {
|
||||
assert_eq!(response.status(), reqwest::StatusCode::BAD_REQUEST);
|
||||
let body = response.text().await?;
|
||||
assert!(body.contains("<Error>"), "pre-stream failure must return an S3 XML error: {body}");
|
||||
assert!(
|
||||
body.contains("<Code>InvalidRequest</Code>"),
|
||||
"invalid SSE-C parameters must preserve the S3 error code: {body}"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn put_object(
|
||||
client: &aws_sdk_s3::Client,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
) -> aws_sdk_s3::operation::put_object::builders::PutObjectFluentBuilder {
|
||||
client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key(object)
|
||||
.body(ByteStream::from_static(CSV_BODY))
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn select_projects_encryption_headers_and_rejects_invalid_sse_c_before_streaming() -> TestResult {
|
||||
let mut kms = LocalKMSTestEnvironment::new().await?;
|
||||
let log_path = format!("{}/server.log", kms.base_env.temp_dir);
|
||||
kms.base_env.capture_log_path = Some(log_path.clone());
|
||||
kms.base_env
|
||||
.start_rustfs_server_with_env(Vec::new(), &[("RUST_LOG", "s3s=debug,rustfs=info")])
|
||||
.await?;
|
||||
let key_id = kms.configure_local_kms().await?;
|
||||
start_kms(&kms.base_env.url, &kms.base_env.access_key, &kms.base_env.secret_key).await?;
|
||||
|
||||
let client = kms.base_env.create_s3_client();
|
||||
let bucket = format!("select-sse-{}", Uuid::new_v4().simple());
|
||||
client.create_bucket().bucket(&bucket).send().await?;
|
||||
|
||||
put_object(&client, &bucket, "plain.csv").send().await?;
|
||||
put_object(&client, &bucket, "sse-s3.csv")
|
||||
.server_side_encryption(ServerSideEncryption::Aes256)
|
||||
.send()
|
||||
.await?;
|
||||
put_object(&client, &bucket, "sse-kms.csv")
|
||||
.server_side_encryption(ServerSideEncryption::AwsKms)
|
||||
.ssekms_key_id(&key_id)
|
||||
.ssekms_encryption_context(KMS_CONTEXT)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
let customer_key = "01234567890123456789012345678901";
|
||||
let customer_key_b64 = BASE64.encode_to_string(customer_key);
|
||||
let customer_key_md5 = sse_customer_key_md5_base64(customer_key);
|
||||
put_object(&client, &bucket, "sse-c.csv")
|
||||
.sse_customer_algorithm("AES256")
|
||||
.sse_customer_key(&customer_key_b64)
|
||||
.sse_customer_key_md5(&customer_key_md5)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
assert_success_headers(
|
||||
raw_select(&kms.base_env, &bucket, "plain.csv", &HeaderMap::new()).await?,
|
||||
&[],
|
||||
&[
|
||||
SSE_ALGORITHM,
|
||||
SSE_KMS_KEY_ID,
|
||||
SSE_KMS_CONTEXT,
|
||||
SSE_C_ALGORITHM,
|
||||
SSE_C_KEY,
|
||||
SSE_C_KEY_MD5,
|
||||
],
|
||||
)
|
||||
.await?;
|
||||
assert_success_headers(
|
||||
raw_select(&kms.base_env, &bucket, "sse-s3.csv", &HeaderMap::new()).await?,
|
||||
&[(SSE_ALGORITHM, "AES256")],
|
||||
&[SSE_KMS_KEY_ID, SSE_KMS_CONTEXT, SSE_C_ALGORITHM, SSE_C_KEY, SSE_C_KEY_MD5],
|
||||
)
|
||||
.await?;
|
||||
assert_success_headers(
|
||||
raw_select(&kms.base_env, &bucket, "sse-kms.csv", &HeaderMap::new()).await?,
|
||||
&[
|
||||
(SSE_ALGORITHM, "aws:kms"),
|
||||
(SSE_KMS_KEY_ID, &key_id),
|
||||
(SSE_KMS_CONTEXT, KMS_CONTEXT),
|
||||
],
|
||||
&[SSE_C_ALGORITHM, SSE_C_KEY, SSE_C_KEY_MD5],
|
||||
)
|
||||
.await?;
|
||||
|
||||
let mut sse_c_headers = HeaderMap::new();
|
||||
sse_c_headers.insert(SSE_C_ALGORITHM, "AES256".parse()?);
|
||||
sse_c_headers.insert(SSE_C_KEY, customer_key_b64.parse()?);
|
||||
sse_c_headers.insert(SSE_C_KEY_MD5, customer_key_md5.parse()?);
|
||||
assert_success_headers(
|
||||
raw_select(&kms.base_env, &bucket, "sse-c.csv", &sse_c_headers).await?,
|
||||
&[(SSE_C_ALGORITHM, "AES256"), (SSE_C_KEY_MD5, &customer_key_md5)],
|
||||
&[SSE_ALGORITHM, SSE_KMS_KEY_ID, SSE_KMS_CONTEXT, SSE_C_KEY],
|
||||
)
|
||||
.await?;
|
||||
|
||||
assert_pre_stream_failure(raw_select(&kms.base_env, &bucket, "sse-c.csv", &HeaderMap::new()).await?).await?;
|
||||
|
||||
let mut missing_algorithm_headers = HeaderMap::new();
|
||||
missing_algorithm_headers.insert(SSE_C_KEY, customer_key_b64.parse()?);
|
||||
missing_algorithm_headers.insert(SSE_C_KEY_MD5, customer_key_md5.parse()?);
|
||||
assert_pre_stream_failure(raw_select(&kms.base_env, &bucket, "sse-c.csv", &missing_algorithm_headers).await?).await?;
|
||||
|
||||
let mut wrong_algorithm_headers = sse_c_headers.clone();
|
||||
wrong_algorithm_headers.insert(SSE_C_ALGORITHM, "AES128".parse()?);
|
||||
assert_pre_stream_failure(raw_select(&kms.base_env, &bucket, "sse-c.csv", &wrong_algorithm_headers).await?).await?;
|
||||
|
||||
let wrong_md5 = sse_customer_key_md5_base64("99999999999999999999999999999999");
|
||||
let mut wrong_md5_headers = sse_c_headers.clone();
|
||||
wrong_md5_headers.insert(SSE_C_KEY_MD5, wrong_md5.parse()?);
|
||||
assert_pre_stream_failure(raw_select(&kms.base_env, &bucket, "sse-c.csv", &wrong_md5_headers).await?).await?;
|
||||
|
||||
let wrong_key = "99999999999999999999999999999999";
|
||||
let wrong_key_b64 = BASE64.encode_to_string(wrong_key);
|
||||
let mut wrong_key_headers = HeaderMap::new();
|
||||
wrong_key_headers.insert(SSE_C_ALGORITHM, "AES256".parse()?);
|
||||
wrong_key_headers.insert(SSE_C_KEY, wrong_key_b64.parse()?);
|
||||
wrong_key_headers.insert(SSE_C_KEY_MD5, wrong_md5.parse()?);
|
||||
assert_pre_stream_failure(raw_select(&kms.base_env, &bucket, "sse-c.csv", &wrong_key_headers).await?).await?;
|
||||
|
||||
put_object(&client, &bucket, LOG_FLUSH_SENTINEL).send().await?;
|
||||
assert_success_headers(
|
||||
raw_select(&kms.base_env, &bucket, LOG_FLUSH_SENTINEL, &HeaderMap::new()).await?,
|
||||
&[],
|
||||
&[
|
||||
SSE_ALGORITHM,
|
||||
SSE_KMS_KEY_ID,
|
||||
SSE_KMS_CONTEXT,
|
||||
SSE_C_ALGORITHM,
|
||||
SSE_C_KEY,
|
||||
SSE_C_KEY_MD5,
|
||||
],
|
||||
)
|
||||
.await?;
|
||||
let mut logs = String::new();
|
||||
for _ in 0..100 {
|
||||
logs = tokio::fs::read_to_string(&log_path).await?;
|
||||
if logs.contains(LOG_FLUSH_SENTINEL) {
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
|
||||
}
|
||||
assert!(logs.contains(LOG_FLUSH_SENTINEL), "timed out waiting for the log sink to flush");
|
||||
for secret in [customer_key, customer_key_b64.as_str(), wrong_key, wrong_key_b64.as_str()] {
|
||||
assert!(!logs.contains(secret), "Select request logging leaked SSE-C customer key material");
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -61,9 +61,6 @@ mod get_codec_streaming_compat_test;
|
||||
#[cfg(test)]
|
||||
mod version_id_regression_test;
|
||||
|
||||
#[cfg(test)]
|
||||
mod select_request_root_alias_test;
|
||||
|
||||
// Pinned previous-release -> current-build on-disk compatibility.
|
||||
#[cfg(test)]
|
||||
mod upgrade_compatibility_test;
|
||||
@@ -167,10 +164,6 @@ mod delete_objects_versioning_test;
|
||||
#[cfg(test)]
|
||||
mod delete_object_no_content_length_test;
|
||||
|
||||
// Regression test for signed empty PutObject requests without Content-Length.
|
||||
#[cfg(test)]
|
||||
mod put_object_no_content_length_test;
|
||||
|
||||
// Delete-marker visibility baseline for data-movement migration proof.
|
||||
#[cfg(test)]
|
||||
mod delete_marker_migration_semantics_test;
|
||||
|
||||
@@ -38,10 +38,14 @@ use aws_sdk_s3::types::{
|
||||
NotificationConfiguration, NotificationConfigurationFilter, ObjectIdentifier, QueueConfiguration, S3KeyFilter,
|
||||
VersioningConfiguration,
|
||||
};
|
||||
use http::header::{CONTENT_TYPE, HOST};
|
||||
use local_ip_address::local_ip;
|
||||
use reqwest::StatusCode;
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS;
|
||||
use rustfs_utils::http::headers::{AMZ_REQUEST_ID, REQUEST_ID_HEADER};
|
||||
use s3s::Body;
|
||||
use serde_json::Value;
|
||||
use std::error::Error;
|
||||
use std::io::Cursor;
|
||||
@@ -411,16 +415,42 @@ async fn collect_until(
|
||||
// Admin target configuration (signed admin HTTP)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Thin wrapper over [`crate::common::signed_request`] with this suite's
|
||||
/// root credentials; a `Some` body is always JSON here.
|
||||
async fn signed_admin_request(
|
||||
env: &RustFSTestEnvironment,
|
||||
method: http::Method,
|
||||
url: &str,
|
||||
body: Option<Vec<u8>>,
|
||||
) -> Result<reqwest::Response, BoxError> {
|
||||
let content_type = body.is_some().then_some("application/json");
|
||||
crate::common::signed_request(method, url, &env.access_key, &env.secret_key, body, content_type).await
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
|
||||
let mut builder = http::Request::builder()
|
||||
.method(method.clone())
|
||||
.uri(uri)
|
||||
.header(HOST, authority)
|
||||
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
if body.is_some() {
|
||||
builder = builder.header(CONTENT_TYPE, "application/json");
|
||||
}
|
||||
|
||||
let content_len = body.as_ref().map(|b| b.len() as i64).unwrap_or_default();
|
||||
let signed = sign_v4(
|
||||
builder.body(Body::empty())?,
|
||||
content_len,
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
"",
|
||||
"us-east-1",
|
||||
);
|
||||
|
||||
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
|
||||
let mut request = crate::common::local_http_client().request(reqwest_method, url);
|
||||
for (name, value) in signed.headers() {
|
||||
request = request.header(name, value);
|
||||
}
|
||||
if let Some(body) = body {
|
||||
request = request.body(body);
|
||||
}
|
||||
Ok(request.send().await?)
|
||||
}
|
||||
|
||||
async fn enable_notify_module(env: &RustFSTestEnvironment) -> TestResult {
|
||||
|
||||
@@ -15,24 +15,28 @@
|
||||
//! Tests for AWS IAM policy variables with single-value, multi-value, and nested scenarios
|
||||
|
||||
use crate::common::{
|
||||
AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via,
|
||||
awscurl_delete, awscurl_put, build_test_s3_config, build_test_sts_client, init_logging,
|
||||
RustFSTestEnvironment, awscurl_delete, awscurl_put, build_test_s3_config, build_test_sts_client, init_logging,
|
||||
};
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use tracing::info;
|
||||
|
||||
/// Helper function to create a regular user with given credentials.
|
||||
///
|
||||
/// This suite deliberately drives the admin API through the external `awscurl`
|
||||
/// binary, so the shared helpers are pinned to `AdminTransport::Awscurl`.
|
||||
/// Helper function to create a regular user with given credentials
|
||||
async fn create_user(
|
||||
env: &RustFSTestEnvironment,
|
||||
username: &str,
|
||||
password: &str,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
admin_create_user_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, username, password).await
|
||||
let create_user_body = serde_json::json!({
|
||||
"secretKey": password,
|
||||
"status": "enabled"
|
||||
})
|
||||
.to_string();
|
||||
|
||||
let create_user_url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
|
||||
awscurl_put(&create_user_url, &create_user_body, &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Helper function to create and attach a policy
|
||||
@@ -42,17 +46,18 @@ async fn create_and_attach_policy(
|
||||
username: &str,
|
||||
policy_document: serde_json::Value,
|
||||
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
|
||||
admin_add_canned_policy_via(
|
||||
AdminTransport::Awscurl,
|
||||
&env.url,
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
policy_name,
|
||||
&policy_document.to_string(),
|
||||
)
|
||||
.await?;
|
||||
admin_attach_user_policy_via(AdminTransport::Awscurl, &env.url, &env.access_key, &env.secret_key, policy_name, username)
|
||||
.await?;
|
||||
let policy_string = policy_document.to_string();
|
||||
|
||||
// Create policy
|
||||
let add_policy_url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", env.url, policy_name);
|
||||
awscurl_put(&add_policy_url, &policy_string, &env.access_key, &env.secret_key).await?;
|
||||
|
||||
// Attach policy to user
|
||||
let attach_policy_url = format!(
|
||||
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
|
||||
env.url, policy_name, username
|
||||
);
|
||||
awscurl_put(&attach_policy_url, "", &env.access_key, &env.secret_key).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -87,9 +87,7 @@ fn valid_config() -> PresigningConfig {
|
||||
}
|
||||
|
||||
/// Flip bytes inside the `X-Amz-Signature=` query value without changing its
|
||||
/// length, producing a structurally valid but incorrect signature. Every hex
|
||||
/// digit is replaced by its complement (15 - v), which has no fixed point, so
|
||||
/// the tamper changes the value no matter which digits the signature contains.
|
||||
/// length, producing a structurally valid but incorrect signature.
|
||||
fn tamper_signature(uri: &str) -> String {
|
||||
let marker = "X-Amz-Signature=";
|
||||
let idx = uri.find(marker).expect("presigned uri must carry X-Amz-Signature") + marker.len();
|
||||
@@ -98,9 +96,10 @@ fn tamper_signature(uri: &str) -> String {
|
||||
let (sig, tail) = rest.split_at(end);
|
||||
let tampered: String = sig
|
||||
.chars()
|
||||
.map(|c| {
|
||||
let v = c.to_digit(16).expect("X-Amz-Signature value must be hex");
|
||||
char::from_digit(15 - v, 16).expect("complement of a hex digit is a hex digit")
|
||||
.map(|c| match c {
|
||||
'0' => 'f',
|
||||
'a' => '0',
|
||||
other => other,
|
||||
})
|
||||
.collect();
|
||||
assert_ne!(sig, tampered, "tamper must actually change the signature hex");
|
||||
|
||||
@@ -31,11 +31,15 @@
|
||||
//!
|
||||
//! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx>
|
||||
|
||||
use crate::common::local_http_client;
|
||||
use crate::common::rustfs_binary_path_with_features;
|
||||
use crate::common::{AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via};
|
||||
use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
|
||||
use anyhow::Result;
|
||||
use http::header::{CONTENT_TYPE, HOST};
|
||||
use reqwest::Client;
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use s3s::Body;
|
||||
use tokio::process::Command;
|
||||
use tracing::info;
|
||||
|
||||
@@ -63,43 +67,92 @@ fn basic_auth_header_for(access_key: &str, secret_key: &str) -> String {
|
||||
format!("Basic {}", encoded)
|
||||
}
|
||||
|
||||
async fn admin_create_user(base_url: &str, username: &str, secret_key: &str) -> Result<()> {
|
||||
admin_create_user_via(
|
||||
AdminTransport::Signed,
|
||||
base_url,
|
||||
async fn signed_admin_request(
|
||||
method: http::Method,
|
||||
url: &str,
|
||||
body: Option<Vec<u8>>,
|
||||
content_type: Option<&str>,
|
||||
) -> Result<reqwest::Response> {
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri
|
||||
.authority()
|
||||
.ok_or_else(|| anyhow::anyhow!("request URL missing authority"))?
|
||||
.to_string();
|
||||
let mut request = http::Request::builder().method(method.clone()).uri(uri);
|
||||
request = request.header(HOST, authority);
|
||||
request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
if let Some(content_type) = content_type {
|
||||
request = request.header(CONTENT_TYPE, content_type);
|
||||
}
|
||||
|
||||
let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
|
||||
let signed = sign_v4(
|
||||
request.body(Body::empty())?,
|
||||
content_len,
|
||||
DEFAULT_ACCESS_KEY,
|
||||
DEFAULT_SECRET_KEY,
|
||||
username,
|
||||
secret_key,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!(e))
|
||||
"",
|
||||
"us-east-1",
|
||||
);
|
||||
|
||||
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
|
||||
let mut request_builder = local_http_client().request(reqwest_method, url);
|
||||
for (name, value) in signed.headers() {
|
||||
request_builder = request_builder.header(name, value);
|
||||
}
|
||||
if let Some(body) = body {
|
||||
request_builder = request_builder.body(body);
|
||||
}
|
||||
|
||||
Ok(request_builder.send().await?)
|
||||
}
|
||||
|
||||
async fn admin_create_user(base_url: &str, username: &str, secret_key: &str) -> Result<()> {
|
||||
let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", base_url, username);
|
||||
let body = serde_json::json!({
|
||||
"secretKey": secret_key,
|
||||
"status": "enabled"
|
||||
});
|
||||
let response =
|
||||
signed_admin_request(http::Method::PUT, &url, Some(body.to_string().into_bytes()), Some("application/json")).await?;
|
||||
|
||||
if response.status() != reqwest::StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
anyhow::bail!("create user failed: {status} {body}");
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn admin_add_canned_policy(base_url: &str, policy_name: &str, policy: &serde_json::Value) -> Result<()> {
|
||||
admin_add_canned_policy_via(
|
||||
AdminTransport::Signed,
|
||||
base_url,
|
||||
DEFAULT_ACCESS_KEY,
|
||||
DEFAULT_SECRET_KEY,
|
||||
policy_name,
|
||||
&policy.to_string(),
|
||||
)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!(e))
|
||||
let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", base_url, policy_name);
|
||||
let response =
|
||||
signed_admin_request(http::Method::PUT, &url, Some(policy.to_string().into_bytes()), Some("application/json")).await?;
|
||||
|
||||
if response.status() != reqwest::StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
anyhow::bail!("add canned policy failed: {status} {body}");
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn admin_attach_policy_to_user(base_url: &str, policy_name: &str, username: &str) -> Result<()> {
|
||||
admin_attach_user_policy_via(
|
||||
AdminTransport::Signed,
|
||||
base_url,
|
||||
DEFAULT_ACCESS_KEY,
|
||||
DEFAULT_SECRET_KEY,
|
||||
policy_name,
|
||||
username,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!(e))
|
||||
let url = format!(
|
||||
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
|
||||
base_url, policy_name, username
|
||||
);
|
||||
let response = signed_admin_request(http::Method::PUT, &url, Some(Vec::new()), None).await?;
|
||||
|
||||
if response.status() != reqwest::StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
anyhow::bail!("attach policy failed: {status} {body}");
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Test WebDAV: MKCOL (create bucket), PUT, GET, DELETE, PROPFIND operations
|
||||
|
||||
@@ -1,152 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Regression coverage for rustfs#6830: a signed empty `PutObject` request
|
||||
//! without `Content-Length` and without `Transfer-Encoding` is still a
|
||||
//! zero-length object upload.
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use crate::common::{RustFSTestEnvironment, init_logging};
|
||||
use http::header::{CONTENT_LENGTH, HOST, TRANSFER_ENCODING};
|
||||
use rustfs_signer::sign_v4;
|
||||
use rustfs_utils::hash::EMPTY_STRING_SHA256_HASH;
|
||||
use s3s::Body;
|
||||
use std::error::Error;
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::net::TcpStream;
|
||||
use tokio::time::{Duration, timeout};
|
||||
use tracing::info;
|
||||
|
||||
const RAW_RESPONSE_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
|
||||
fn parse_status(raw_response: &str) -> Option<u16> {
|
||||
raw_response.lines().next()?.split_whitespace().nth(1)?.parse().ok()
|
||||
}
|
||||
|
||||
async fn send_raw_signed_put(
|
||||
url: &str,
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
transfer_encoding: Option<&str>,
|
||||
raw_body: &[u8],
|
||||
) -> Result<String, Box<dyn Error + Send + Sync>> {
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
|
||||
let path_and_query = uri.path_and_query().ok_or("request URL missing path")?.as_str().to_string();
|
||||
|
||||
let mut request = http::Request::builder()
|
||||
.method(http::Method::PUT)
|
||||
.uri(uri)
|
||||
.header(HOST, authority.clone())
|
||||
.header("x-amz-content-sha256", EMPTY_STRING_SHA256_HASH);
|
||||
if let Some(value) = transfer_encoding {
|
||||
request = request.header(TRANSFER_ENCODING, value);
|
||||
}
|
||||
|
||||
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
|
||||
|
||||
let mut raw_request = format!("PUT {path_and_query} HTTP/1.1\r\nHost: {authority}\r\nConnection: close\r\n");
|
||||
for (name, value) in signed.headers() {
|
||||
if name == HOST || name == CONTENT_LENGTH {
|
||||
continue;
|
||||
}
|
||||
raw_request.push_str(name.as_str());
|
||||
raw_request.push_str(": ");
|
||||
raw_request.push_str(value.to_str()?);
|
||||
raw_request.push_str("\r\n");
|
||||
}
|
||||
raw_request.push_str("\r\n");
|
||||
|
||||
assert!(
|
||||
!raw_request.to_ascii_lowercase().contains("\r\ncontent-length:"),
|
||||
"raw regression request must omit Content-Length; request was:\n{raw_request}"
|
||||
);
|
||||
|
||||
let mut stream = TcpStream::connect(&authority).await?;
|
||||
stream.write_all(raw_request.as_bytes()).await?;
|
||||
stream.write_all(raw_body).await?;
|
||||
stream.flush().await?;
|
||||
|
||||
let mut response = Vec::new();
|
||||
timeout(RAW_RESPONSE_TIMEOUT, stream.read_to_end(&mut response))
|
||||
.await
|
||||
.map_err(|_| std::io::Error::new(std::io::ErrorKind::TimedOut, "timed out reading raw PUT response"))??;
|
||||
Ok(String::from_utf8_lossy(&response).into_owned())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_put_object_without_content_length_boundaries() -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
init_logging();
|
||||
info!("TEST: PutObject without Content-Length boundaries");
|
||||
|
||||
let mut env = RustFSTestEnvironment::new().await?;
|
||||
env.start_rustfs_server(vec![]).await?;
|
||||
|
||||
let client = env.create_s3_client();
|
||||
let empty_bucket = "put-no-content-length";
|
||||
let empty_key = "empty.bin";
|
||||
let chunked_bucket = "put-chunked-no-length";
|
||||
let chunked_key = "chunked.bin";
|
||||
|
||||
client.create_bucket().bucket(empty_bucket).send().await?;
|
||||
client.create_bucket().bucket(chunked_bucket).send().await?;
|
||||
|
||||
let url = format!("{}/{}/{}", env.url, empty_bucket, empty_key);
|
||||
let raw_response = send_raw_signed_put(&url, &env.access_key, &env.secret_key, None, b"").await?;
|
||||
info!("raw empty PUT response:\n{}", raw_response);
|
||||
|
||||
assert_eq!(
|
||||
parse_status(&raw_response),
|
||||
Some(200),
|
||||
"empty PutObject without Content-Length should succeed, got:\n{raw_response}"
|
||||
);
|
||||
assert!(
|
||||
raw_response.to_ascii_lowercase().contains("\r\netag:"),
|
||||
"successful PutObject should return an ETag header: {raw_response}"
|
||||
);
|
||||
|
||||
let head = client.head_object().bucket(empty_bucket).key(empty_key).send().await?;
|
||||
assert_eq!(head.content_length(), Some(0), "stored object must be zero length");
|
||||
|
||||
let url = format!("{}/{}/{}", env.url, chunked_bucket, chunked_key);
|
||||
let raw_response = send_raw_signed_put(&url, &env.access_key, &env.secret_key, Some("chunked"), b"0\r\n\r\n").await?;
|
||||
info!("raw chunked PUT response:\n{}", raw_response);
|
||||
|
||||
assert_eq!(
|
||||
parse_status(&raw_response),
|
||||
Some(411),
|
||||
"unknown-length chunked PutObject must stay rejected, got:\n{raw_response}"
|
||||
);
|
||||
assert!(
|
||||
raw_response.contains("<Code>MissingContentLength</Code>"),
|
||||
"expected MissingContentLength, got:\n{raw_response}"
|
||||
);
|
||||
|
||||
let missing = client
|
||||
.head_object()
|
||||
.bucket(chunked_bucket)
|
||||
.key(chunked_key)
|
||||
.send()
|
||||
.await
|
||||
.expect_err("rejected unknown-length PUT must not create an object");
|
||||
assert_eq!(
|
||||
missing.raw_response().map(|response| response.status().as_u16()),
|
||||
Some(404),
|
||||
"rejected unknown-length PUT absence probe must return HTTP 404, got {missing:?}"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -23,9 +23,9 @@
|
||||
//!
|
||||
//! There are no containers, no external S3 backend and no `awscurl`: the
|
||||
//! `AddTier` admin call is signed in-process with `rustfs_signer`, exactly like
|
||||
//! the other admin-API e2e suites in this crate. The source server uses the
|
||||
//! explicit test-only loopback opt-in to tier to `cold` over
|
||||
//! `http://127.0.0.1:<port>` while production keeps the SSRF guard enabled.
|
||||
//! the other admin-API e2e suites in this crate. The RustFS warm backend has no
|
||||
//! loopback/SSRF restriction (that guard is replication-only), so `hot` can tier
|
||||
//! to `cold` over `http://127.0.0.1:<port>`.
|
||||
//!
|
||||
//! The hermetic tests drive the transition and restore paths and pin the
|
||||
//! chains required by ilm-7 and the restore follow-up:
|
||||
@@ -46,7 +46,7 @@
|
||||
//! retry serves the object locally until expiry, and expiry leaves the
|
||||
//! remote object available for a second restore.
|
||||
|
||||
use crate::common::RustFSTestEnvironment;
|
||||
use crate::common::{RustFSTestEnvironment, local_http_client};
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
@@ -56,6 +56,10 @@ use aws_sdk_s3::types::{
|
||||
VersioningConfiguration,
|
||||
};
|
||||
use http::Method;
|
||||
use http::header::HOST;
|
||||
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
|
||||
use rustfs_signer::sign_v4;
|
||||
use s3s::Body;
|
||||
use serde::Deserialize;
|
||||
use std::time::{Duration as StdDuration, Instant};
|
||||
use time::{OffsetDateTime, format_description::well_known::Rfc3339};
|
||||
@@ -96,7 +100,6 @@ const MANUAL_ACTIVE_CANCEL_OBJECTS: usize = 512;
|
||||
const MANUAL_RESTART_CANCEL_OBJECTS: usize = 512;
|
||||
const MANUAL_ACTIVE_CANCEL_RUNNING_TIMEOUT: StdDuration = StdDuration::from_secs(15);
|
||||
const MANUAL_TRANSITION_CANCEL_BARRIER_ENV: &str = "RUSTFS_E2E_MANUAL_TRANSITION_CANCEL_BARRIER";
|
||||
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: (&str, &str) = ("RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT", "true");
|
||||
const MANUAL_ASYNC_CONFLICT_TERMINAL_TIMEOUT: StdDuration = StdDuration::from_secs(90);
|
||||
const MANUAL_RESTART_RECOVERY_TIMEOUT: StdDuration = StdDuration::from_secs(80);
|
||||
const OBJECT_KEY: &str = "tier/鲁A12345/report.bin";
|
||||
@@ -113,20 +116,6 @@ const HDR_SOURCE_REPLICATION_REQUEST: &str = "x-rustfs-source-replication-reques
|
||||
const HDR_SOURCE_MTIME: &str = "x-rustfs-source-mtime";
|
||||
const TIER_MUTATION_RECOVERY_CHANGED: &str = "Remote tier mutation recovery changed before publish";
|
||||
|
||||
async fn start_tier_source(hot: &mut RustFSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
|
||||
let mut env = Vec::with_capacity(extra_env.len() + 1);
|
||||
env.push(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV);
|
||||
env.extend_from_slice(extra_env);
|
||||
hot.start_rustfs_server_with_env(vec![], &env).await
|
||||
}
|
||||
|
||||
async fn restart_tier_source(hot: &mut RustFSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
|
||||
let mut env = Vec::with_capacity(extra_env.len() + 1);
|
||||
env.push(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV);
|
||||
env.extend_from_slice(extra_env);
|
||||
hot.restart_server_preserving_data(vec![], &env).await
|
||||
}
|
||||
|
||||
/// 5 MiB — the S3 minimum size for a non-final multipart part; the object's only
|
||||
/// internal part boundary sits at this offset.
|
||||
const PART0_SIZE: usize = 5 * 1024 * 1024;
|
||||
@@ -142,8 +131,9 @@ fn payload() -> Vec<u8> {
|
||||
|
||||
/// Sign and send an admin request in-process (no `awscurl`).
|
||||
///
|
||||
/// Thin wrapper over [`crate::common::admin_request`], kept local so the call
|
||||
/// sites below keep their `Option<&str>` body shape.
|
||||
/// Mirrors the shared admin-API e2e pattern: the SigV4 signature is computed
|
||||
/// over `UNSIGNED_PAYLOAD`, so the JSON body rides on the wire without being
|
||||
/// pre-hashed. Returns the response status and body text.
|
||||
async fn signed_admin_request(
|
||||
base_url: &str,
|
||||
method: Method,
|
||||
@@ -152,7 +142,30 @@ async fn signed_admin_request(
|
||||
access_key: &str,
|
||||
secret_key: &str,
|
||||
) -> Result<(reqwest::StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
|
||||
crate::common::admin_request(base_url, method, path, body.map(str::to_string), access_key, secret_key).await
|
||||
let url = format!("{base_url}{path}");
|
||||
let uri = url.parse::<http::Uri>()?;
|
||||
let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
|
||||
let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
|
||||
|
||||
let request = http::Request::builder()
|
||||
.method(method.clone())
|
||||
.uri(uri)
|
||||
.header(HOST, authority)
|
||||
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
|
||||
let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
|
||||
|
||||
let client = local_http_client();
|
||||
let mut request_builder = client.request(method, url.as_str());
|
||||
for (name, value) in signed.headers() {
|
||||
request_builder = request_builder.header(name, value);
|
||||
}
|
||||
if !body_bytes.is_empty() {
|
||||
request_builder = request_builder.body(body_bytes);
|
||||
}
|
||||
let response = request_builder.send().await?;
|
||||
let status = response.status();
|
||||
let text = response.text().await?;
|
||||
Ok((status, text))
|
||||
}
|
||||
|
||||
/// Wire `hot` -> `cold` as a `TierType::RustFS` remote tier via `AddTier`.
|
||||
@@ -210,27 +223,19 @@ async fn add_rustfs_tier(hot: &RustFSTestEnvironment, cold: &RustFSTestEnvironme
|
||||
}
|
||||
}
|
||||
|
||||
fn clear_tiers_confirmation_token(now: OffsetDateTime) -> String {
|
||||
let mut rand = "AGD1R25GI3I1GJGUGJFD7FBS4DFAASDF".to_string();
|
||||
rand.insert_str(3, &now.day().to_string());
|
||||
rand.insert_str(17, &now.month().to_string());
|
||||
rand.insert_str(23, &now.year().to_string());
|
||||
rand
|
||||
}
|
||||
|
||||
async fn clear_rustfs_tiers_force(hot: &RustFSTestEnvironment) -> TestResult {
|
||||
async fn remove_rustfs_tier_force(hot: &RustFSTestEnvironment) -> TestResult {
|
||||
let path = format!("/rustfs/admin/v3/tier/{TIER_NAME}?force=true");
|
||||
let deadline = Instant::now() + StdDuration::from_secs(30);
|
||||
loop {
|
||||
let rand = clear_tiers_confirmation_token(OffsetDateTime::now_utc());
|
||||
let path = format!("/rustfs/admin/v3/tier/clear?rand={rand}&force=true");
|
||||
let (status, resp) = signed_admin_request(&hot.url, Method::POST, &path, None, &hot.access_key, &hot.secret_key).await?;
|
||||
let (status, resp) =
|
||||
signed_admin_request(&hot.url, Method::DELETE, &path, None, &hot.access_key, &hot.secret_key).await?;
|
||||
if status.is_success() {
|
||||
return Ok(());
|
||||
}
|
||||
if (!resp.contains("TierNameBackendInUse") && !resp.contains(TIER_MUTATION_RECOVERY_CHANGED))
|
||||
|| Instant::now() >= deadline
|
||||
{
|
||||
return Err(format!("ClearTier(RustFS) failed: status={status}, body={resp}").into());
|
||||
return Err(format!("RemoveTier(RustFS) failed: status={status}, body={resp}").into());
|
||||
}
|
||||
// Tier mutation cleanup and startup recovery are asynchronous.
|
||||
tokio::time::sleep(StdDuration::from_millis(100)).await;
|
||||
@@ -883,7 +888,8 @@ async fn test_hermetic_transition_main_path() -> TestResult {
|
||||
// Hot/source server. A 1s scanner cycle is a backstop; transition is
|
||||
// primarily driven immediately by the multipart completion path.
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_CYCLE", "1")]).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_CYCLE", "1")])
|
||||
.await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
|
||||
// Wire the RustFS remote tier (real connectivity probe, no force).
|
||||
@@ -981,7 +987,8 @@ async fn test_hermetic_transition_restore_failure_expiry_and_retry() -> TestResu
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_ILM_DEBUG_DAY_SECS", "5")]).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_ILM_DEBUG_DAY_SECS", "5")])
|
||||
.await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -1109,7 +1116,8 @@ async fn test_manual_transition_run_black_box_semantics() -> TestResult {
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
let due_mtime = OffsetDateTime::now_utc() - time::Duration::hours(25);
|
||||
@@ -1214,7 +1222,8 @@ async fn test_manual_transition_async_job_status_polling() -> TestResult {
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -1312,7 +1321,8 @@ async fn test_manual_transition_async_limit_reports_terminal_partial() -> TestRe
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -1473,8 +1483,8 @@ async fn test_manual_transition_async_scope_conflicts_report_active_job() -> Tes
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(
|
||||
&mut hot,
|
||||
hot.start_rustfs_server_with_env(
|
||||
vec![],
|
||||
&[
|
||||
("RUSTFS_SCANNER_ENABLED", "false"),
|
||||
("RUSTFS_SCANNER_CYCLE", "3600"),
|
||||
@@ -1583,7 +1593,8 @@ async fn test_manual_transition_async_different_buckets_admit_concurrently() ->
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -1703,7 +1714,8 @@ async fn test_manual_transition_async_tier_failure_reports_terminal_partial() ->
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -1716,7 +1728,7 @@ async fn test_manual_transition_async_tier_failure_reports_terminal_partial() ->
|
||||
0,
|
||||
)
|
||||
.await?;
|
||||
clear_rustfs_tiers_force(&hot).await?;
|
||||
remove_rustfs_tier_force(&hot).await?;
|
||||
|
||||
let due_mtime = OffsetDateTime::now_utc() - time::Duration::hours(25);
|
||||
put_backdated_single_part_object(
|
||||
@@ -1795,7 +1807,8 @@ async fn test_manual_transition_async_worker_failure_reports_terminal_partial()
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
cold.stop_server();
|
||||
@@ -1887,8 +1900,8 @@ async fn test_manual_transition_async_active_cancel_reports_terminal_cancelled()
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(
|
||||
&mut hot,
|
||||
hot.start_rustfs_server_with_env(
|
||||
vec![],
|
||||
&[
|
||||
("RUSTFS_SCANNER_ENABLED", "false"),
|
||||
("RUSTFS_SCANNER_CYCLE", "3600"),
|
||||
@@ -1993,7 +2006,7 @@ async fn test_manual_transition_async_cancel_after_process_restart_recovers_term
|
||||
("RUSTFS_TRANSITION_QUEUE_CAPACITY", "512"),
|
||||
];
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &restart_env).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &restart_env).await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -2021,7 +2034,7 @@ async fn test_manual_transition_async_cancel_after_process_restart_recovers_term
|
||||
.ok_or("async response must include status_endpoint")?;
|
||||
assert_eq!(accepted.cancel_endpoint.as_deref(), Some(status_endpoint));
|
||||
|
||||
restart_tier_source(&mut hot, &restart_env).await?;
|
||||
hot.restart_server_preserving_data(vec![], &restart_env).await?;
|
||||
|
||||
let restarted = manual_transition_job_status(&hot, status_endpoint).await?;
|
||||
assert_eq!(restarted.job_id, job_id);
|
||||
@@ -2145,7 +2158,8 @@ async fn test_manual_transition_run_contract_no_status_cancel_fields() -> TestRe
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -2183,7 +2197,8 @@ async fn test_manual_transition_run_continuation_token_resumes_without_raw_marke
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
let hot_client = hot.create_s3_client();
|
||||
add_rustfs_tier(&hot, &cold).await?;
|
||||
|
||||
@@ -2223,7 +2238,8 @@ async fn test_manual_transition_run_continuation_token_resumes_without_raw_marke
|
||||
"continuation token must not expose the raw object prefix: {continuation}"
|
||||
);
|
||||
|
||||
restart_tier_source(&mut hot, &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")]).await?;
|
||||
hot.restart_server_preserving_data(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
|
||||
.await?;
|
||||
|
||||
let second = manual_transition_run_with_max_and_continuation(
|
||||
&hot,
|
||||
@@ -2256,8 +2272,8 @@ async fn test_manual_transition_run_queue_pressure_partial() -> TestResult {
|
||||
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
|
||||
|
||||
let mut hot = RustFSTestEnvironment::new().await?;
|
||||
start_tier_source(
|
||||
&mut hot,
|
||||
hot.start_rustfs_server_with_env(
|
||||
vec![],
|
||||
&[
|
||||
("RUSTFS_SCANNER_ENABLED", "false"),
|
||||
("RUSTFS_SCANNER_CYCLE", "3600"),
|
||||
|
||||
@@ -13,9 +13,8 @@
|
||||
// limitations under the License.
|
||||
|
||||
use crate::common::{
|
||||
AdminTransport, RustFSTestEnvironment, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user,
|
||||
awscurl_post_sts_form_urlencoded, init_logging, local_http_client, replication_fast_env, rustfs_binary_path, signed_request,
|
||||
signed_request_with_client, signed_request_with_session_token,
|
||||
RustFSTestEnvironment, admin_create_user, awscurl_post_sts_form_urlencoded, init_logging, local_http_client,
|
||||
replication_fast_env, rustfs_binary_path, signed_request, signed_request_with_client, signed_request_with_session_token,
|
||||
};
|
||||
use crate::fake_s3_target::{
|
||||
FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction as FakeTargetFault, Operation as FakeTargetOperation,
|
||||
@@ -26,7 +25,7 @@ use crate::kms::common::{
|
||||
sse_customer_key_md5_base64,
|
||||
};
|
||||
use crate::storage_api::replication_extension::BucketTargetSys;
|
||||
use aws_sdk_s3::Client;
|
||||
use aws_sdk_s3::config::{Credentials, Region};
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
use aws_sdk_s3::operation::list_object_versions::ListObjectVersionsOutput;
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
@@ -34,6 +33,7 @@ use aws_sdk_s3::types::{
|
||||
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, DeleteMarkerEntry, ObjectVersion, ServerSideEncryption,
|
||||
VersioningConfiguration,
|
||||
};
|
||||
use aws_sdk_s3::{Client, Config};
|
||||
use base64_simd::STANDARD as BASE64_STANDARD;
|
||||
use bytes::Bytes;
|
||||
use flate2::read::GzDecoder;
|
||||
@@ -895,7 +895,15 @@ async fn wait_for_replicated_object_over_https(
|
||||
}
|
||||
|
||||
fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
|
||||
env.create_s3_client_with_credentials(access_key, secret_key)
|
||||
let credentials = Credentials::new(access_key, secret_key, None, None, "e2e-site-replication");
|
||||
let config = Config::builder()
|
||||
.credentials_provider(credentials)
|
||||
.region(Region::new("us-east-1"))
|
||||
.endpoint_url(&env.url)
|
||||
.force_path_style(true)
|
||||
.behavior_version_latest()
|
||||
.build();
|
||||
Client::from_conf(config)
|
||||
}
|
||||
|
||||
async fn admin_add_canned_policy(
|
||||
@@ -903,15 +911,24 @@ async fn admin_add_canned_policy(
|
||||
policy_name: &str,
|
||||
policy: &serde_json::Value,
|
||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
admin_add_canned_policy_via(
|
||||
AdminTransport::Signed,
|
||||
&env.url,
|
||||
let url = format!("{}/rustfs/admin/v3/add-canned-policy?name={}", env.url, policy_name);
|
||||
let response = signed_request(
|
||||
http::Method::PUT,
|
||||
&url,
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
policy_name,
|
||||
&policy.to_string(),
|
||||
Some(policy.to_string().into_bytes()),
|
||||
Some("application/json"),
|
||||
)
|
||||
.await
|
||||
.await?;
|
||||
|
||||
if response.status() != StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
return Err(format!("add canned policy failed: {status} {body}").into());
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn admin_attach_policy_to_user(
|
||||
@@ -919,7 +936,19 @@ async fn admin_attach_policy_to_user(
|
||||
policy_name: &str,
|
||||
username: &str,
|
||||
) -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
admin_attach_user_policy_via(AdminTransport::Signed, &env.url, &env.access_key, &env.secret_key, policy_name, username).await
|
||||
let url = format!(
|
||||
"{}/rustfs/admin/v3/set-user-or-group-policy?policyName={}&userOrGroup={}&isGroup=false",
|
||||
env.url, policy_name, username
|
||||
);
|
||||
let response = signed_request(http::Method::PUT, &url, &env.access_key, &env.secret_key, Some(Vec::new()), None).await?;
|
||||
|
||||
if response.status() != StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
return Err(format!("attach policy to user failed: {status} {body}").into());
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn admin_update_group_members(
|
||||
@@ -1912,21 +1941,6 @@ async fn site_replication_info(env: &RustFSTestEnvironment) -> Result<SiteReplic
|
||||
Ok(serde_json::from_slice(&response.bytes().await?)?)
|
||||
}
|
||||
|
||||
async fn site_replication_rotate_svc_acct(
|
||||
env: &RustFSTestEnvironment,
|
||||
) -> Result<ReplicateEditStatus, Box<dyn Error + Send + Sync>> {
|
||||
let url = format!("{}/rustfs/admin/v3/site-replication/rotate-svc-acct", env.url);
|
||||
let response = signed_request(http::Method::POST, &url, &env.access_key, &env.secret_key, None, None).await?;
|
||||
|
||||
if response.status() != StatusCode::OK {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
return Err(format!("site replication rotate-svc-acct failed: {status} {body}").into());
|
||||
}
|
||||
|
||||
Ok(serde_json::from_slice(&response.bytes().await?)?)
|
||||
}
|
||||
|
||||
async fn site_replication_resync_op(
|
||||
env: &RustFSTestEnvironment,
|
||||
operation: &str,
|
||||
@@ -6325,112 +6339,6 @@ async fn test_site_replication_remove_all_real_dual_node() -> Result<(), Box<dyn
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_site_replication_rotate_svc_acct_completes_and_replication_survives_real_dual_node()
|
||||
-> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
init_logging();
|
||||
|
||||
let mut source_env = RustFSTestEnvironment::new().await?;
|
||||
source_env
|
||||
.start_rustfs_server_with_env(vec![], LOOPBACK_REPLICATION_TARGET_ENV)
|
||||
.await?;
|
||||
|
||||
let mut target_env = RustFSTestEnvironment::new().await?;
|
||||
target_env
|
||||
.start_rustfs_server_without_cleanup_with_env(LOOPBACK_REPLICATION_TARGET_ENV)
|
||||
.await?;
|
||||
|
||||
let source_client = source_env.create_s3_client();
|
||||
let target_client = target_env.create_s3_client();
|
||||
let bucket = "site-repl-rotate-svc-acct";
|
||||
|
||||
let add_status = site_replication_add(
|
||||
&source_env,
|
||||
&[
|
||||
PeerSite {
|
||||
name: "source-site".to_string(),
|
||||
endpoint: source_env.url.clone(),
|
||||
access_key: source_env.access_key.clone(),
|
||||
secret_key: source_env.secret_key.clone(),
|
||||
..Default::default()
|
||||
},
|
||||
PeerSite {
|
||||
name: "target-site".to_string(),
|
||||
endpoint: target_env.url.clone(),
|
||||
access_key: target_env.access_key.clone(),
|
||||
secret_key: target_env.secret_key.clone(),
|
||||
..Default::default()
|
||||
},
|
||||
],
|
||||
)
|
||||
.await?;
|
||||
assert!(add_status.success, "unexpected site add result: {add_status:?}");
|
||||
|
||||
let _source_info = wait_for_site_replication_enabled(&source_env, 2).await?;
|
||||
let _target_info = wait_for_site_replication_enabled(&target_env, 2).await?;
|
||||
|
||||
source_client.create_bucket().bucket(bucket).send().await?;
|
||||
enable_bucket_versioning(&source_env, bucket).await?;
|
||||
wait_for_bucket_on_target(&target_client, bucket).await?;
|
||||
let baseline_payload = b"before rotation".to_vec();
|
||||
source_client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key("before-rotate.txt")
|
||||
.body(ByteStream::from(baseline_payload.clone()))
|
||||
.send()
|
||||
.await?;
|
||||
let replicated_baseline = wait_for_object_on_target(&target_client, bucket, "before-rotate.txt").await?;
|
||||
assert_eq!(replicated_baseline, baseline_payload);
|
||||
|
||||
// A single rotation call must finish the whole hand-over. Before the fix
|
||||
// the join push could only sign with the freshly installed secret, every
|
||||
// peer rejected it, the rotation stayed pending forever, and both
|
||||
// replication directions were dead until an operator retried.
|
||||
let rotate_status = site_replication_rotate_svc_acct(&source_env).await?;
|
||||
assert!(rotate_status.success, "rotation did not complete in one call: {rotate_status:?}");
|
||||
|
||||
for env in [&source_env, &target_env] {
|
||||
let deadline = std::time::Instant::now() + Duration::from_secs(30);
|
||||
loop {
|
||||
let info = site_replication_info(env).await?;
|
||||
if info.enabled && info.pending_operation.is_none() {
|
||||
break;
|
||||
}
|
||||
if std::time::Instant::now() > deadline {
|
||||
return Err(format!("rotation left {} with a pending operation: {:?}", env.url, info.pending_operation).into());
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
}
|
||||
}
|
||||
|
||||
// Replication must actually flow again in both directions with the
|
||||
// rotated service-account secret.
|
||||
let forward_payload = b"after rotation from source".to_vec();
|
||||
source_client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key("after-rotate-forward.txt")
|
||||
.body(ByteStream::from(forward_payload.clone()))
|
||||
.send()
|
||||
.await?;
|
||||
let replicated_forward = wait_for_object_on_target(&target_client, bucket, "after-rotate-forward.txt").await?;
|
||||
assert_eq!(replicated_forward, forward_payload);
|
||||
|
||||
let reverse_payload = b"after rotation from target".to_vec();
|
||||
target_client
|
||||
.put_object()
|
||||
.bucket(bucket)
|
||||
.key("after-rotate-reverse.txt")
|
||||
.body(ByteStream::from(reverse_payload.clone()))
|
||||
.send()
|
||||
.await?;
|
||||
let replicated_reverse = wait_for_object_on_target(&source_client, bucket, "after-rotate-reverse.txt").await?;
|
||||
assert_eq!(replicated_reverse, reverse_payload);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_site_replication_state_edit_fresh_and_stale_real_dual_node() -> Result<(), Box<dyn Error + Send + Sync>> {
|
||||
init_logging();
|
||||
|
||||
@@ -1,84 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Raw HTTP regression coverage for the Select request root alias (backlog#1626).
|
||||
|
||||
use crate::common::{RustFSTestEnvironment, signed_s3_request};
|
||||
use aws_sdk_s3::primitives::ByteStream;
|
||||
use http::Method;
|
||||
use std::error::Error;
|
||||
use uuid::Uuid;
|
||||
|
||||
type TestResult<T = ()> = Result<T, Box<dyn Error + Send + Sync>>;
|
||||
|
||||
const CSV_BODY: &[u8] = b"name\nGatewayJ-root-alias\nignored\n";
|
||||
const EXPECTED_RECORD: &[u8] = b"GatewayJ-root-alias";
|
||||
|
||||
fn select_request(root: &str) -> String {
|
||||
format!(
|
||||
r#"<{root} xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
||||
<Expression>SELECT s.name FROM S3Object s WHERE s.name = 'GatewayJ-root-alias'</Expression>
|
||||
<ExpressionType>SQL</ExpressionType>
|
||||
<InputSerialization><CSV><FileHeaderInfo>USE</FileHeaderInfo></CSV></InputSerialization>
|
||||
<OutputSerialization><CSV/></OutputSerialization>
|
||||
</{root}>"#
|
||||
)
|
||||
}
|
||||
|
||||
async fn raw_select(env: &RustFSTestEnvironment, bucket: &str, object: &str, root: &str) -> TestResult {
|
||||
let response = signed_s3_request(
|
||||
Method::POST,
|
||||
&format!("{}/{bucket}/{object}?select&select-type=2", env.url),
|
||||
Some(select_request(root)),
|
||||
Some("application/xml"),
|
||||
&env.access_key,
|
||||
&env.secret_key,
|
||||
)
|
||||
.await?;
|
||||
let status = response.status();
|
||||
let body = response.bytes().await?.to_vec();
|
||||
assert_eq!(
|
||||
status,
|
||||
reqwest::StatusCode::OK,
|
||||
"{root} root was rejected: {}",
|
||||
String::from_utf8_lossy(&body)
|
||||
);
|
||||
assert!(
|
||||
body.windows(EXPECTED_RECORD.len()).any(|window| window == EXPECTED_RECORD),
|
||||
"{root} root did not return the projected record"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn select_request_root_alias_reaches_select_endpoint() -> TestResult {
|
||||
let mut env = RustFSTestEnvironment::new().await?;
|
||||
env.start_rustfs_server(Vec::new()).await?;
|
||||
let client = env.create_s3_client();
|
||||
let bucket = format!("select-root-{}", Uuid::new_v4().simple());
|
||||
let object = "input.csv";
|
||||
|
||||
client.create_bucket().bucket(&bucket).send().await?;
|
||||
client
|
||||
.put_object()
|
||||
.bucket(&bucket)
|
||||
.key(object)
|
||||
.body(ByteStream::from_static(CSV_BODY))
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
raw_select(&env, &bucket, object, "SelectObjectContentRequest").await?;
|
||||
raw_select(&env, &bucket, object, "SelectRequest").await?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -146,6 +146,7 @@ bytes = { workspace = true, features = ["serde"] }
|
||||
byteorder = { workspace = true }
|
||||
chrono = { workspace = true, features = ["serde"] }
|
||||
jiff = { workspace = true, features = ["serde"] }
|
||||
glob = { workspace = true }
|
||||
thiserror.workspace = true
|
||||
flatbuffers.workspace = true
|
||||
futures.workspace = true
|
||||
|
||||
@@ -194,10 +194,9 @@ pub mod bucket {
|
||||
BucketReplicationResyncStatus, BucketReplicationStat, BucketReplicationStats, BucketStats,
|
||||
DeleteReplicationConfigSnapshot, DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, InQueueMetric,
|
||||
MrfOpKind, MrfReplicateEntry, MustReplicateOptions, ObjectOpts, OperatorRuleContract,
|
||||
REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_READ_ONLY_HISTORICAL_FIELDS,
|
||||
REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE,
|
||||
REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
|
||||
ReplicateDecision, ReplicateObjectInfo, ReplicationBatchAdmission, ReplicationConfig,
|
||||
REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS,
|
||||
REPLICATE_INCOMING_DELETE, REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS,
|
||||
REPLICATION_WRITABLE_FIELDS, ReplicateDecision, ReplicateObjectInfo, ReplicationBatchAdmission, ReplicationConfig,
|
||||
ReplicationConfigStructureError, ReplicationConfigurationExt, ReplicationDeleteScheduleInput,
|
||||
ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge, ReplicationObjectIO,
|
||||
ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission, ReplicationScannerBridge,
|
||||
@@ -406,7 +405,7 @@ pub mod notification {
|
||||
pub use crate::services::notification_sys::{
|
||||
CrossPoolFenceFleetProofToken, NotificationPeerErr, NotificationSys, ScannerPublicationLeaseGrant,
|
||||
acquire_cross_pool_fence_fleet_proof, cross_pool_fence_fleet_proof_matches, get_global_notification_sys,
|
||||
new_global_notification_sys, scanner_peer_transport_error_message_is_retryable, start_remote_version_state_fleet_probe,
|
||||
new_global_notification_sys, start_remote_version_state_fleet_probe,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -416,10 +415,9 @@ pub mod object {
|
||||
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
|
||||
ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
|
||||
RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
|
||||
SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError,
|
||||
ScannerPublicationCommitState, StreamConsumer, get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook,
|
||||
register_get_object_body_cache_hook, register_object_mutation_hook, unregister_get_object_body_cache_hook,
|
||||
unregister_object_mutation_hook,
|
||||
SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, StreamConsumer, get_object_body_cache_plaintext_len,
|
||||
lookup_get_object_body_cache_hook, register_get_object_body_cache_hook, register_object_mutation_hook,
|
||||
unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
|
||||
};
|
||||
pub use crate::store::{
|
||||
PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
|
||||
|
||||
@@ -22,7 +22,6 @@ use crate::bucket::target::{self, BucketTarget, BucketTargets, Credentials};
|
||||
use crate::bucket::versioning_sys::BucketVersioningSys;
|
||||
use crate::runtime::sources as runtime_sources;
|
||||
use aws_credential_types::Credentials as SdkCredentials;
|
||||
use aws_credential_types::provider::{ProvideCredentials, error::CredentialsError, future};
|
||||
use aws_sdk_s3::config::Region as SdkRegion;
|
||||
use aws_sdk_s3::config::SharedHttpClient;
|
||||
use aws_sdk_s3::error::ProvideErrorMetadata;
|
||||
@@ -78,7 +77,7 @@ use std::str::FromStr as _;
|
||||
use std::sync::Arc;
|
||||
use std::sync::OnceLock;
|
||||
use std::sync::Weak;
|
||||
use std::time::{Duration, Instant, SystemTime};
|
||||
use std::time::{Duration, Instant};
|
||||
use time::{OffsetDateTime, format_description::well_known::Rfc3339};
|
||||
use tokio::sync::Mutex;
|
||||
use tokio::sync::RwLock;
|
||||
@@ -90,71 +89,6 @@ use uuid::Uuid;
|
||||
|
||||
const MAX_CONCURRENT_TARGET_HEALTH_CHECKS: usize = 16;
|
||||
const REDACTED_CREDENTIAL: &str = "<redacted>";
|
||||
const EXPIRED_REMOTE_TARGET_CREDENTIALS: &str = "remote target credentials have expired";
|
||||
|
||||
#[derive(Clone)]
|
||||
struct RemoteTargetCredentialsProvider {
|
||||
credentials: SdkCredentials,
|
||||
}
|
||||
|
||||
impl RemoteTargetCredentialsProvider {
|
||||
fn resolve_at(&self, now: SystemTime) -> aws_credential_types::provider::Result {
|
||||
if self.credentials.expiry().is_some_and(|expiration| expiration <= now) {
|
||||
return Err(CredentialsError::provider_error(std::io::Error::other(EXPIRED_REMOTE_TARGET_CREDENTIALS)));
|
||||
}
|
||||
Ok(self.credentials.clone())
|
||||
}
|
||||
}
|
||||
|
||||
impl fmt::Debug for RemoteTargetCredentialsProvider {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
f.debug_struct("RemoteTargetCredentialsProvider")
|
||||
.field("temporary", &self.credentials.session_token().is_some())
|
||||
.field("expiration", &self.credentials.expiry())
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
impl ProvideCredentials for RemoteTargetCredentialsProvider {
|
||||
fn provide_credentials<'a>(&'a self) -> future::ProvideCredentials<'a>
|
||||
where
|
||||
Self: 'a,
|
||||
{
|
||||
future::ProvideCredentials::ready(self.resolve_at(SystemTime::now()))
|
||||
}
|
||||
|
||||
fn fallback_on_interrupt(&self) -> Option<SdkCredentials> {
|
||||
self.resolve_at(SystemTime::now()).ok()
|
||||
}
|
||||
}
|
||||
|
||||
fn remote_target_sdk_credentials(
|
||||
credentials: &Credentials,
|
||||
account_id: &str,
|
||||
now: SystemTime,
|
||||
) -> Result<SdkCredentials, &'static str> {
|
||||
let session_token = credentials.effective_session_token();
|
||||
let expiration = credentials.effective_expiration().map(SystemTime::from);
|
||||
if expiration.is_some() && session_token.is_none() {
|
||||
return Err("remote target credential expiration requires a session token");
|
||||
}
|
||||
if expiration.is_some_and(|expiration| expiration <= now) {
|
||||
return Err(EXPIRED_REMOTE_TARGET_CREDENTIALS);
|
||||
}
|
||||
|
||||
let mut builder = SdkCredentials::builder()
|
||||
.access_key_id(credentials.access_key.clone())
|
||||
.secret_access_key(credentials.secret_key.clone())
|
||||
.account_id(account_id.to_string())
|
||||
.provider_name("bucket_target_sys");
|
||||
if let Some(session_token) = session_token {
|
||||
builder = builder.session_token(session_token.to_string());
|
||||
}
|
||||
if let Some(expiration) = expiration {
|
||||
builder = builder.expiry(expiration);
|
||||
}
|
||||
Ok(builder.build())
|
||||
}
|
||||
|
||||
pub type HeadObjectSdkError = Box<SdkError<HeadObjectError>>;
|
||||
pub type GetObjectSdkError = Box<SdkError<GetObjectError>>;
|
||||
@@ -418,28 +352,6 @@ pub struct BucketTargetSys {
|
||||
heartbeat_started: OnceLock<()>,
|
||||
}
|
||||
|
||||
/// Build the bucket-target health-check HTTP client without panicking when
|
||||
/// the host has no system CA bundle (issue #6734).
|
||||
///
|
||||
/// `BucketTargetSys::get()` initializes lazily on the startup path (bucket
|
||||
/// metadata install calls it on the main thread), and `reqwest::Client::new()`
|
||||
/// panics when the TLS backend cannot load any system trust root — the state
|
||||
/// of a minimal container image. Fall back to a client with an explicit empty
|
||||
/// trust store: HTTP health checks keep working, and HTTPS targets fail closed
|
||||
/// at the TLS handshake with a clear certificate error instead of aborting
|
||||
/// the whole process at startup.
|
||||
fn build_health_check_client() -> HttpClient {
|
||||
HttpClient::builder().build().unwrap_or_else(|error| {
|
||||
warn!(
|
||||
"bucket target health-check HTTP client could not load system TLS roots ({error}); continuing with an empty trust store — HTTPS target health checks will fail until a CA bundle is installed"
|
||||
);
|
||||
HttpClient::builder()
|
||||
.tls_certs_only(std::iter::empty::<reqwest::Certificate>())
|
||||
.build()
|
||||
.expect("HTTP client construction must succeed with an explicit empty trust store")
|
||||
})
|
||||
}
|
||||
|
||||
impl BucketTargetSys {
|
||||
pub fn get() -> &'static Self {
|
||||
GLOBAL_BUCKET_TARGET_SYS.get_or_init(Self::new)
|
||||
@@ -452,7 +364,7 @@ impl BucketTargetSys {
|
||||
targets_map: Arc::new(RwLock::new(HashMap::new())),
|
||||
h_mutex: Arc::new(RwLock::new(HashMap::new())),
|
||||
target_h_mutex: Arc::new(RwLock::new(HashMap::new())),
|
||||
hc_client: Arc::new(build_health_check_client()),
|
||||
hc_client: Arc::new(HttpClient::new()),
|
||||
a_mutex: Arc::new(Mutex::new(HashMap::new())),
|
||||
arn_errs_map: Arc::new(RwLock::new(HashMap::new())),
|
||||
target_update_mutexes: Arc::new(Mutex::new(HashMap::new())),
|
||||
@@ -911,26 +823,13 @@ impl BucketTargetSys {
|
||||
Ok(BucketTargets { targets: new_targets })
|
||||
}
|
||||
|
||||
async fn mark_refresh_attempt(&self, arn: &str) {
|
||||
// Rate-limit a failed config fetch as well as a failed client build.
|
||||
// A successful rebuild replaces this timestamp during publication.
|
||||
self.arn_remotes_map
|
||||
.write()
|
||||
.await
|
||||
.entry(arn.to_string())
|
||||
.or_default()
|
||||
.last_refresh = OffsetDateTime::now_utc();
|
||||
}
|
||||
|
||||
pub async fn mark_refresh_in_progress(&self, bucket: &str, arn: &str) {
|
||||
let mut arn_errs = self.arn_errs_map.write().await;
|
||||
let err = arn_errs.entry(arn.to_string()).or_insert_with(|| ArnErrs {
|
||||
count: 1,
|
||||
arn_errs.entry(arn.to_string()).or_insert_with(|| ArnErrs {
|
||||
bucket: bucket.to_string(),
|
||||
..Default::default()
|
||||
update_in_progress: true,
|
||||
count: 1,
|
||||
});
|
||||
err.update_in_progress = true;
|
||||
err.bucket = bucket.to_string();
|
||||
}
|
||||
|
||||
pub async fn mark_refresh_done(&self, bucket: &str, arn: &str) {
|
||||
@@ -942,21 +841,15 @@ impl BucketTargetSys {
|
||||
}
|
||||
|
||||
pub async fn is_reloading_target(&self, _bucket: &str, arn: &str) -> bool {
|
||||
self.arn_errs_map
|
||||
.read()
|
||||
.await
|
||||
.get(arn)
|
||||
.is_some_and(|err| err.update_in_progress)
|
||||
let arn_errs = self.arn_errs_map.read().await;
|
||||
arn_errs.get(arn).map(|err| err.update_in_progress).unwrap_or(false)
|
||||
}
|
||||
|
||||
pub async fn inc_arn_errs(&self, bucket: &str, arn: &str) {
|
||||
pub async fn inc_arn_errs(&self, _bucket: &str, arn: &str) {
|
||||
let mut arn_errs = self.arn_errs_map.write().await;
|
||||
let err = arn_errs.entry(arn.to_string()).or_insert_with(|| ArnErrs {
|
||||
bucket: bucket.to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
err.count += 1;
|
||||
err.bucket = bucket.to_string();
|
||||
if let Some(err) = arn_errs.get_mut(arn) {
|
||||
err.count += 1;
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn get_remote_target_client(&self, bucket: &str, arn: &str) -> Option<Arc<TargetClient>> {
|
||||
@@ -969,15 +862,15 @@ impl BucketTargetSys {
|
||||
.unwrap_or((None, None))
|
||||
};
|
||||
|
||||
let credentials_expired = cli
|
||||
.as_ref()
|
||||
.is_some_and(|client| client.credentials_expired_at(jiff::Timestamp::now()));
|
||||
if let Some(cli) = cli
|
||||
&& !credentials_expired
|
||||
{
|
||||
if let Some(cli) = cli {
|
||||
return Some(cli);
|
||||
}
|
||||
|
||||
// TODO(backlog): spawn an async task to proactively reload the replication target
|
||||
if self.is_reloading_target(bucket, arn).await {
|
||||
return None;
|
||||
}
|
||||
|
||||
if let Some(last_refresh) = last_refresh {
|
||||
let now = OffsetDateTime::now_utc();
|
||||
if now - last_refresh < Duration::from_secs(60 * 5) {
|
||||
@@ -985,24 +878,16 @@ impl BucketTargetSys {
|
||||
}
|
||||
}
|
||||
|
||||
// The existing per-bucket publication lock is also the reload claim:
|
||||
// try-locking keeps the request path non-blocking, is cancellation-safe,
|
||||
// and prevents a stale reload from publishing after a credential update.
|
||||
let update_mutex = self.target_update_mutex(bucket).await;
|
||||
let Ok(update_guard) = update_mutex.try_lock() else {
|
||||
return None;
|
||||
};
|
||||
self.mark_refresh_attempt(arn).await;
|
||||
|
||||
match get_bucket_targets_config(bucket).await {
|
||||
Ok(bucket_targets) => {
|
||||
self.update_all_targets_locked(bucket, Some(&bucket_targets)).await;
|
||||
self.mark_refresh_in_progress(bucket, arn).await;
|
||||
self.update_all_targets(bucket, Some(&bucket_targets)).await;
|
||||
self.mark_refresh_done(bucket, arn).await;
|
||||
}
|
||||
Err(e) => {
|
||||
error!("get bucket targets config error:{}", e);
|
||||
}
|
||||
};
|
||||
drop(update_guard);
|
||||
|
||||
let cli = self
|
||||
.arn_remotes_map
|
||||
@@ -1010,10 +895,8 @@ impl BucketTargetSys {
|
||||
.await
|
||||
.get(arn)
|
||||
.and_then(|target| target.client.clone());
|
||||
if let Some(cli) = cli
|
||||
&& !cli.credentials_expired_at(jiff::Timestamp::now())
|
||||
{
|
||||
return Some(cli);
|
||||
if cli.is_some() {
|
||||
return cli;
|
||||
}
|
||||
|
||||
self.inc_arn_errs(bucket, arn).await;
|
||||
@@ -1043,13 +926,12 @@ impl BucketTargetSys {
|
||||
});
|
||||
};
|
||||
|
||||
let creds = remote_target_sdk_credentials(credentials, &target.reset_id, SystemTime::now()).map_err(|error| {
|
||||
BucketTargetError::RemoteTargetConnectionErr {
|
||||
bucket: target.target_bucket.clone(),
|
||||
access_key: credentials.access_key.clone(),
|
||||
error: error.to_string(),
|
||||
}
|
||||
})?;
|
||||
let creds = SdkCredentials::builder()
|
||||
.access_key_id(credentials.access_key.clone())
|
||||
.secret_access_key(credentials.secret_key.clone())
|
||||
.account_id(target.reset_id.clone())
|
||||
.provider_name("bucket_target_sys")
|
||||
.build();
|
||||
|
||||
let endpoint = if target.secure {
|
||||
format!("https://{}", target.endpoint)
|
||||
@@ -1069,7 +951,7 @@ impl BucketTargetSys {
|
||||
|
||||
let mut config_builder = S3Config::builder()
|
||||
.endpoint_url(endpoint.clone())
|
||||
.credentials_provider(SharedCredentialsProvider::new(RemoteTargetCredentialsProvider { credentials: creds }))
|
||||
.credentials_provider(SharedCredentialsProvider::new(creds))
|
||||
.region(SdkRegion::new(target.region.clone()))
|
||||
.behavior_version(aws_sdk_s3::config::BehaviorVersion::latest());
|
||||
|
||||
@@ -1143,13 +1025,6 @@ impl BucketTargetSys {
|
||||
let update_mutex = self.target_update_mutex(bucket).await;
|
||||
let _update_guard = update_mutex.lock().await;
|
||||
|
||||
self.update_all_targets_locked(bucket, targets).await;
|
||||
}
|
||||
|
||||
/// Builds and publishes one bucket snapshot while its update mutex is held.
|
||||
/// Keeping persisted-config reads under the same mutex prevents a stale
|
||||
/// reload from overwriting a concurrent credential rotation.
|
||||
async fn update_all_targets_locked(&self, bucket: &str, targets: Option<&BucketTargets>) {
|
||||
let mut clients = Vec::new();
|
||||
if let Some(new_targets) = targets {
|
||||
for target in &new_targets.targets {
|
||||
@@ -1181,17 +1056,6 @@ impl BucketTargetSys {
|
||||
&& !new_targets.is_empty()
|
||||
{
|
||||
for (target, client) in clients {
|
||||
// Keep a timestamped placeholder for configured targets whose
|
||||
// client cannot be built. Replication records these attempts as
|
||||
// failed, while the placeholder prevents every object from
|
||||
// triggering another metadata reload/client build for five minutes.
|
||||
arn_remotes_map.insert(
|
||||
target.arn.clone(),
|
||||
ArnTarget {
|
||||
client: None,
|
||||
last_refresh: OffsetDateTime::now_utc(),
|
||||
},
|
||||
);
|
||||
match client {
|
||||
Ok(client) => {
|
||||
arn_remotes_map.insert(
|
||||
@@ -1204,6 +1068,11 @@ impl BucketTargetSys {
|
||||
health_map.insert(client.arn.clone(), target_health(&client));
|
||||
self.update_bandwidth_limit(bucket, &target.arn, target.bandwidth_limit);
|
||||
}
|
||||
// The target stays in `targets_map`, so it keeps showing up in
|
||||
// `bucket remote ls` while no client exists to replicate through it —
|
||||
// replication then drops every object for this ARN. Without this the
|
||||
// rejection (loopback endpoint, bad CA, unparseable URL) left no trace
|
||||
// anywhere.
|
||||
Err(err) => warn!(
|
||||
bucket = %bucket,
|
||||
arn = %target.arn,
|
||||
@@ -2071,13 +1940,6 @@ pub struct TargetClient {
|
||||
}
|
||||
|
||||
impl TargetClient {
|
||||
fn credentials_expired_at(&self, now: jiff::Timestamp) -> bool {
|
||||
self.credentials
|
||||
.as_ref()
|
||||
.and_then(Credentials::effective_expiration)
|
||||
.is_some_and(|expiration| expiration <= now)
|
||||
}
|
||||
|
||||
pub fn to_url(&self) -> Url {
|
||||
Url::parse(&self.endpoint).unwrap()
|
||||
}
|
||||
@@ -2482,21 +2344,6 @@ impl TargetClient {
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn abort_multipart_upload(&self, bucket: &str, object: &str, upload_id: &str) -> Result<(), S3ClientError> {
|
||||
match self
|
||||
.client
|
||||
.abort_multipart_upload()
|
||||
.bucket(bucket)
|
||||
.key(object)
|
||||
.upload_id(upload_id)
|
||||
.send()
|
||||
.await
|
||||
{
|
||||
Ok(_) => Ok(()),
|
||||
Err(e) => Err(e.into()),
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn remove_object(
|
||||
&self,
|
||||
bucket: &str,
|
||||
@@ -2643,18 +2490,6 @@ mod tests {
|
||||
use super::*;
|
||||
use rcgen::generate_simple_self_signed;
|
||||
|
||||
// The startup panic fix for hosts without a CA bundle (issue #6734) rests
|
||||
// on two properties: the health-check client constructor never panics, and
|
||||
// its degraded fallback — an explicit empty trust store — always builds.
|
||||
#[test]
|
||||
fn health_check_client_construction_never_panics() {
|
||||
let _ = build_health_check_client();
|
||||
HttpClient::builder()
|
||||
.tls_certs_only(std::iter::empty::<reqwest::Certificate>())
|
||||
.build()
|
||||
.expect("empty-trust-store client build must succeed without touching system roots");
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
struct RecordingHttpConnector {
|
||||
request_uris: Arc<std::sync::Mutex<Vec<String>>>,
|
||||
@@ -2673,26 +2508,6 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
struct RecordingAuthConnector {
|
||||
signed_requests: Arc<std::sync::Mutex<Vec<(bool, bool)>>>,
|
||||
}
|
||||
|
||||
impl SmithyHttpConnector for RecordingAuthConnector {
|
||||
fn call(&self, request: HttpRequest) -> HttpConnectorFuture {
|
||||
let has_expected_token = request.headers().get("x-amz-security-token") == Some("temporary-session-token");
|
||||
let has_authorization = request.headers().contains_key("authorization");
|
||||
self.signed_requests
|
||||
.lock()
|
||||
.expect("recorded auth request lock should not be poisoned")
|
||||
.push((has_expected_token, has_authorization));
|
||||
HttpConnectorFuture::ready(Ok(HttpResponse::new(
|
||||
aws_smithy_runtime_api::http::StatusCode::try_from(200_u16).expect("200 should be a valid response status"),
|
||||
SdkBody::empty(),
|
||||
)))
|
||||
}
|
||||
}
|
||||
|
||||
fn recording_target_client() -> (TargetClient, Arc<std::sync::Mutex<Vec<String>>>) {
|
||||
let request_uris = Arc::new(std::sync::Mutex::new(Vec::new()));
|
||||
let connector = SharedHttpConnector::new(RecordingHttpConnector {
|
||||
@@ -2718,150 +2533,6 @@ mod tests {
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remote_target_sdk_credentials_preserve_temporary_credential_fields() {
|
||||
let now = SystemTime::UNIX_EPOCH + Duration::from_secs(1_000);
|
||||
let expiration = SystemTime::UNIX_EPOCH + Duration::from_secs(2_000);
|
||||
let credentials = Credentials {
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
session_token: Some("temporary-session-token".to_string()),
|
||||
expiration: Some(jiff::Timestamp::try_from(expiration).expect("test expiration should convert")),
|
||||
};
|
||||
|
||||
let sdk_credentials =
|
||||
remote_target_sdk_credentials(&credentials, "account", now).expect("unexpired temporary credentials should build");
|
||||
|
||||
assert_eq!(sdk_credentials.session_token(), Some("temporary-session-token"));
|
||||
assert_eq!(sdk_credentials.expiry(), Some(expiration));
|
||||
assert_eq!(sdk_credentials.account_id().map(|id| id.as_str()), Some("account"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remote_target_sdk_credentials_normalize_go_zero_expiration() {
|
||||
let credentials = Credentials {
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
session_token: None,
|
||||
expiration: Some("0001-01-01T00:00:00Z".parse().expect("Go zero time should parse")),
|
||||
};
|
||||
|
||||
let sdk_credentials = remote_target_sdk_credentials(&credentials, "", SystemTime::now())
|
||||
.expect("Go zero expiration should remain compatible with static credentials");
|
||||
|
||||
assert!(sdk_credentials.session_token().is_none());
|
||||
assert!(sdk_credentials.expiry().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remote_target_sdk_credentials_reject_invalid_expiration_boundaries() {
|
||||
let expiration = SystemTime::UNIX_EPOCH + Duration::from_secs(2_000);
|
||||
let mut credentials = Credentials {
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
session_token: None,
|
||||
expiration: Some(jiff::Timestamp::try_from(expiration).expect("test expiration should convert")),
|
||||
};
|
||||
|
||||
assert_eq!(
|
||||
remote_target_sdk_credentials(&credentials, "", SystemTime::UNIX_EPOCH + Duration::from_secs(1_000))
|
||||
.expect_err("expiration without a session token must fail"),
|
||||
"remote target credential expiration requires a session token"
|
||||
);
|
||||
|
||||
credentials.session_token = Some("temporary-session-token".to_string());
|
||||
assert_eq!(
|
||||
remote_target_sdk_credentials(&credentials, "", expiration)
|
||||
.expect_err("credentials expire at the exact expiration boundary"),
|
||||
EXPIRED_REMOTE_TARGET_CREDENTIALS
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remote_target_credentials_provider_fails_closed_after_expiration() {
|
||||
let expiration = SystemTime::UNIX_EPOCH + Duration::from_secs(2_000);
|
||||
let provider = RemoteTargetCredentialsProvider {
|
||||
credentials: SdkCredentials::new(
|
||||
"access",
|
||||
"secret",
|
||||
Some("temporary-session-token".to_string()),
|
||||
Some(expiration),
|
||||
"test",
|
||||
),
|
||||
};
|
||||
|
||||
assert!(provider.resolve_at(expiration - Duration::from_nanos(1)).is_ok());
|
||||
let err = provider
|
||||
.resolve_at(expiration)
|
||||
.expect_err("expired credentials must not be returned");
|
||||
assert_eq!(err.source().map(ToString::to_string).as_deref(), Some(EXPIRED_REMOTE_TARGET_CREDENTIALS));
|
||||
assert!(!format!("{provider:?}").contains("temporary-session-token"));
|
||||
assert!(!format!("{provider:?}").contains("secret"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn target_client_detects_expiration_for_cache_refresh() {
|
||||
let expiration: jiff::Timestamp = "2099-01-01T00:00:00Z".parse().expect("expiration should parse");
|
||||
let (mut client, _) = recording_target_client();
|
||||
client.credentials = Some(Credentials {
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
session_token: Some("temporary-session-token".to_string()),
|
||||
expiration: Some(expiration),
|
||||
});
|
||||
|
||||
assert!(!client.credentials_expired_at("2098-12-31T23:59:59Z".parse().expect("pre-expiration timestamp should parse")));
|
||||
assert!(client.credentials_expired_at(expiration));
|
||||
|
||||
client.credentials.as_mut().expect("credentials should exist").expiration =
|
||||
Some("0001-01-01T00:00:00Z".parse().expect("Go zero time should parse"));
|
||||
assert!(!client.credentials_expired_at(jiff::Timestamp::now()));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn temporary_credentials_add_security_token_to_sigv4_requests() {
|
||||
let signed_requests = Arc::new(std::sync::Mutex::new(Vec::new()));
|
||||
let connector = SharedHttpConnector::new(RecordingAuthConnector {
|
||||
signed_requests: Arc::clone(&signed_requests),
|
||||
});
|
||||
let http_client = http_client_fn(move |_settings, _components| connector.clone());
|
||||
let credentials = Credentials {
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
session_token: Some("temporary-session-token".to_string()),
|
||||
expiration: Some("2099-01-01T00:00:00Z".parse().expect("future expiration should parse")),
|
||||
};
|
||||
let sdk_credentials = remote_target_sdk_credentials(&credentials, "", SystemTime::now())
|
||||
.expect("unexpired temporary credentials should build");
|
||||
let client = S3Client::from_conf(
|
||||
S3Config::builder()
|
||||
.endpoint_url("https://target.example")
|
||||
.credentials_provider(SharedCredentialsProvider::new(RemoteTargetCredentialsProvider {
|
||||
credentials: sdk_credentials,
|
||||
}))
|
||||
.region(SdkRegion::new("us-east-1"))
|
||||
.http_client(http_client)
|
||||
.behavior_version(aws_sdk_s3::config::BehaviorVersion::latest())
|
||||
.build(),
|
||||
);
|
||||
|
||||
client
|
||||
.head_bucket()
|
||||
.bucket("target-bucket")
|
||||
.send()
|
||||
.await
|
||||
.expect("recording connector should accept the signed request");
|
||||
|
||||
assert_eq!(
|
||||
signed_requests
|
||||
.lock()
|
||||
.expect("recorded auth request lock should not be poisoned")
|
||||
.as_slice(),
|
||||
&[(true, true)],
|
||||
"SigV4 request must include both authorization and the session-token header"
|
||||
);
|
||||
}
|
||||
|
||||
fn spawn_https_server(cert: &rcgen::CertifiedKey<rcgen::KeyPair>, requests: usize) -> (u16, std::thread::JoinHandle<()>) {
|
||||
use std::io::{Read, Write};
|
||||
|
||||
@@ -3793,29 +3464,6 @@ mod tests {
|
||||
assert!(mutexes.contains_key("second"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn target_refresh_attempt_updates_retry_timestamp_and_error_count() {
|
||||
let sys = BucketTargetSys::default();
|
||||
|
||||
sys.mark_refresh_attempt("arn:reload").await;
|
||||
let last_refresh = sys.arn_remotes_map.read().await["arn:reload"].last_refresh;
|
||||
assert!(OffsetDateTime::now_utc() - last_refresh < Duration::from_secs(5));
|
||||
|
||||
sys.inc_arn_errs("bucket", "arn:reload").await;
|
||||
sys.inc_arn_errs("bucket", "arn:reload").await;
|
||||
let errors = sys.arn_errs_map.read().await;
|
||||
assert_eq!(errors["arn:reload"].count, 2);
|
||||
assert_eq!(errors["arn:reload"].bucket, "bucket");
|
||||
drop(errors);
|
||||
|
||||
sys.mark_refresh_in_progress("bucket", "arn:reload").await;
|
||||
assert!(sys.is_reloading_target("bucket", "arn:reload").await);
|
||||
sys.mark_refresh_done("bucket", "arn:reload").await;
|
||||
assert!(!sys.is_reloading_target("bucket", "arn:reload").await);
|
||||
sys.mark_refresh_in_progress("bucket", "arn:reload").await;
|
||||
assert!(sys.is_reloading_target("bucket", "arn:reload").await);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_all_targets_publishes_disable_proxy_on_target_client() {
|
||||
// The read-proxy selector (replication_proxy::get_proxy_targets) skips
|
||||
@@ -3854,88 +3502,6 @@ mod tests {
|
||||
assert!(opted_out.disable_proxy, "disable_proxy must reach the published TargetClient");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_all_targets_keeps_failed_client_placeholder() {
|
||||
let sys = BucketTargetSys::default();
|
||||
let target = BucketTarget {
|
||||
arn: "arn:expired".to_string(),
|
||||
endpoint: "192.168.1.10:9000".to_string(),
|
||||
target_bucket: "target-bucket".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
credentials: Some(Credentials {
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
session_token: Some("temporary-session-token".to_string()),
|
||||
expiration: Some("2000-01-01T00:00:00Z".parse().expect("expired timestamp should parse")),
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let targets = BucketTargets { targets: vec![target] };
|
||||
|
||||
sys.update_all_targets("bucket", Some(&targets)).await;
|
||||
|
||||
let remotes = sys.arn_remotes_map.read().await;
|
||||
let placeholder = remotes
|
||||
.get("arn:expired")
|
||||
.expect("configured target should retain a cache entry");
|
||||
assert!(placeholder.client.is_none());
|
||||
assert!(OffsetDateTime::now_utc() - placeholder.last_refresh < Duration::from_secs(5));
|
||||
drop(remotes);
|
||||
assert!(sys.get_remote_target_client("bucket", "arn:expired").await.is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn credential_rotation_atomically_replaces_published_client() {
|
||||
let sys = BucketTargetSys::default();
|
||||
let target = |session_token: &str| BucketTarget {
|
||||
arn: "arn:rotating".to_string(),
|
||||
endpoint: "192.168.1.10:9000".to_string(),
|
||||
target_bucket: "target-bucket".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
credentials: Some(Credentials {
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
session_token: Some(session_token.to_string()),
|
||||
expiration: None,
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
sys.update_all_targets(
|
||||
"bucket",
|
||||
Some(&BucketTargets {
|
||||
targets: vec![target("old-session-token")],
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
let old_client = sys
|
||||
.get_remote_target_client("bucket", "arn:rotating")
|
||||
.await
|
||||
.expect("initial client should be published");
|
||||
|
||||
sys.update_all_targets(
|
||||
"bucket",
|
||||
Some(&BucketTargets {
|
||||
targets: vec![target("new-session-token")],
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
let new_client = sys
|
||||
.get_remote_target_client("bucket", "arn:rotating")
|
||||
.await
|
||||
.expect("rotated client should be published");
|
||||
|
||||
assert!(!Arc::ptr_eq(&old_client, &new_client));
|
||||
assert_eq!(
|
||||
old_client.credentials.as_ref().and_then(Credentials::effective_session_token),
|
||||
Some("old-session-token")
|
||||
);
|
||||
assert_eq!(
|
||||
new_client.credentials.as_ref().and_then(Credentials::effective_session_token),
|
||||
Some("new-session-token")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn target_updates_serialize_client_build_through_publication_per_bucket() {
|
||||
let sys = Arc::new(BucketTargetSys::default());
|
||||
|
||||
@@ -490,7 +490,7 @@ impl ExpiryStats {
|
||||
}
|
||||
|
||||
fn add_nonnegative(counter: &AtomicI64, delta: i64) {
|
||||
let _ = counter.try_update(Ordering::Relaxed, Ordering::Relaxed, |current| Some(current.saturating_add(delta).max(0)));
|
||||
let _ = counter.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| Some(current.saturating_add(delta).max(0)));
|
||||
}
|
||||
|
||||
fn increment_missed_expiry_tasks(&self) {
|
||||
|
||||
@@ -20,13 +20,14 @@ mod durable_namespace;
|
||||
pub mod evaluator;
|
||||
pub mod manual_transition_job;
|
||||
mod metadata_boundary;
|
||||
pub(crate) use metadata_boundary::{LifecycleExpiryConfigs, get_expiry_configs, get_lifecycle_config};
|
||||
pub(crate) use metadata_boundary::{LifecycleExpiryConfigs, get_expiry_configs};
|
||||
mod object_handlers_common;
|
||||
mod object_lock_boundary;
|
||||
pub use self::core as lifecycle;
|
||||
mod replication_sink;
|
||||
pub mod rule;
|
||||
mod runtime_boundary;
|
||||
mod tagging_boundary;
|
||||
pub mod tier_delete_journal;
|
||||
pub mod tier_free_version_recovery;
|
||||
pub mod tier_last_day_stats;
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use std::collections::HashMap;
|
||||
|
||||
#[allow(
|
||||
dead_code,
|
||||
reason = "declared boundary surface for the ECStore replication split plan; no caller in this port (backlog#1823)"
|
||||
)]
|
||||
pub(crate) fn decode_tags_to_map(tags: &str) -> HashMap<String, String> {
|
||||
crate::bucket::tagging::decode_tags_to_map(tags)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::decode_tags_to_map;
|
||||
|
||||
#[test]
|
||||
fn decode_tags_to_map_preserves_bucket_tagging_parser_behavior() {
|
||||
let tags = decode_tags_to_map("env=prod&encoded=a%2Fb&=ignored");
|
||||
|
||||
assert_eq!(tags.get("env").map(String::as_str), Some("prod"));
|
||||
assert_eq!(tags.get("encoded").map(String::as_str), Some("a/b"));
|
||||
assert!(!tags.contains_key(""));
|
||||
}
|
||||
}
|
||||
@@ -44,14 +44,14 @@ mod replication_versioning_boundary;
|
||||
mod runtime_boundary;
|
||||
|
||||
pub use replication_config_boundary::{
|
||||
ObjectOpts, OperatorRuleContract, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_READ_ONLY_HISTORICAL_FIELDS,
|
||||
REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATION_CAPABILITY_CONTRACT_VERSION,
|
||||
REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS, ReplicationConfigStructureError,
|
||||
ReplicationConfigurationExt, ReplicationTargetValidationError, assign_site_replication_rule_priorities,
|
||||
invalid_replication_config_status_field, is_site_replication_role, is_site_replication_rule,
|
||||
merge_incoming_replication_config, merge_user_replication_config, replication_target_arn_deployment_id,
|
||||
replication_target_arns, should_remove_replication_target, site_replication_rule_deployment_id,
|
||||
unsupported_replication_config_field, validate_replication_config_structure, validate_replication_config_target_arns,
|
||||
ObjectOpts, OperatorRuleContract, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS,
|
||||
REMOTE_TARGET_WRITABLE_FIELDS, REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS,
|
||||
REPLICATION_WRITABLE_FIELDS, ReplicationConfigStructureError, ReplicationConfigurationExt, ReplicationTargetValidationError,
|
||||
assign_site_replication_rule_priorities, invalid_replication_config_status_field, is_site_replication_role,
|
||||
is_site_replication_rule, merge_incoming_replication_config, merge_user_replication_config,
|
||||
replication_target_arn_deployment_id, replication_target_arns, should_remove_replication_target,
|
||||
site_replication_rule_deployment_id, unsupported_replication_config_field, validate_replication_config_structure,
|
||||
validate_replication_config_target_arns,
|
||||
};
|
||||
pub(crate) use replication_filemeta_boundary::version_purge_statuses_map;
|
||||
pub use replication_filemeta_boundary::{
|
||||
|
||||
@@ -13,12 +13,12 @@
|
||||
// limitations under the License.
|
||||
|
||||
pub use rustfs_replication::{
|
||||
ObjectOpts, OperatorRuleContract, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_READ_ONLY_HISTORICAL_FIELDS,
|
||||
REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATION_CAPABILITY_CONTRACT_VERSION,
|
||||
REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS, ReplicationConfigStructureError,
|
||||
ReplicationConfigurationExt, ReplicationRuleExt, ReplicationTargetValidationError, assign_site_replication_rule_priorities,
|
||||
invalid_replication_config_status_field, is_site_replication_role, is_site_replication_rule,
|
||||
merge_incoming_replication_config, merge_user_replication_config, replication_target_arn_deployment_id,
|
||||
replication_target_arns, should_remove_replication_target, site_replication_rule_deployment_id,
|
||||
unsupported_replication_config_field, validate_replication_config_structure, validate_replication_config_target_arns,
|
||||
ObjectOpts, OperatorRuleContract, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS,
|
||||
REMOTE_TARGET_WRITABLE_FIELDS, REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS,
|
||||
REPLICATION_WRITABLE_FIELDS, ReplicationConfigStructureError, ReplicationConfigurationExt, ReplicationRuleExt,
|
||||
ReplicationTargetValidationError, assign_site_replication_rule_priorities, invalid_replication_config_status_field,
|
||||
is_site_replication_role, is_site_replication_rule, merge_incoming_replication_config, merge_user_replication_config,
|
||||
replication_target_arn_deployment_id, replication_target_arns, should_remove_replication_target,
|
||||
site_replication_rule_deployment_id, unsupported_replication_config_field, validate_replication_config_structure,
|
||||
validate_replication_config_target_arns,
|
||||
};
|
||||
|
||||
@@ -436,21 +436,16 @@ pub(crate) async fn check_replicate_delete_strict(
|
||||
}
|
||||
|
||||
for target in decision.targets_map.values_mut() {
|
||||
let replicate_sync = ReplicationTargetStore::remote_target_client(bucket, &target.arn)
|
||||
.await
|
||||
.map(|client| client.replicate_sync);
|
||||
apply_target_delivery_mode(target, replicate_sync);
|
||||
if let Some(client) = ReplicationTargetStore::remote_target_client(bucket, &target.arn).await {
|
||||
target.synchronous = client.replicate_sync;
|
||||
} else {
|
||||
target.replicate = false;
|
||||
target.synchronous = false;
|
||||
}
|
||||
}
|
||||
Ok(decision)
|
||||
}
|
||||
|
||||
fn apply_target_delivery_mode(target: &mut ReplicateTargetDecision, replicate_sync: Option<bool>) {
|
||||
// A missing runtime client is a delivery failure, not a rule mismatch.
|
||||
// Preserve admission and fall back to the asynchronous worker, which can
|
||||
// persist FAILED state for the heal/retry path.
|
||||
target.synchronous = replicate_sync.unwrap_or(false);
|
||||
}
|
||||
|
||||
pub(crate) fn check_replicate_delete_with_snapshot(
|
||||
dobj: &ObjectToDelete,
|
||||
oi: &ObjectInfo,
|
||||
@@ -634,23 +629,6 @@ mod tests {
|
||||
}));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_target_client_preserves_delete_admission_as_async() {
|
||||
let mut target = ReplicateTargetDecision::new("arn:target".to_string(), true, true);
|
||||
|
||||
apply_target_delivery_mode(&mut target, None);
|
||||
|
||||
assert!(target.replicate, "a runtime client miss must not erase the replication rule decision");
|
||||
assert!(
|
||||
!target.synchronous,
|
||||
"unavailable synchronous targets must fall back to the async retry path"
|
||||
);
|
||||
|
||||
apply_target_delivery_mode(&mut target, Some(true));
|
||||
assert!(target.replicate);
|
||||
assert!(target.synchronous);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn must_replicate_options_preserve_request_flag() {
|
||||
let user_defined = HashMap::new();
|
||||
|
||||
@@ -19,9 +19,9 @@ pub use rustfs_replication::{
|
||||
};
|
||||
pub(crate) use rustfs_replication::{
|
||||
ReplicationDeleteSource, ReplicationMultipartPartInput, ReplicationResyncTargetObject, delete_marker_purge_mrf_entry,
|
||||
delete_marker_purge_version_id, delete_replication_creates_marker, delete_replication_missing_source_decision,
|
||||
delete_replication_object_opts, heal_uses_delete_replication_path, is_retryable_delete_replication_head_error,
|
||||
is_version_delete_replication, replicate_delete_outcome, replication_etags_match, replication_multipart_complete_actual_size,
|
||||
delete_marker_purge_version_id, delete_replication_missing_source_decision, delete_replication_object_opts,
|
||||
heal_uses_delete_replication_path, is_retryable_delete_replication_head_error, is_version_delete_replication,
|
||||
replicate_delete_outcome, replication_etags_match, replication_multipart_complete_actual_size,
|
||||
replication_multipart_part_plan, resync_existing_delete_replication_info, resync_target_for_object,
|
||||
should_retry_delete_marker_purge, target_delete_version_id,
|
||||
};
|
||||
|
||||
@@ -1048,6 +1048,7 @@ pub fn resync_start_conflict_id(error: &EcstoreError) -> Option<&str> {
|
||||
}
|
||||
|
||||
/// Main replication pool structure
|
||||
#[derive(Debug)]
|
||||
pub struct ReplicationPool<S: ReplicationStorage> {
|
||||
// Atomic counters for active workers
|
||||
active_workers: Arc<AtomicI32>,
|
||||
@@ -1093,16 +1094,6 @@ pub struct ReplicationPool<S: ReplicationStorage> {
|
||||
resyncer: Arc<ReplicationResyncer>,
|
||||
}
|
||||
|
||||
impl<S: ReplicationStorage> std::fmt::Debug for ReplicationPool<S> {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("ReplicationPool")
|
||||
.field("active_workers", &self.active_workers.load(Ordering::Relaxed))
|
||||
.field("active_lrg_workers", &self.active_lrg_workers.load(Ordering::Relaxed))
|
||||
.field("active_mrf_workers", &self.active_mrf_workers.load(Ordering::Relaxed))
|
||||
.finish_non_exhaustive()
|
||||
}
|
||||
}
|
||||
|
||||
impl<S: ReplicationStorage> ReplicationPool<S> {
|
||||
/// Creates a new replication pool with specified options
|
||||
pub async fn new(opts: ReplicationPoolOpts, stats: Arc<ReplicationStats>, storage: Arc<S>) -> Arc<Self> {
|
||||
@@ -2141,7 +2132,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
|
||||
}
|
||||
|
||||
/// Load bucket replication resync statuses into memory
|
||||
#[instrument(skip(self, buckets, _cancellation_token), fields(bucket_count = buckets.len()))]
|
||||
#[instrument(skip(_cancellation_token))]
|
||||
async fn load_resync(
|
||||
self: Arc<Self>,
|
||||
buckets: &[String],
|
||||
|
||||
@@ -30,8 +30,8 @@ use super::replication_msgp_boundary::ReplicationMsgpCodec;
|
||||
use super::replication_object_config::{ReplicationConfig, get_replication_config, must_replicate};
|
||||
use super::replication_object_decision_boundary::{
|
||||
MustReplicateOptions, ReplicationMultipartPartInput, delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
|
||||
delete_replication_creates_marker, heal_uses_delete_replication_path, is_retryable_delete_replication_head_error,
|
||||
is_version_delete_replication, replicate_delete_outcome, replication_etags_match, replication_multipart_complete_actual_size,
|
||||
heal_uses_delete_replication_path, is_retryable_delete_replication_head_error, is_version_delete_replication,
|
||||
replicate_delete_outcome, replication_etags_match, replication_multipart_complete_actual_size,
|
||||
replication_multipart_part_plan, resync_existing_delete_replication_info, should_retry_delete_marker_purge,
|
||||
target_delete_version_id,
|
||||
};
|
||||
@@ -54,12 +54,11 @@ use super::replication_storage_boundary::{
|
||||
};
|
||||
use super::replication_target_boundary::{
|
||||
ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions,
|
||||
ReplicationTargetStore, S3ClientError, SsecPassthroughCapability, SsecPassthroughGate, TargetClient,
|
||||
is_replication_target_offline_error, replication_action_for_target_head, replication_complete_multipart_options,
|
||||
replication_delete_marker_purge_remove_options, replication_delete_remove_options, replication_force_delete_remove_options,
|
||||
replication_object_is_ssec_encrypted, replication_put_object_header_size, replication_put_object_options,
|
||||
replication_target_head_is_newer_null_version, resolve_read_api_version_id, ssec_passthrough_evidence_present,
|
||||
ssec_passthrough_gate, version_identity_drifted,
|
||||
ReplicationTargetStore, SsecPassthroughCapability, SsecPassthroughGate, TargetClient, is_replication_target_offline_error,
|
||||
replication_action_for_target_head, replication_complete_multipart_options, replication_delete_marker_purge_remove_options,
|
||||
replication_delete_remove_options, replication_force_delete_remove_options, replication_object_is_ssec_encrypted,
|
||||
replication_put_object_header_size, replication_put_object_options, replication_target_head_is_newer_null_version,
|
||||
resolve_read_api_version_id, ssec_passthrough_evidence_present, ssec_passthrough_gate, version_identity_drifted,
|
||||
};
|
||||
use super::replication_versioning_boundary::ReplicationVersioningStore;
|
||||
use super::runtime_boundary as runtime_sources;
|
||||
@@ -102,7 +101,6 @@ const BACKGROUND_WALKDIR_TIMEOUT: TokioDuration = TokioDuration::from_secs(60);
|
||||
const ENV_REPL_RESYNC_MAX_JOBS: &str = "RUSTFS_REPL_RESYNC_MAX_JOBS";
|
||||
const DEFAULT_REPL_RESYNC_MAX_JOBS: usize = 2;
|
||||
const MAX_REPL_RESYNC_MAX_JOBS: usize = 32;
|
||||
const TARGET_CLIENT_UNAVAILABLE_ERROR: &str = "replication target client is unavailable";
|
||||
use uuid::Uuid;
|
||||
|
||||
const EVENT_RESYNC_STATUS_UPDATE_SKIPPED: &str = "replication_resync_status_update_skipped";
|
||||
@@ -1619,6 +1617,7 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
|
||||
}
|
||||
|
||||
let bucket = dobj.bucket.clone();
|
||||
let mut source_state_verified = true;
|
||||
let version_id = if let Some(version_id) = &dobj.delete_object.delete_marker_version_id {
|
||||
Some(version_id.to_owned())
|
||||
} else {
|
||||
@@ -1676,12 +1675,7 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
|
||||
return purge_stale_delete_marker_targets(&bucket, &dobj).await;
|
||||
}
|
||||
Err(err) => {
|
||||
// A transient source error (lock timeout, IO error) must not
|
||||
// fall through to the marker-creation send below: that DELETE
|
||||
// omits the versionId, so every such retry lets a generic S3
|
||||
// target mint one more delete marker (rustfs#6823). Fail the
|
||||
// entry without touching the target; the MRF replay / heal
|
||||
// scanner retries once the source is readable again.
|
||||
source_state_verified = false;
|
||||
debug!(
|
||||
event = EVENT_REPLICATION_DELETE_SKIPPED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
@@ -1693,20 +1687,6 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
|
||||
reason = "source_state_verification_failed",
|
||||
"Failed to verify source delete-marker state before replication"
|
||||
);
|
||||
send_local_event(EventArgs {
|
||||
event_name: EventName::ObjectReplicationNotTracked.to_string(),
|
||||
bucket_name: bucket.clone(),
|
||||
object: ObjectInfo {
|
||||
bucket: bucket.clone(),
|
||||
name: dobj.delete_object.object_name.clone(),
|
||||
version_id,
|
||||
delete_marker: dobj.delete_object.delete_marker,
|
||||
..Default::default()
|
||||
},
|
||||
user_agent: "Internal: [Replication]".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1848,7 +1828,19 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
|
||||
reason = "target_client_missing",
|
||||
"Skipping replication delete because target client is unavailable"
|
||||
);
|
||||
rinfos.targets.push(unavailable_delete_target_info(&dobj, &tgt_entry.arn));
|
||||
send_local_event(EventArgs {
|
||||
event_name: EventName::ObjectReplicationNotTracked.to_string(),
|
||||
bucket_name: bucket.clone(),
|
||||
object: ObjectInfo {
|
||||
bucket: bucket.clone(),
|
||||
name: dobj.delete_object.object_name.clone(),
|
||||
version_id,
|
||||
delete_marker: dobj.delete_object.delete_marker,
|
||||
..Default::default()
|
||||
},
|
||||
user_agent: "Internal: [Replication]".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
continue;
|
||||
};
|
||||
|
||||
@@ -2016,9 +2008,7 @@ pub(crate) async fn replicate_delete_with_outcome<S: ReplicationStorage>(
|
||||
expected_targets,
|
||||
rinfos.targets.len(),
|
||||
state_persisted,
|
||||
// Source state is verified by construction here: a verification
|
||||
// error returns early above instead of replicating unverified.
|
||||
true,
|
||||
source_state_verified,
|
||||
&replication_status,
|
||||
)
|
||||
}
|
||||
@@ -2573,32 +2563,6 @@ async fn replicate_force_delete_to_targets<S: ReplicationStorage>(dobj: &Deleted
|
||||
all_succeeded
|
||||
}
|
||||
|
||||
fn unavailable_delete_target_info(dobj: &DeletedObjectReplicationInfo, arn: &str) -> ReplicatedTargetInfo {
|
||||
let mut rinfo = dobj
|
||||
.delete_object
|
||||
.replication_state
|
||||
.as_ref()
|
||||
.map(|state| state.target_state(arn))
|
||||
.unwrap_or_else(|| ReplicatedTargetInfo {
|
||||
arn: arn.to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
rinfo.op_type = dobj.op_type;
|
||||
if is_version_delete_replication(&dobj.delete_object) {
|
||||
if rinfo.version_purge_status != VersionPurgeStatusType::Complete {
|
||||
rinfo.version_purge_status = VersionPurgeStatusType::Failed;
|
||||
rinfo.error = Some(TARGET_CLIENT_UNAVAILABLE_ERROR.to_string());
|
||||
}
|
||||
} else if rinfo.prev_replication_status == ReplicationStatusType::Completed && dobj.op_type != ReplicationType::ExistingObject
|
||||
{
|
||||
rinfo.replication_status = ReplicationStatusType::Completed;
|
||||
} else {
|
||||
rinfo.replication_status = ReplicationStatusType::Failed;
|
||||
rinfo.error = Some(TARGET_CLIENT_UNAVAILABLE_ERROR.to_string());
|
||||
}
|
||||
rinfo
|
||||
}
|
||||
|
||||
async fn replicate_delete_to_target(dobj: &DeletedObjectReplicationInfo, tgt_client: Arc<TargetClient>) -> ReplicatedTargetInfo {
|
||||
let version_id = if let Some(version_id) = &dobj.delete_object.delete_marker_version_id {
|
||||
version_id.to_owned()
|
||||
@@ -2673,14 +2637,7 @@ async fn replicate_delete_to_target(dobj: &DeletedObjectReplicationInfo, tgt_cli
|
||||
&tgt_client.bucket,
|
||||
&dobj.delete_object.object_name,
|
||||
version_id.clone(),
|
||||
// A version purge must keep the versionId on the DELETE even when
|
||||
// the purged version is a delete marker: marker-creation semantics
|
||||
// would drop it and a generic S3 target would mint a fresh marker
|
||||
// on every retry (rustfs#6823).
|
||||
replication_delete_remove_options(
|
||||
delete_replication_creates_marker(&dobj.delete_object),
|
||||
dobj.delete_object.delete_marker_mtime,
|
||||
),
|
||||
replication_delete_remove_options(dobj.delete_object.delete_marker, dobj.delete_object.delete_marker_mtime),
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -2811,10 +2768,6 @@ pub(crate) async fn replicate_object_with_outcome<S: ReplicationStorage>(
|
||||
};
|
||||
|
||||
let mut join_set = JoinSet::new();
|
||||
let mut rinfos = ReplicatedInfos {
|
||||
replication_timestamp: Some(OffsetDateTime::now_utc()),
|
||||
targets: Vec::with_capacity(tgt_arns.len()),
|
||||
};
|
||||
|
||||
for arn in tgt_arns {
|
||||
let Some(tgt_client) = ReplicationTargetStore::remote_target_client(&bucket, &arn).await else {
|
||||
@@ -2822,8 +2775,7 @@ pub(crate) async fn replicate_object_with_outcome<S: ReplicationStorage>(
|
||||
// stays unreachable would flood the log from the replication hot path. The
|
||||
// condition is reported once per pass by the site-replication reconciler and
|
||||
// once per rebuild by `update_all_targets`, which is where an operator can act
|
||||
// on it; the FAILED state below preserves retry visibility and the
|
||||
// aggregate result emits the user-visible failure event once.
|
||||
// on it; the per-object event below still records each dropped object.
|
||||
debug!(
|
||||
event = EVENT_RESYNC_RUNTIME_SKIPPED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
@@ -2832,9 +2784,15 @@ pub(crate) async fn replicate_object_with_outcome<S: ReplicationStorage>(
|
||||
object = %object,
|
||||
arn = %arn,
|
||||
reason = "target_client_missing",
|
||||
"Replication target client unavailable"
|
||||
"Replication rule has no bucket target for its destination ARN; object not replicated"
|
||||
);
|
||||
rinfos.targets.push(unavailable_object_target_info(&roi, &arn));
|
||||
send_local_event(EventArgs {
|
||||
event_name: EventName::ObjectReplicationNotTracked.to_string(),
|
||||
bucket_name: bucket.clone(),
|
||||
object: roi.to_object_info(),
|
||||
user_agent: "Internal: [Replication]".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
continue;
|
||||
};
|
||||
|
||||
@@ -2849,6 +2807,11 @@ pub(crate) async fn replicate_object_with_outcome<S: ReplicationStorage>(
|
||||
});
|
||||
}
|
||||
|
||||
let mut rinfos = ReplicatedInfos {
|
||||
replication_timestamp: Some(OffsetDateTime::now_utc()),
|
||||
targets: Vec::with_capacity(join_set.len()),
|
||||
};
|
||||
|
||||
while let Some(result) = join_set.join_next().await {
|
||||
match result {
|
||||
Ok(tgt_info) => {
|
||||
@@ -2954,23 +2917,6 @@ pub(crate) async fn replicate_object_with_outcome<S: ReplicationStorage>(
|
||||
(merged_state, state_persisted)
|
||||
}
|
||||
|
||||
fn unavailable_object_target_info(roi: &ReplicateObjectInfo, arn: &str) -> ReplicatedTargetInfo {
|
||||
ReplicatedTargetInfo {
|
||||
arn: arn.to_string(),
|
||||
size: roi.actual_size,
|
||||
replication_action: if roi.op_type == ReplicationType::Object {
|
||||
ReplicationAction::All
|
||||
} else {
|
||||
ReplicationAction::Metadata
|
||||
},
|
||||
op_type: roi.op_type,
|
||||
replication_status: ReplicationStatusType::Failed,
|
||||
prev_replication_status: roi.target_replication_status(arn),
|
||||
error: Some(TARGET_CLIENT_UNAVAILABLE_ERROR.to_string()),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
trait ReplicateObjectInfoExt {
|
||||
async fn replicate_object<S: ReplicationObjectIO>(
|
||||
&self,
|
||||
@@ -3005,14 +2951,11 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
|
||||
};
|
||||
|
||||
if ReplicationTargetStore::target_is_offline(&tgt_client).await {
|
||||
// The object is reported FAILED here, so this must be as loud as a
|
||||
// per-object put_object failure or the key never reaches the logs.
|
||||
warn!(
|
||||
debug!(
|
||||
event = EVENT_RESYNC_RUNTIME_SKIPPED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object,
|
||||
arn = %tgt_client.arn,
|
||||
reason = "target_offline",
|
||||
endpoint = %tgt_client.to_url(),
|
||||
@@ -3069,7 +3012,6 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object,
|
||||
arn = %tgt_client.arn,
|
||||
error = %e,
|
||||
reason = "object_reader_unavailable",
|
||||
@@ -3101,7 +3043,6 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object,
|
||||
arn = %tgt_client.arn,
|
||||
error = %e,
|
||||
reason = "actual_size_unavailable",
|
||||
@@ -3127,7 +3068,6 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object,
|
||||
arn = %tgt_client.arn,
|
||||
reason = "target_bucket_empty",
|
||||
"Skipping replication object target"
|
||||
@@ -3190,7 +3130,6 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object,
|
||||
arn = %tgt_client.arn,
|
||||
operation = "head_object_fallback",
|
||||
error = %e2,
|
||||
@@ -3206,7 +3145,6 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object,
|
||||
arn = %tgt_client.arn,
|
||||
operation = "head_object",
|
||||
error = %e,
|
||||
@@ -3236,7 +3174,6 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object,
|
||||
arn = %tgt_client.arn,
|
||||
operation = "build_put_options",
|
||||
error = %e,
|
||||
@@ -3506,14 +3443,11 @@ fn replicate_all_target_info(roi: &ReplicateObjectInfo, tgt_client: &TargetClien
|
||||
|
||||
/// Log and notify that replication was skipped because the target is offline.
|
||||
fn note_replicate_all_target_offline(roi: &ReplicateObjectInfo, bucket: &str, tgt_client: &TargetClient) {
|
||||
// The object is reported FAILED here, so this must be as loud as a
|
||||
// per-object put_object failure or the key never reaches the logs.
|
||||
warn!(
|
||||
debug!(
|
||||
event = EVENT_RESYNC_RUNTIME_SKIPPED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %roi.name,
|
||||
arn = %tgt_client.arn,
|
||||
target = %tgt_client.to_url(),
|
||||
reason = "target_offline",
|
||||
@@ -3551,7 +3485,6 @@ fn note_replicate_all_reader_unavailable(roi: &ReplicateObjectInfo, bucket: &str
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %roi.name,
|
||||
arn = %tgt_client.arn,
|
||||
error = %e,
|
||||
reason = "object_reader_unavailable",
|
||||
@@ -3575,7 +3508,6 @@ fn note_replicate_all_size_unavailable(bucket: &str, tgt_client: &TargetClient,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object_info.name,
|
||||
arn = %tgt_client.arn,
|
||||
error = %e,
|
||||
reason = "actual_size_unavailable",
|
||||
@@ -3598,7 +3530,6 @@ fn note_replicate_all_target_bucket_empty(bucket: &str, tgt_client: &TargetClien
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object_info.name,
|
||||
arn = %tgt_client.arn,
|
||||
reason = "target_bucket_empty",
|
||||
"Skipped replication because target bucket is empty"
|
||||
@@ -3807,7 +3738,6 @@ async fn resolve_replicate_all_action(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object,
|
||||
arn = %tgt_client.arn,
|
||||
error = %e2,
|
||||
reason = "head_object_fallback_failed",
|
||||
@@ -3838,7 +3768,6 @@ async fn resolve_replicate_all_action(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object,
|
||||
arn = %tgt_client.arn,
|
||||
error = %e,
|
||||
reason = "head_object_failed",
|
||||
@@ -3882,7 +3811,6 @@ fn fail_replicate_all_put_options(
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
bucket = %bucket,
|
||||
object = %object_info.name,
|
||||
arn = %tgt_client.arn,
|
||||
operation = "build_put_options",
|
||||
error = %e,
|
||||
@@ -4007,13 +3935,20 @@ struct MultipartReplicationContext<'a, S: ReplicationObjectIO> {
|
||||
}
|
||||
|
||||
async fn replicate_object_with_multipart<S: ReplicationObjectIO>(ctx: MultipartReplicationContext<'_, S>) -> std::io::Result<()> {
|
||||
let MultipartReplicationContext {
|
||||
storage,
|
||||
cli,
|
||||
src_bucket,
|
||||
dst_bucket,
|
||||
object,
|
||||
object_info,
|
||||
obj_opts,
|
||||
arn,
|
||||
put_opts,
|
||||
} = ctx;
|
||||
let mut attempts = 1;
|
||||
let upload_id = loop {
|
||||
match ctx
|
||||
.cli
|
||||
.create_multipart_upload(ctx.dst_bucket, ctx.object, &ctx.put_opts)
|
||||
.await
|
||||
{
|
||||
match cli.create_multipart_upload(dst_bucket, object, &put_opts).await {
|
||||
Ok(id) => {
|
||||
break id;
|
||||
}
|
||||
@@ -4030,71 +3965,6 @@ async fn replicate_object_with_multipart<S: ReplicationObjectIO>(ctx: MultipartR
|
||||
}
|
||||
};
|
||||
|
||||
let cli = ctx.cli.clone();
|
||||
let dst_bucket = ctx.dst_bucket;
|
||||
let object = ctx.object;
|
||||
let arn = ctx.arn;
|
||||
|
||||
let result = replicate_multipart_parts_and_complete(ctx, &upload_id).await;
|
||||
abort_multipart_on_failure(result, dst_bucket, object, &upload_id, arn, || async {
|
||||
cli.abort_multipart_upload(dst_bucket, object, &upload_id).await
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Best-effort abort of the target-side multipart upload once the transfer has
|
||||
/// failed past CreateMultipartUpload; without it every failed attempt leaves an
|
||||
/// invisible incomplete upload on the target that keeps billing for its parts.
|
||||
/// The abort outcome never replaces the transfer error: an abort failure is
|
||||
/// only logged and `result` is returned as-is.
|
||||
async fn abort_multipart_on_failure<F, Fut>(
|
||||
result: std::io::Result<()>,
|
||||
dst_bucket: &str,
|
||||
object: &str,
|
||||
upload_id: &str,
|
||||
arn: &str,
|
||||
abort: F,
|
||||
) -> std::io::Result<()>
|
||||
where
|
||||
F: FnOnce() -> Fut,
|
||||
Fut: std::future::Future<Output = std::result::Result<(), S3ClientError>>,
|
||||
{
|
||||
if result.is_ok() {
|
||||
return result;
|
||||
}
|
||||
if let Err(abort_err) = abort().await {
|
||||
warn!(
|
||||
event = EVENT_RESYNC_TARGET_OPERATION_FAILED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
|
||||
target_bucket = %dst_bucket,
|
||||
object = %object,
|
||||
arn = %arn,
|
||||
upload_id = %upload_id,
|
||||
operation = "abort_multipart_upload",
|
||||
error = %abort_err,
|
||||
"Replication target operation failed"
|
||||
);
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
async fn replicate_multipart_parts_and_complete<S: ReplicationObjectIO>(
|
||||
ctx: MultipartReplicationContext<'_, S>,
|
||||
upload_id: &str,
|
||||
) -> std::io::Result<()> {
|
||||
let MultipartReplicationContext {
|
||||
storage,
|
||||
cli,
|
||||
src_bucket,
|
||||
dst_bucket,
|
||||
object,
|
||||
object_info,
|
||||
obj_opts,
|
||||
arn,
|
||||
put_opts,
|
||||
} = ctx;
|
||||
|
||||
let mut uploaded_parts: Vec<CompletedPart> = Vec::new();
|
||||
|
||||
let mut header_size = replication_put_object_header_size(&put_opts);
|
||||
@@ -4133,7 +4003,7 @@ async fn replicate_multipart_parts_and_complete<S: ReplicationObjectIO>(
|
||||
.put_object_part(
|
||||
dst_bucket,
|
||||
object,
|
||||
upload_id,
|
||||
&upload_id,
|
||||
part_plan.part_number,
|
||||
part_plan.part_size,
|
||||
byte_stream,
|
||||
@@ -4158,7 +4028,7 @@ async fn replicate_multipart_parts_and_complete<S: ReplicationObjectIO>(
|
||||
.complete_multipart_upload(
|
||||
dst_bucket,
|
||||
object,
|
||||
upload_id,
|
||||
&upload_id,
|
||||
uploaded_parts,
|
||||
&replication_complete_multipart_options(
|
||||
actual_size,
|
||||
@@ -4183,88 +4053,6 @@ async fn replicate_multipart_parts_and_complete<S: ReplicationObjectIO>(
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::replication_filemeta_boundary::ReplicateTargetDecision;
|
||||
|
||||
#[test]
|
||||
fn unavailable_object_target_is_persisted_as_failed() {
|
||||
let arn = "arn:object-target";
|
||||
let roi = ReplicateObjectInfo {
|
||||
actual_size: 42,
|
||||
op_type: ReplicationType::Object,
|
||||
replication_status_internal: Some(format!("{arn}=PENDING;")),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let target_info = unavailable_object_target_info(&roi, arn);
|
||||
let merged = get_replication_state(
|
||||
&ReplicatedInfos {
|
||||
replication_timestamp: Some(OffsetDateTime::now_utc()),
|
||||
targets: vec![target_info.clone()],
|
||||
},
|
||||
&ReplicationState::default(),
|
||||
None,
|
||||
);
|
||||
|
||||
assert_eq!(target_info.replication_status, ReplicationStatusType::Failed);
|
||||
assert_eq!(target_info.prev_replication_status, ReplicationStatusType::Pending);
|
||||
assert_eq!(target_info.replication_action, ReplicationAction::All);
|
||||
assert_eq!(target_info.error.as_deref(), Some(TARGET_CLIENT_UNAVAILABLE_ERROR));
|
||||
assert_eq!(merged.targets.get(arn), Some(&ReplicationStatusType::Failed));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unavailable_delete_target_is_failed_without_overwriting_completed_state() {
|
||||
let arn = "arn:delete-target";
|
||||
let mut previous_state = ReplicationState::default();
|
||||
previous_state.targets.insert(arn.to_string(), ReplicationStatusType::Pending);
|
||||
let mut dobj = DeletedObjectReplicationInfo {
|
||||
delete_object: ReplicationDeletedObject {
|
||||
delete_marker: true,
|
||||
replication_state: Some(previous_state),
|
||||
..Default::default()
|
||||
},
|
||||
op_type: ReplicationType::Delete,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let failed = unavailable_delete_target_info(&dobj, arn);
|
||||
assert_eq!(failed.replication_status, ReplicationStatusType::Failed);
|
||||
assert_eq!(failed.prev_replication_status, ReplicationStatusType::Pending);
|
||||
assert_eq!(failed.error.as_deref(), Some(TARGET_CLIENT_UNAVAILABLE_ERROR));
|
||||
|
||||
dobj.delete_object
|
||||
.replication_state
|
||||
.as_mut()
|
||||
.expect("previous state should exist")
|
||||
.targets
|
||||
.insert(arn.to_string(), ReplicationStatusType::Completed);
|
||||
let completed = unavailable_delete_target_info(&dobj, arn);
|
||||
assert_eq!(completed.replication_status, ReplicationStatusType::Completed);
|
||||
assert!(completed.error.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unavailable_version_purge_target_is_persisted_as_failed() {
|
||||
let arn = "arn:purge-target";
|
||||
let mut previous_state = ReplicationState::default();
|
||||
previous_state
|
||||
.purge_targets
|
||||
.insert(arn.to_string(), VersionPurgeStatusType::Pending);
|
||||
let dobj = DeletedObjectReplicationInfo {
|
||||
delete_object: ReplicationDeletedObject {
|
||||
version_id: Some(Uuid::new_v4()),
|
||||
replication_state: Some(previous_state),
|
||||
..Default::default()
|
||||
},
|
||||
op_type: ReplicationType::Delete,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let target_info = unavailable_delete_target_info(&dobj, arn);
|
||||
|
||||
assert_eq!(target_info.version_purge_status, VersionPurgeStatusType::Failed);
|
||||
assert_eq!(target_info.error.as_deref(), Some(TARGET_CLIENT_UNAVAILABLE_ERROR));
|
||||
}
|
||||
|
||||
fn resync_target_state(resync_id: &str, status: ResyncStatusType, replicated_count: i64) -> TargetReplicationResyncStatus {
|
||||
TargetReplicationResyncStatus {
|
||||
resync_id: resync_id.to_string(),
|
||||
@@ -5353,42 +5141,4 @@ mod tests {
|
||||
assert!(resync_state_accepts_update(¤t, &matching));
|
||||
assert!(!resync_state_accepts_update(¤t, &stale));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn abort_multipart_on_failure_skips_abort_when_transfer_succeeded() {
|
||||
let aborted = Arc::new(AtomicBool::new(false));
|
||||
let flag = aborted.clone();
|
||||
|
||||
let result = abort_multipart_on_failure(Ok(()), "dst-bucket", "obj", "upload-1", "arn:dest", move || async move {
|
||||
flag.store(true, Ordering::SeqCst);
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
|
||||
assert!(result.is_ok());
|
||||
assert!(!aborted.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn abort_multipart_on_failure_aborts_and_keeps_transfer_error() {
|
||||
let aborted = Arc::new(AtomicBool::new(false));
|
||||
let flag = aborted.clone();
|
||||
|
||||
// The abort itself failing must not mask the transfer error.
|
||||
let result = abort_multipart_on_failure(
|
||||
Err(std::io::Error::other("transfer failed")),
|
||||
"dst-bucket",
|
||||
"obj",
|
||||
"upload-1",
|
||||
"arn:dest",
|
||||
move || async move {
|
||||
flag.store(true, Ordering::SeqCst);
|
||||
Err(S3ClientError::new("abort failed"))
|
||||
},
|
||||
)
|
||||
.await;
|
||||
|
||||
assert!(aborted.load(Ordering::SeqCst));
|
||||
assert_eq!(result.unwrap_err().to_string(), "transfer failed");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,8 +36,8 @@ use time::OffsetDateTime;
|
||||
use time::format_description::well_known::Rfc3339;
|
||||
|
||||
pub(crate) use crate::bucket::bucket_target_sys::{
|
||||
AdvancedPutOptions, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, S3ClientError,
|
||||
TargetClient, resolve_read_api_version_id,
|
||||
AdvancedPutOptions, HeadObjectSdkError, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient,
|
||||
resolve_read_api_version_id,
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use crate::bucket::target::BucketTarget;
|
||||
|
||||
@@ -25,8 +25,6 @@ use time::OffsetDateTime;
|
||||
use url::Url;
|
||||
|
||||
const REDACTED_CREDENTIAL: &str = "<redacted>";
|
||||
const GO_YEAR_ONE_START_UNIX_SECONDS: i64 = -62_135_596_800;
|
||||
const GO_YEAR_TWO_START_UNIX_SECONDS: i64 = -62_104_060_800;
|
||||
|
||||
#[derive(Deserialize, Serialize, Default, Clone)]
|
||||
pub struct Credentials {
|
||||
@@ -43,26 +41,6 @@ pub struct Credentials {
|
||||
}
|
||||
|
||||
impl Credentials {
|
||||
/// Returns the session token used for request signing.
|
||||
///
|
||||
/// MinIO-compatible payloads may carry an empty token. Treat whitespace-only
|
||||
/// values as absent without rewriting a real token, whose bytes are opaque.
|
||||
pub fn effective_session_token(&self) -> Option<&str> {
|
||||
self.session_token.as_deref().filter(|token| !token.trim().is_empty())
|
||||
}
|
||||
|
||||
/// Returns the credential expiry after normalizing Go's zero `time.Time`.
|
||||
///
|
||||
/// Go JSON encoders emit year 1 for an unset `time.Time`; persisted MinIO
|
||||
/// target metadata can therefore contain that sentinel even for static
|
||||
/// credentials.
|
||||
pub fn effective_expiration(&self) -> Option<Timestamp> {
|
||||
self.expiration.filter(|expiration| {
|
||||
let unix_seconds = expiration.as_second();
|
||||
!(GO_YEAR_ONE_START_UNIX_SECONDS..GO_YEAR_TWO_START_UNIX_SECONDS).contains(&unix_seconds)
|
||||
})
|
||||
}
|
||||
|
||||
pub fn redacted(&self) -> Self {
|
||||
Self {
|
||||
access_key: self.access_key.clone(),
|
||||
@@ -377,24 +355,6 @@ mod tests {
|
||||
use std::time::Duration;
|
||||
use time::OffsetDateTime;
|
||||
|
||||
#[test]
|
||||
fn credential_effective_values_normalize_only_compatibility_sentinels() {
|
||||
let mut credentials = Credentials {
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
session_token: Some(" ".to_string()),
|
||||
expiration: Some("0001-01-01T08:00:00+08:00".parse().expect("Go zero time should parse")),
|
||||
};
|
||||
|
||||
assert!(credentials.effective_session_token().is_none());
|
||||
assert!(credentials.effective_expiration().is_none());
|
||||
|
||||
credentials.session_token = Some(" opaque token ".to_string());
|
||||
credentials.expiration = Some("2099-01-01T00:00:00Z".parse().expect("future timestamp should parse"));
|
||||
assert_eq!(credentials.effective_session_token(), Some(" opaque token "));
|
||||
assert_eq!(credentials.effective_expiration(), credentials.expiration);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_bucket_target_json_deserialize() {
|
||||
let json = r#"
|
||||
|
||||
@@ -122,14 +122,6 @@ fn control_plane_failure(op: &str, bucket: Option<&str>, error_code: Option<i32>
|
||||
if error_code == Some(rustfs_protos::proto_gen::node_service::ControlPlaneErrorCode::ControlPlaneErrorNotInitialized as i32) {
|
||||
return Error::RemoteNotInitialized;
|
||||
}
|
||||
if error_code == Some(rustfs_protos::proto_gen::node_service::ControlPlaneErrorCode::ControlPlaneErrorInvalidArgument as i32)
|
||||
{
|
||||
return Error::InvalidArgument(
|
||||
"control-plane".to_string(),
|
||||
op.to_string(),
|
||||
error_info.unwrap_or_else(|| format!("{op}: peer rejected invalid argument without details")),
|
||||
);
|
||||
}
|
||||
match error_info {
|
||||
Some(msg) => Error::other(msg),
|
||||
None => peer_failure_without_details(op, bucket),
|
||||
@@ -733,7 +725,7 @@ impl PeerRestClient {
|
||||
/// never take it offline no matter what its message says. The substring
|
||||
/// fallback only covers failures that exist purely as text, such as the
|
||||
/// dial errors `get_client` wraps.
|
||||
pub(crate) fn is_network_like_error(err: &Error) -> bool {
|
||||
fn is_network_like_error(err: &Error) -> bool {
|
||||
if let Error::Io(io_err) = err
|
||||
&& let Some(status) = embedded_tonic_status(io_err)
|
||||
{
|
||||
@@ -2343,29 +2335,6 @@ mod tests {
|
||||
use rustfs_protos::proto_gen::node_service::ControlPlaneErrorCode;
|
||||
assert_eq!(ControlPlaneErrorCode::ControlPlaneErrorUnspecified as i32, 0);
|
||||
assert_eq!(ControlPlaneErrorCode::ControlPlaneErrorNotInitialized as i32, 1);
|
||||
assert_eq!(ControlPlaneErrorCode::ControlPlaneErrorInvalidArgument as i32, 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn control_plane_failure_preserves_typed_invalid_argument_reason() {
|
||||
use rustfs_protos::proto_gen::node_service::ControlPlaneErrorCode;
|
||||
|
||||
let reason = "durable unresolved-entry recovery requires pool metadata V2 or V3";
|
||||
let err = control_plane_failure(
|
||||
"start_decommission",
|
||||
None,
|
||||
Some(ControlPlaneErrorCode::ControlPlaneErrorInvalidArgument as i32),
|
||||
Some(reason.to_string()),
|
||||
);
|
||||
|
||||
assert!(
|
||||
matches!(
|
||||
err,
|
||||
Error::InvalidArgument(ref scope, ref operation, ref actual_reason)
|
||||
if scope == "control-plane" && operation == "start_decommission" && actual_reason == reason
|
||||
),
|
||||
"forwarded validation failures must remain typed and actionable"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -27,7 +27,7 @@ use rustfs_protos::{
|
||||
ConnectionEvictionLogLevel, evict_failed_connection_with_log_level, models::PingBodyBuilder,
|
||||
proto_gen::node_service::node_service_client::NodeServiceClient,
|
||||
};
|
||||
use std::{sync::OnceLock, time::Duration};
|
||||
use std::time::Duration;
|
||||
use tokio::time::timeout;
|
||||
use tonic::Request;
|
||||
use tonic::service::interceptor::InterceptedService;
|
||||
@@ -44,35 +44,11 @@ pub struct RemoteClient {
|
||||
}
|
||||
|
||||
impl RemoteClient {
|
||||
const ONLINE_CHECK_RESOURCE: &'static str = "health-lock-online";
|
||||
|
||||
pub fn new(endpoint: String) -> Self {
|
||||
Self { addr: endpoint }
|
||||
}
|
||||
|
||||
fn ping_body() -> Bytes {
|
||||
static BODY: OnceLock<Bytes> = OnceLock::new();
|
||||
BODY.get_or_init(|| {
|
||||
let mut fbb = flatbuffers::FlatBufferBuilder::new();
|
||||
let payload = fbb.create_vector(b"health-check");
|
||||
let mut builder = PingBodyBuilder::new(&mut fbb);
|
||||
builder.add_payload(payload);
|
||||
let root = builder.finish();
|
||||
fbb.finish(root, None);
|
||||
Bytes::copy_from_slice(fbb.finished_data())
|
||||
})
|
||||
.clone()
|
||||
}
|
||||
|
||||
fn build_ping_request() -> PingRequest {
|
||||
PingRequest {
|
||||
version: 1,
|
||||
body: Self::ping_body(),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn build_fresh_ping_request_for_test() -> PingRequest {
|
||||
let mut fbb = flatbuffers::FlatBufferBuilder::new();
|
||||
let payload = fbb.create_vector(b"health-check");
|
||||
let mut builder = PingBodyBuilder::new(&mut fbb);
|
||||
@@ -188,16 +164,6 @@ impl RemoteClient {
|
||||
)
|
||||
}
|
||||
|
||||
fn online_check_timeout() -> Duration {
|
||||
Duration::from_millis(
|
||||
rustfs_utils::get_env_u64(
|
||||
rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
|
||||
rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS,
|
||||
)
|
||||
.max(1),
|
||||
)
|
||||
}
|
||||
|
||||
async fn execute_rpc<T, F>(&self, op: &'static str, resource_summary: &str, future: F) -> std::result::Result<T, LockError>
|
||||
where
|
||||
F: std::future::Future<Output = std::result::Result<T, tonic::Status>>,
|
||||
@@ -581,37 +547,24 @@ impl LockClient for RemoteClient {
|
||||
}
|
||||
|
||||
async fn is_online(&self) -> bool {
|
||||
let online_timeout = Self::online_check_timeout();
|
||||
match timeout(online_timeout, async {
|
||||
let mut client = self.get_client().await?;
|
||||
let ping_req = Request::new(Self::build_ping_request());
|
||||
self.execute_rpc("ping", Self::ONLINE_CHECK_RESOURCE, client.ping(ping_req))
|
||||
.await?;
|
||||
Ok::<(), LockError>(())
|
||||
})
|
||||
.await
|
||||
{
|
||||
Ok(Ok(())) => {
|
||||
debug!(addr = %self.addr, timeout_ms = online_timeout.as_millis(), "remote lock client is online");
|
||||
// Use Ping interface to test if remote service is online
|
||||
let mut client = match self.get_client().await {
|
||||
Ok(client) => client,
|
||||
Err(_) => {
|
||||
info!("remote client {} connection failed", self.addr);
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
let ping_req = Request::new(Self::build_ping_request());
|
||||
|
||||
match client.ping(ping_req).await {
|
||||
Ok(_) => {
|
||||
info!("remote client {} is online", self.addr);
|
||||
true
|
||||
}
|
||||
Ok(Err(err)) => {
|
||||
debug!(
|
||||
addr = %self.addr,
|
||||
timeout_ms = online_timeout.as_millis(),
|
||||
error = %err,
|
||||
"remote lock client online check failed"
|
||||
);
|
||||
false
|
||||
}
|
||||
Err(_) => {
|
||||
let reason = format!("online check timed out after {:?}", online_timeout);
|
||||
warn!(
|
||||
addr = %self.addr,
|
||||
timeout_ms = online_timeout.as_millis(),
|
||||
"remote lock client online check timed out"
|
||||
);
|
||||
self.evict_connection("ping", &reason, Self::ONLINE_CHECK_RESOURCE).await;
|
||||
info!("remote client {} ping failed", self.addr);
|
||||
false
|
||||
}
|
||||
}
|
||||
@@ -698,15 +651,6 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cached_ping_request_matches_fresh_flatbuffer_payload() {
|
||||
let cached = RemoteClient::build_ping_request();
|
||||
let fresh = RemoteClient::build_fresh_ping_request_for_test();
|
||||
|
||||
assert_eq!(cached.version, fresh.version);
|
||||
assert_eq!(cached.body, fresh.body);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn test_remote_client_acquire_lock_uses_rpc_timeout_and_evicts_connection() {
|
||||
@@ -835,48 +779,6 @@ mod tests {
|
||||
accept_task.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn test_remote_client_is_online_uses_health_timeout_and_evicts_connection() {
|
||||
ensure_test_rpc_secret();
|
||||
let Some((addr, accept_task)) = spawn_hanging_listener().await else {
|
||||
return;
|
||||
};
|
||||
cache_lazy_channel(&addr).await;
|
||||
assert!(runtime_sources::test_node_channel_is_cached(&addr).await);
|
||||
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50")),
|
||||
(rustfs_config::ENV_OBJECT_LOCK_RPC_TIMEOUT_MS, Some("1000")),
|
||||
],
|
||||
async {
|
||||
let client = RemoteClient::new(addr.clone());
|
||||
let started_at = tokio::time::Instant::now();
|
||||
|
||||
let online = client.is_online().await;
|
||||
let elapsed = started_at.elapsed();
|
||||
|
||||
assert!(!online, "hanging remote lock peer must not be reported online");
|
||||
assert!(
|
||||
elapsed >= Duration::from_millis(40),
|
||||
"remote online check should honor configured health timeout, got {elapsed:?}"
|
||||
);
|
||||
assert!(
|
||||
elapsed < Duration::from_secs(1),
|
||||
"health timeout should keep readiness probes bounded, got {elapsed:?}"
|
||||
);
|
||||
assert!(
|
||||
!runtime_sources::test_node_channel_is_cached(&addr).await,
|
||||
"online-check timeout should evict cached connection"
|
||||
);
|
||||
},
|
||||
)
|
||||
.await;
|
||||
|
||||
accept_task.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn test_remote_client_refresh_tonic_error_evicts_connection() {
|
||||
@@ -1004,21 +906,4 @@ mod tests {
|
||||
assert_eq!(RemoteClient::rpc_timeout(), Duration::from_millis(1));
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn test_remote_client_online_timeout_honors_configured_deadline() {
|
||||
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, None::<&str>, || {
|
||||
assert_eq!(
|
||||
RemoteClient::online_check_timeout(),
|
||||
Duration::from_millis(rustfs_config::DEFAULT_HEALTH_LOCK_ONLINE_TIMEOUT_MS)
|
||||
);
|
||||
});
|
||||
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("50"), || {
|
||||
assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(50));
|
||||
});
|
||||
temp_env::with_var(rustfs_config::ENV_HEALTH_LOCK_ONLINE_TIMEOUT_MS, Some("0"), || {
|
||||
assert_eq!(RemoteClient::online_check_timeout(), Duration::from_millis(1));
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,28 +12,35 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::target_defaults::{amqp_kvs, kafka_kvs, mysql_kvs, nats_kvs, postgres_kvs, pulsar_kvs, redis_kvs};
|
||||
use rustfs_config::audit::AUDIT_REDIS_DEFAULT_CHANNEL;
|
||||
use rustfs_config::server_config::{KV, KVS};
|
||||
use rustfs_config::{
|
||||
COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY, EVENT_DEFAULT_DIR, EnableState, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD,
|
||||
MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY,
|
||||
MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, WEBHOOK_AUTH_TOKEN,
|
||||
WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT,
|
||||
WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, WEBHOOK_SKIP_TLS_VERIFY,
|
||||
AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
|
||||
AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
|
||||
EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
|
||||
KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
|
||||
KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR,
|
||||
MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
|
||||
MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT,
|
||||
MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT,
|
||||
MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS,
|
||||
NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR,
|
||||
NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
|
||||
NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
|
||||
POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
|
||||
PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
|
||||
PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
|
||||
REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
|
||||
REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
|
||||
REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
|
||||
REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY,
|
||||
WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL,
|
||||
WEBHOOK_SKIP_TLS_VERIFY,
|
||||
};
|
||||
use std::sync::LazyLock;
|
||||
|
||||
#[allow(clippy::declare_interior_mutable_const)]
|
||||
/// Default KVS for audit webhook settings.
|
||||
///
|
||||
/// `WEBHOOK_BATCH_SIZE`/`WEBHOOK_MAX_RETRY`/`WEBHOOK_RETRY_INTERVAL`/`WEBHOOK_HTTP_TIMEOUT`
|
||||
/// exist here but not in [`crate::config::notify::DEFAULT_NOTIFY_WEBHOOK_KVS`]. This mirrors
|
||||
/// MinIO upstream: `internal/logger/config.go`'s `DefaultAuditWebhookKVS` carries the same
|
||||
/// four keys with the same defaults (`"1"`/`"0"`/`"3s"`/`"5s"`), while
|
||||
/// `internal/config/notify/parse.go`'s `DefaultWebhookKVS` (bucket event notifications) does
|
||||
/// not — the notify webhook delivery path never supported them. Not a copy/paste gap
|
||||
/// (backlog#2054).
|
||||
pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
@@ -49,7 +56,7 @@ pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KV {
|
||||
key: WEBHOOK_AUTH_TOKEN.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true, // Sensitive field; matches notify's webhook auth_token (backlog#2054)
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: WEBHOOK_CLIENT_CERT.to_owned(),
|
||||
@@ -111,15 +118,6 @@ pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
|
||||
#[allow(clippy::declare_interior_mutable_const)]
|
||||
/// Default KVS for audit MQTT settings.
|
||||
///
|
||||
/// `MQTT_QOS`/`MQTT_KEEP_ALIVE_INTERVAL`/`MQTT_RECONNECT_INTERVAL` default to a stronger
|
||||
/// delivery posture here (`"1"`/`"60s"`/`"5s"`) than
|
||||
/// [`crate::config::notify::DEFAULT_NOTIFY_MQTT_KVS`] (`"0"`/`"0s"`/`"0s"`, which matches
|
||||
/// MinIO's own `DefaultMQTTKVS` in `internal/config/notify/parse.go` byte-for-byte). MinIO has
|
||||
/// no MQTT audit target to compare against — audit-over-MQTT is a RustFS-original addition —
|
||||
/// so this divergence cannot be checked against upstream; it is intentional (audit favors
|
||||
/// at-least-once delivery and faster reconnect over notify's opt-in defaults), not a
|
||||
/// copy/paste gap (backlog#2054).
|
||||
pub static DEFAULT_AUDIT_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
@@ -210,18 +208,542 @@ pub static DEFAULT_AUDIT_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
])
|
||||
});
|
||||
|
||||
// The remaining targets declare the same defaults as notify, so both sides build them from
|
||||
// `target_defaults`. Redis and mysql pass in the single default that audit and notify disagree on.
|
||||
pub static DEFAULT_AUDIT_AMQP_KVS: LazyLock<KVS> = LazyLock::new(amqp_kvs);
|
||||
pub static DEFAULT_AUDIT_AMQP_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_URL.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_EXCHANGE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_ROUTING_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_MANDATORY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_PERSISTENT.to_owned(),
|
||||
value: EnableState::On.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_AUDIT_NATS_KVS: LazyLock<KVS> = LazyLock::new(nats_kvs);
|
||||
pub static DEFAULT_AUDIT_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_ADDRESS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_SUBJECT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TOKEN.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_CREDENTIALS_FILE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_REQUIRED.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_STREAM_NAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(),
|
||||
value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_AUDIT_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(pulsar_kvs);
|
||||
pub static DEFAULT_AUDIT_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_BROKER.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TOPIC.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_AUTH_TOKEN.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_ALLOW_INSECURE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_HOSTNAME_VERIFICATION.to_owned(),
|
||||
value: EnableState::On.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_AUDIT_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| redis_kvs(AUDIT_REDIS_DEFAULT_CHANNEL));
|
||||
pub static DEFAULT_AUDIT_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_URL.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_CHANNEL.to_owned(),
|
||||
value: AUDIT_REDIS_DEFAULT_CHANNEL.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_KEEP_ALIVE_INTERVAL.to_owned(),
|
||||
value: "15".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MAX_RETRY_ATTEMPTS.to_owned(),
|
||||
value: "3".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_RECONNECT_RETRY_ATTEMPTS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MIN_RETRY_DELAY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MAX_RETRY_DELAY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_CONNECTION_TIMEOUT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_RESPONSE_TIMEOUT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_PIPELINE_BUFFER_SIZE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_POLICY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_ALLOW_INSECURE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_AUDIT_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(postgres_kvs);
|
||||
pub static DEFAULT_AUDIT_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_DSN_STRING.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TABLE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_FORMAT.to_owned(),
|
||||
value: "namespace".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_REQUIRED.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(kafka_kvs);
|
||||
pub static DEFAULT_AUDIT_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_BROKERS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TOPIC.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_ACKS.to_owned(),
|
||||
value: "1".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_MECHANISM.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_AUDIT_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| mysql_kvs("rustfs_audit_logs"));
|
||||
pub static DEFAULT_AUDIT_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_DSN_STRING.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TABLE.to_owned(),
|
||||
value: "rustfs_audit_logs".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_FORMAT.to_owned(),
|
||||
value: "access".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_MAX_OPEN_CONNECTIONS.to_owned(),
|
||||
value: "2".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
@@ -800,22 +800,11 @@ where
|
||||
if log_error {
|
||||
error!("save_config_with_opts: err: {:?}, file: {}", err, file);
|
||||
}
|
||||
Err(map_system_metadata_write_error(err, file))
|
||||
Err(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A system metadata volume outage must remain retryable instead of being
|
||||
/// exposed as the user-facing bucket-not-found response.
|
||||
pub(crate) fn map_system_metadata_write_error(err: Error, file: &str) -> Error {
|
||||
match err {
|
||||
Error::BucketNotFound(_) | Error::VolumeNotFound => {
|
||||
Error::InsufficientWriteQuorum(RUSTFS_META_BUCKET.to_string(), file.to_string())
|
||||
}
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
|
||||
fn new_server_config() -> Config {
|
||||
Config::new()
|
||||
}
|
||||
@@ -2372,7 +2361,6 @@ where
|
||||
scan_mode: HealScanMode::Deep,
|
||||
update_parity: false,
|
||||
no_lock: false,
|
||||
read_repair: false,
|
||||
pool: None,
|
||||
set: None,
|
||||
};
|
||||
@@ -2805,16 +2793,15 @@ where
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, build_scalar_config_object,
|
||||
config_task_join_error, configs_semantically_equal, decode_server_config_blob, encode_server_config_blob,
|
||||
heal_config_descriptor, is_standard_object_server_config, lookup_configs, map_system_metadata_write_error,
|
||||
new_and_save_server_config, read_config, read_config_no_lock_preserve_empty_with_metadata, read_config_preserve_empty,
|
||||
read_config_with_metadata, read_config_without_migrate, read_server_config_snapshot, save_config_with_opts_inner,
|
||||
SERVER_CONFIG_LOCK, ServerConfigSnapshot, apply_dynamic_config_for_sub_sys_with, config_task_join_error,
|
||||
configs_semantically_equal, decode_server_config_blob, encode_server_config_blob, is_standard_object_server_config,
|
||||
lookup_configs, new_and_save_server_config, read_config, read_config_no_lock_preserve_empty_with_metadata,
|
||||
read_config_preserve_empty, read_config_with_metadata, read_config_without_migrate, read_server_config_snapshot,
|
||||
save_server_config, save_server_config_snapshot, save_server_config_snapshot_with_generation,
|
||||
server_config_transaction_lock_path, should_warn_ignored_scalar_section, storage_class_kvs_mut,
|
||||
};
|
||||
use crate::config::{audit, heal, notify, oidc, scanner};
|
||||
use crate::disk::{RUSTFS_META_BUCKET, endpoint::Endpoint};
|
||||
use crate::disk::endpoint::Endpoint;
|
||||
use crate::error::{Error, Result};
|
||||
use crate::layout::endpoints::SetupType;
|
||||
use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader};
|
||||
@@ -2846,72 +2833,6 @@ mod tests {
|
||||
assert!(rendered.contains("panicked"));
|
||||
assert!(!rendered.contains("do-not-expose-payload"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn system_metadata_volume_failures_map_to_retryable_write_errors() {
|
||||
for error in [Error::VolumeNotFound, Error::BucketNotFound(RUSTFS_META_BUCKET.to_string())] {
|
||||
assert_eq!(
|
||||
map_system_metadata_write_error(error, "buckets/example/.metadata.bin"),
|
||||
Error::InsufficientWriteQuorum(RUSTFS_META_BUCKET.to_string(), "buckets/example/.metadata.bin".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
let other = Error::other("metadata encoding failed");
|
||||
assert_eq!(map_system_metadata_write_error(other.clone(), "buckets/example/.metadata.bin"), other);
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct MetadataWriteStore {
|
||||
error: Option<Error>,
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl crate::storage_api_contracts::object::ObjectIO for MetadataWriteStore {
|
||||
type Error = Error;
|
||||
type RangeSpec = HTTPRangeSpec;
|
||||
type HeaderMap = HeaderMap;
|
||||
type ObjectOptions = ObjectOptions;
|
||||
type ObjectInfo = ObjectInfo;
|
||||
type GetObjectReader = GetObjectReader;
|
||||
type PutObjectReader = PutObjReader;
|
||||
|
||||
async fn get_object_reader(
|
||||
&self,
|
||||
_bucket: &str,
|
||||
_object: &str,
|
||||
_range: Option<Self::RangeSpec>,
|
||||
_headers: Self::HeaderMap,
|
||||
_opts: &Self::ObjectOptions,
|
||||
) -> core::result::Result<Self::GetObjectReader, Self::Error> {
|
||||
Err(Error::FileNotFound)
|
||||
}
|
||||
|
||||
async fn put_object(
|
||||
&self,
|
||||
_bucket: &str,
|
||||
_object: &str,
|
||||
_data: &mut Self::PutObjectReader,
|
||||
_opts: &Self::ObjectOptions,
|
||||
) -> core::result::Result<Self::ObjectInfo, Self::Error> {
|
||||
Err(self.error.clone().expect("test store error should be configured"))
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn save_config_preserves_retryable_system_volume_errors() {
|
||||
let store = Arc::new(MetadataWriteStore {
|
||||
error: Some(Error::BucketNotFound(RUSTFS_META_BUCKET.to_string())),
|
||||
});
|
||||
let error =
|
||||
save_config_with_opts_inner(store, "buckets/example/.metadata.bin", Vec::new(), &ObjectOptions::default(), false)
|
||||
.await
|
||||
.expect_err("missing metadata volume must fail");
|
||||
|
||||
assert_eq!(
|
||||
error,
|
||||
Error::InsufficientWriteQuorum(RUSTFS_META_BUCKET.to_string(), "buckets/example/.metadata.bin".to_string())
|
||||
);
|
||||
}
|
||||
use rustfs_lock::client::LockClient;
|
||||
use rustfs_lock::client::local::LocalClient;
|
||||
use rustfs_lock::{LockError, LockInfo, LockResponse, LockStats};
|
||||
@@ -3620,31 +3541,6 @@ mod tests {
|
||||
cfg
|
||||
}
|
||||
|
||||
/// `Config::new()` (and every decode built on it) reads the process-global
|
||||
/// `rustfs_config::server_config::DEFAULT_KVS` OnceLock at call time, and
|
||||
/// other tests in this binary register it via `crate::config::init()`
|
||||
/// mid-run. Equality assertions must therefore normalize both sides with
|
||||
/// one snapshot taken after both configs exist, never against a later
|
||||
/// `Config::new()`.
|
||||
fn default_kvs_snapshot() -> Option<&'static std::collections::HashMap<String, KVS>> {
|
||||
rustfs_config::server_config::DEFAULT_KVS.get()
|
||||
}
|
||||
|
||||
/// Fills the default sections `cfg` is missing from an explicit
|
||||
/// [`default_kvs_snapshot`], mirroring `Config::set_defaults`.
|
||||
fn filled_with_default_kvs(mut cfg: Config, snapshot: Option<&std::collections::HashMap<String, KVS>>) -> Config {
|
||||
if let Some(defaults) = snapshot {
|
||||
for (sub_sys, kvs) in defaults {
|
||||
cfg.0
|
||||
.entry(sub_sys.clone())
|
||||
.or_default()
|
||||
.entry(DEFAULT_DELIMITER.to_string())
|
||||
.or_insert_with(|| kvs.clone());
|
||||
}
|
||||
}
|
||||
cfg
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_external_scanner_config_decodes_with_defaults() {
|
||||
let cfg =
|
||||
@@ -3734,9 +3630,7 @@ mod tests {
|
||||
}"#;
|
||||
|
||||
let cfg = decode_server_config_blob(seed).expect("root heal null should mean no persisted override");
|
||||
// The heal section may hold registered defaults, so assert on the
|
||||
// semantic diff instead of the section's presence.
|
||||
assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
|
||||
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
|
||||
assert!(!is_standard_object_server_config(seed));
|
||||
|
||||
let encoded = encode_server_config_blob(&cfg, Some(seed)).expect("legacy seed should canonicalize on an authorized save");
|
||||
@@ -3771,12 +3665,7 @@ mod tests {
|
||||
let input = format!(r#"{{"version":"33","storageclass":{{"standard":"","rrs":""}},{section}}}"#);
|
||||
let cfg = decode_server_config_blob(input.as_bytes())
|
||||
.unwrap_or_else(|err| panic!("legacy scalar section {section} should be ignored, got: {err}"));
|
||||
let snapshot = default_kvs_snapshot();
|
||||
assert_eq!(
|
||||
filled_with_default_kvs(cfg.clone(), snapshot),
|
||||
filled_with_default_kvs(base.clone(), snapshot),
|
||||
"ignored section {section} must contribute no overrides"
|
||||
);
|
||||
assert_eq!(cfg, base, "ignored section {section} must contribute no overrides");
|
||||
assert!(
|
||||
!is_standard_object_server_config(input.as_bytes()),
|
||||
"seed with {section} must not count as standard so a save rewrites it"
|
||||
@@ -3854,19 +3743,12 @@ mod tests {
|
||||
fn valid_heal_object_and_kvs_array_shapes_remain_accepted() {
|
||||
let empty_object = br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":{}}"#;
|
||||
let cfg = decode_server_config_blob(empty_object).expect("empty heal object should decode as no override");
|
||||
// The heal section may hold registered defaults, so assert on the
|
||||
// semantic diff instead of the section's presence.
|
||||
assert!(build_scalar_config_object(&cfg, heal_config_descriptor()).is_empty());
|
||||
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_none());
|
||||
|
||||
let kvs_array =
|
||||
br#"{"version":"33","storageclass":{"standard":"","rrs":""},"heal":[{"key":"bitrot_cycle","value":"off"}]}"#;
|
||||
let cfg = decode_server_config_blob(kvs_array).expect("heal KVS array should decode");
|
||||
assert_eq!(
|
||||
build_scalar_config_object(&cfg, heal_config_descriptor())
|
||||
.get(HEAL_BITROT_CYCLE)
|
||||
.and_then(Value::as_str),
|
||||
Some("off")
|
||||
);
|
||||
assert!(cfg.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER).is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -5018,12 +4900,8 @@ mod tests {
|
||||
fn test_fallback_returns_default_config_when_recovery_enabled() {
|
||||
let cfg = fallback_server_config_after_corruption(corrupt_config_error(), "config/config.json", true)
|
||||
.expect("recovery enabled must fall back to the default config");
|
||||
let snapshot = default_kvs_snapshot();
|
||||
assert!(
|
||||
configs_semantically_equal(
|
||||
&filled_with_default_kvs(cfg, snapshot),
|
||||
&filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
|
||||
),
|
||||
configs_semantically_equal(&cfg, &Config::new()),
|
||||
"fallback config should be the default server config"
|
||||
);
|
||||
}
|
||||
@@ -5342,7 +5220,7 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn server_config_snapshot_serializes_read_modify_write_transactions() {
|
||||
let baseline = encode_server_config_blob(&Config::new(), None).expect("baseline config should encode");
|
||||
let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(baseline.clone()), None));
|
||||
let store = Arc::new(RecoveryMockStore::new(RecoveryReadState::Blob(baseline), None));
|
||||
let first = read_server_config_snapshot(store.clone())
|
||||
.await
|
||||
.expect("first config snapshot");
|
||||
@@ -5356,11 +5234,7 @@ mod tests {
|
||||
.await
|
||||
.expect("second transaction should acquire after the first snapshot is dropped")
|
||||
.expect("second config snapshot");
|
||||
// Compare raw bytes against the baseline blob rather than a fresh
|
||||
// Config::new(): the process-global DEFAULT_KVS can be registered by a
|
||||
// sibling test mid-run, which would make a Config::new() evaluated here
|
||||
// diverge from the baseline encoded above.
|
||||
assert_eq!(second.raw.as_deref(), Some(baseline.as_slice()));
|
||||
assert!(configs_semantically_equal(&second.config, &Config::new()));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -5640,12 +5514,8 @@ mod tests {
|
||||
.expect("unrecoverable corruption should fall back to the default config");
|
||||
|
||||
assert_eq!(store.heal_calls.load(Ordering::SeqCst), 1, "heal should be attempted before falling back");
|
||||
let snapshot = default_kvs_snapshot();
|
||||
assert!(
|
||||
configs_semantically_equal(
|
||||
&filled_with_default_kvs(cfg, snapshot),
|
||||
&filled_with_default_kvs(Config(std::collections::HashMap::new()), snapshot)
|
||||
),
|
||||
configs_semantically_equal(&cfg, &Config::new()),
|
||||
"fallback config should be the default server config"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -21,7 +21,6 @@ mod notify;
|
||||
mod oidc;
|
||||
mod scanner;
|
||||
pub mod storageclass;
|
||||
mod target_defaults;
|
||||
|
||||
use crate::error::Result;
|
||||
use crate::store::ECStore;
|
||||
|
||||
@@ -12,26 +12,34 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::target_defaults::{amqp_kvs, kafka_kvs, mysql_kvs, nats_kvs, postgres_kvs, pulsar_kvs, redis_kvs};
|
||||
use rustfs_config::notify::NOTIFY_REDIS_DEFAULT_CHANNEL;
|
||||
use rustfs_config::server_config::{KV, KVS};
|
||||
use rustfs_config::{
|
||||
COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY, EVENT_DEFAULT_DIR, EnableState, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD,
|
||||
MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY,
|
||||
MQTT_TLS_POLICY, MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, WEBHOOK_AUTH_TOKEN,
|
||||
WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT,
|
||||
WEBHOOK_SKIP_TLS_VERIFY,
|
||||
AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
|
||||
AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
|
||||
EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
|
||||
KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
|
||||
KAFKA_TLS_ENABLE, KAFKA_TOPIC, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR,
|
||||
MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TLS_CA, MQTT_TLS_CLIENT_CERT, MQTT_TLS_CLIENT_KEY, MQTT_TLS_POLICY,
|
||||
MQTT_TLS_TRUST_LEAF_AS_CA, MQTT_TOPIC, MQTT_USERNAME, MQTT_WS_PATH_ALLOWLIST, MYSQL_DSN_STRING, MYSQL_FORMAT,
|
||||
MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR, MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT,
|
||||
MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS, NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS,
|
||||
NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE, NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR,
|
||||
NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT, NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN,
|
||||
NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR, POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE,
|
||||
POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY, POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER,
|
||||
PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT, PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA,
|
||||
PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL, REDIS_CONNECTION_TIMEOUT,
|
||||
REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY, REDIS_PASSWORD,
|
||||
REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS, REDIS_RESPONSE_TIMEOUT,
|
||||
REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY, REDIS_TLS_POLICY, REDIS_URL,
|
||||
REDIS_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CA, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT,
|
||||
WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, WEBHOOK_SKIP_TLS_VERIFY,
|
||||
};
|
||||
use std::sync::LazyLock;
|
||||
|
||||
/// The default configuration collection of webhooks，
|
||||
/// Initialized only once during the program life cycle, enabling high-performance lazy loading.
|
||||
///
|
||||
/// This table has no `batch_size`/`max_retry`/`retry_interval`/`http_timeout` keys, unlike
|
||||
/// [`crate::config::audit::DEFAULT_AUDIT_WEBHOOK_KVS`] — matching MinIO upstream, whose
|
||||
/// `internal/config/notify/parse.go` `DefaultWebhookKVS` (bucket event notifications) also
|
||||
/// omits them while `internal/logger/config.go`'s `DefaultAuditWebhookKVS` carries them.
|
||||
/// Intentional, not a copy/paste gap (backlog#2054).
|
||||
pub static DEFAULT_NOTIFY_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
@@ -89,12 +97,6 @@ pub static DEFAULT_NOTIFY_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
});
|
||||
|
||||
/// MQTT's default configuration collection
|
||||
///
|
||||
/// `MQTT_QOS`/`MQTT_KEEP_ALIVE_INTERVAL`/`MQTT_RECONNECT_INTERVAL` default to `"0"`/`"0s"`/`"0s"`
|
||||
/// here, matching MinIO's `DefaultMQTTKVS` in `internal/config/notify/parse.go`
|
||||
/// byte-for-byte — this table is a faithful port. [`crate::config::audit::DEFAULT_AUDIT_MQTT_KVS`]
|
||||
/// uses stronger, RustFS-original defaults instead (MinIO has no MQTT audit target to compare
|
||||
/// against); that divergence is intentional, not a copy/paste gap (backlog#2054).
|
||||
pub static DEFAULT_NOTIFY_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
@@ -186,17 +188,543 @@ pub static DEFAULT_NOTIFY_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_NOTIFY_AMQP_KVS: LazyLock<KVS> = LazyLock::new(amqp_kvs);
|
||||
pub static DEFAULT_NOTIFY_AMQP_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_URL.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_EXCHANGE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_ROUTING_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_MANDATORY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_PERSISTENT.to_owned(),
|
||||
value: EnableState::On.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: AMQP_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock<KVS> = LazyLock::new(nats_kvs);
|
||||
pub static DEFAULT_NOTIFY_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_ADDRESS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_SUBJECT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TOKEN.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_CREDENTIALS_FILE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: NATS_TLS_REQUIRED.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_STREAM_NAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: NATS_JETSTREAM_ACK_TIMEOUT_SECS.to_owned(),
|
||||
value: NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_NOTIFY_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(pulsar_kvs);
|
||||
pub static DEFAULT_NOTIFY_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_BROKER.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TOPIC.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_AUTH_TOKEN.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_ALLOW_INSECURE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_TLS_HOSTNAME_VERIFICATION.to_owned(),
|
||||
value: EnableState::On.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: PULSAR_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_NOTIFY_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| redis_kvs(NOTIFY_REDIS_DEFAULT_CHANNEL));
|
||||
pub static DEFAULT_NOTIFY_REDIS_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_URL.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_CHANNEL.to_owned(),
|
||||
value: NOTIFY_REDIS_DEFAULT_CHANNEL.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_KEEP_ALIVE_INTERVAL.to_owned(),
|
||||
value: "15".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MAX_RETRY_ATTEMPTS.to_owned(),
|
||||
value: "3".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_RECONNECT_RETRY_ATTEMPTS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MIN_RETRY_DELAY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_MAX_RETRY_DELAY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_CONNECTION_TIMEOUT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_RESPONSE_TIMEOUT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_PIPELINE_BUFFER_SIZE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_POLICY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: REDIS_TLS_ALLOW_INSECURE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_NOTIFY_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(postgres_kvs);
|
||||
pub static DEFAULT_NOTIFY_POSTGRES_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_DSN_STRING.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TABLE.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_FORMAT.to_owned(),
|
||||
value: "namespace".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_REQUIRED.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: POSTGRES_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(kafka_kvs);
|
||||
pub static DEFAULT_NOTIFY_KAFKA_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_BROKERS.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TOPIC.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_ACKS.to_owned(),
|
||||
value: "1".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_ENABLE.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_MECHANISM.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_USERNAME.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_SASL_PASSWORD.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: KAFKA_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
/// MySQL notification target default configuration
|
||||
pub static DEFAULT_NOTIFY_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| mysql_kvs("rustfs_events"));
|
||||
pub static DEFAULT_NOTIFY_MYSQL_KVS: LazyLock<KVS> = LazyLock::new(|| {
|
||||
KVS(vec![
|
||||
KV {
|
||||
key: ENABLE_KEY.to_owned(),
|
||||
value: EnableState::Off.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_DSN_STRING.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TABLE.to_owned(),
|
||||
value: "rustfs_events".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_FORMAT.to_owned(),
|
||||
value: "access".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CA.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CLIENT_CERT.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_TLS_CLIENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: true,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_QUEUE_DIR.to_owned(),
|
||||
value: EVENT_DEFAULT_DIR.to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_QUEUE_LIMIT.to_owned(),
|
||||
value: DEFAULT_LIMIT.to_string(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: MYSQL_MAX_OPEN_CONNECTIONS.to_owned(),
|
||||
value: "2".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
KV {
|
||||
key: COMMENT_KEY.to_owned(),
|
||||
value: "".to_owned(),
|
||||
hidden_if_empty: false,
|
||||
},
|
||||
])
|
||||
});
|
||||
|
||||
@@ -246,7 +246,16 @@ impl Config {
|
||||
}
|
||||
|
||||
let shard_size = shard_size as usize;
|
||||
let inline_block = self.effective_inline_block(data_shards);
|
||||
// Keep the historical two-data-shard object budget while preventing
|
||||
// wider EC layouts from multiplying the maximum inline object size.
|
||||
// Use div_ceil to match the shard_file_size calculation (which also uses
|
||||
// div_ceil), avoiding a 1-byte rounding discrepancy that prevents inline
|
||||
// for objects right at the threshold.
|
||||
let inline_block = if self.initialized && self.inline_block_explicit {
|
||||
self.inline_block
|
||||
} else {
|
||||
DEFAULT_INLINE_OBJECT_BUDGET.div_ceil(data_shards).min(DEFAULT_INLINE_BLOCK)
|
||||
};
|
||||
|
||||
if versioned {
|
||||
shard_size <= inline_block / 8
|
||||
@@ -255,27 +264,6 @@ impl Config {
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the per-shard inline budget used by both write admission and
|
||||
/// legacy read fallback.
|
||||
///
|
||||
/// The default budget is scaled by the number of data shards so a wider EC
|
||||
/// layout does not silently increase the maximum inline object size. An
|
||||
/// explicitly configured `inline_block` remains a fixed per-shard limit for
|
||||
/// compatibility with deployments that opted into the historical policy.
|
||||
pub(crate) fn effective_inline_block(&self, data_shards: usize) -> usize {
|
||||
if data_shards == 0 {
|
||||
return 0;
|
||||
}
|
||||
|
||||
if self.initialized && self.inline_block_explicit {
|
||||
self.inline_block
|
||||
} else {
|
||||
// Keep the historical two-data-shard object budget while preventing
|
||||
// wider EC layouts from multiplying the maximum inline object size.
|
||||
DEFAULT_INLINE_OBJECT_BUDGET.div_ceil(data_shards).min(DEFAULT_INLINE_BLOCK)
|
||||
}
|
||||
}
|
||||
|
||||
pub fn inline_block(&self) -> usize {
|
||||
if !self.initialized {
|
||||
DEFAULT_INLINE_BLOCK
|
||||
@@ -614,51 +602,6 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn should_inline_keeps_ec8_and_ec12_object_boundaries_consistent() {
|
||||
let config = Config::default();
|
||||
let object_sizes = [128 * 1024, 256 * 1024, 512 * 1024, 1024 * 1024, 4 * 1024 * 1024];
|
||||
|
||||
for (data_shards, parity_shards) in [(8, 4), (12, 4)] {
|
||||
let erasure = crate::erasure::coding::Erasure::new(data_shards, parity_shards, 1024 * 1024);
|
||||
let mut previous = true;
|
||||
for object_size in object_sizes {
|
||||
let shard_size = erasure.shard_file_size(object_size);
|
||||
let inline = config.should_inline(shard_size, data_shards, false);
|
||||
|
||||
// The effective policy is monotonic across object sizes. This
|
||||
// table covers the boundaries that previously exposed the
|
||||
// fixed-shard read-ahead mismatch, including the 1 MiB case.
|
||||
assert!(!inline || previous, "inline decision must not re-enable at {object_size} bytes");
|
||||
previous = inline;
|
||||
}
|
||||
|
||||
assert!(
|
||||
!config.should_inline(erasure.shard_file_size(1024 * 1024), data_shards, false),
|
||||
"1 MiB must use the non-inline path for EC{data_shards}+{parity_shards}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn effective_inline_block_scales_default_budget_and_preserves_explicit_limit() {
|
||||
let config = Config::default();
|
||||
assert_eq!(config.effective_inline_block(8), 32 * 1024);
|
||||
assert_eq!(config.effective_inline_block(12), 21_846);
|
||||
assert_eq!(config.effective_inline_block(0), 0);
|
||||
|
||||
let explicit = lookup_config_for_pools_with_env(
|
||||
&KVS::new(),
|
||||
&[12],
|
||||
StorageClassEnvOverrides {
|
||||
inline_block: Some("128KiB".to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.expect("explicit inline block should resolve");
|
||||
assert_eq!(explicit.effective_inline_block(12), 128 * 1024);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_inline_block_preserves_fixed_per_shard_rollback() {
|
||||
let overrides = StorageClassEnvOverrides {
|
||||
|
||||
@@ -1,414 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Shared default KVS tables for delivery targets that audit and notify declare identically.
|
||||
//!
|
||||
//! The audit and notify subsystems register one default KVS per delivery target. For amqp, nats,
|
||||
//! pulsar, postgres and kafka both sides declare byte-identical tables; for redis and mysql they
|
||||
//! differ only in a single default literal, which the caller passes in.
|
||||
//!
|
||||
//! Webhook and mqtt are deliberately absent: audit's webhook table carries extra batching/retry
|
||||
//! keys and both tables disagree on key order and on several defaults (mqtt qos, keep-alive and
|
||||
//! reconnect intervals), so they are real behavioral forks, not duplication.
|
||||
//!
|
||||
//! Key order is part of the contract: it drives the order admin config output lists the keys in,
|
||||
//! so every constructor reproduces the existing order exactly.
|
||||
|
||||
use rustfs_config::server_config::{KV, KVS};
|
||||
use rustfs_config::{
|
||||
AMQP_EXCHANGE, AMQP_MANDATORY, AMQP_PASSWORD, AMQP_PERSISTENT, AMQP_QUEUE_DIR, AMQP_QUEUE_LIMIT, AMQP_ROUTING_KEY,
|
||||
AMQP_TLS_CA, AMQP_TLS_CLIENT_CERT, AMQP_TLS_CLIENT_KEY, AMQP_URL, AMQP_USERNAME, COMMENT_KEY, DEFAULT_LIMIT, ENABLE_KEY,
|
||||
EVENT_DEFAULT_DIR, EnableState, KAFKA_ACKS, KAFKA_BROKERS, KAFKA_QUEUE_DIR, KAFKA_QUEUE_LIMIT, KAFKA_SASL_ENABLE,
|
||||
KAFKA_SASL_MECHANISM, KAFKA_SASL_PASSWORD, KAFKA_SASL_USERNAME, KAFKA_TLS_CA, KAFKA_TLS_CLIENT_CERT, KAFKA_TLS_CLIENT_KEY,
|
||||
KAFKA_TLS_ENABLE, KAFKA_TOPIC, MYSQL_DSN_STRING, MYSQL_FORMAT, MYSQL_MAX_OPEN_CONNECTIONS, MYSQL_QUEUE_DIR,
|
||||
MYSQL_QUEUE_LIMIT, MYSQL_TABLE, MYSQL_TLS_CA, MYSQL_TLS_CLIENT_CERT, MYSQL_TLS_CLIENT_KEY, NATS_ADDRESS,
|
||||
NATS_CREDENTIALS_FILE, NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS, NATS_JETSTREAM_ACK_TIMEOUT_SECS, NATS_JETSTREAM_ENABLE,
|
||||
NATS_JETSTREAM_STREAM_NAME, NATS_PASSWORD, NATS_QUEUE_DIR, NATS_QUEUE_LIMIT, NATS_SUBJECT, NATS_TLS_CA, NATS_TLS_CLIENT_CERT,
|
||||
NATS_TLS_CLIENT_KEY, NATS_TLS_REQUIRED, NATS_TOKEN, NATS_USERNAME, POSTGRES_DSN_STRING, POSTGRES_FORMAT, POSTGRES_QUEUE_DIR,
|
||||
POSTGRES_QUEUE_LIMIT, POSTGRES_TABLE, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY,
|
||||
POSTGRES_TLS_REQUIRED, PULSAR_AUTH_TOKEN, PULSAR_BROKER, PULSAR_PASSWORD, PULSAR_QUEUE_DIR, PULSAR_QUEUE_LIMIT,
|
||||
PULSAR_TLS_ALLOW_INSECURE, PULSAR_TLS_CA, PULSAR_TLS_HOSTNAME_VERIFICATION, PULSAR_TOPIC, PULSAR_USERNAME, REDIS_CHANNEL,
|
||||
REDIS_CONNECTION_TIMEOUT, REDIS_KEEP_ALIVE_INTERVAL, REDIS_MAX_RETRY_ATTEMPTS, REDIS_MAX_RETRY_DELAY, REDIS_MIN_RETRY_DELAY,
|
||||
REDIS_PASSWORD, REDIS_PIPELINE_BUFFER_SIZE, REDIS_QUEUE_DIR, REDIS_QUEUE_LIMIT, REDIS_RECONNECT_RETRY_ATTEMPTS,
|
||||
REDIS_RESPONSE_TIMEOUT, REDIS_TLS_ALLOW_INSECURE, REDIS_TLS_CA, REDIS_TLS_CLIENT_CERT, REDIS_TLS_CLIENT_KEY,
|
||||
REDIS_TLS_POLICY, REDIS_URL, REDIS_USERNAME,
|
||||
};
|
||||
|
||||
/// Builds one default entry. `hidden_if_empty` marks values the admin API elides when unset.
|
||||
fn kv(key: &str, value: impl Into<String>, hidden_if_empty: bool) -> KV {
|
||||
KV {
|
||||
key: key.to_owned(),
|
||||
value: value.into(),
|
||||
hidden_if_empty,
|
||||
}
|
||||
}
|
||||
|
||||
/// Default KVS for the amqp delivery target.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn amqp_kvs() -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(AMQP_URL, "", false),
|
||||
kv(AMQP_EXCHANGE, "", false),
|
||||
kv(AMQP_ROUTING_KEY, "", false),
|
||||
kv(AMQP_MANDATORY, EnableState::Off.to_string(), false),
|
||||
kv(AMQP_PERSISTENT, EnableState::On.to_string(), false),
|
||||
kv(AMQP_USERNAME, "", false),
|
||||
kv(AMQP_PASSWORD, "", true),
|
||||
kv(AMQP_TLS_CA, "", true),
|
||||
kv(AMQP_TLS_CLIENT_CERT, "", true),
|
||||
kv(AMQP_TLS_CLIENT_KEY, "", true),
|
||||
kv(AMQP_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(AMQP_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the nats delivery target.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn nats_kvs() -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(NATS_ADDRESS, "", false),
|
||||
kv(NATS_SUBJECT, "", false),
|
||||
kv(NATS_USERNAME, "", false),
|
||||
kv(NATS_PASSWORD, "", true),
|
||||
kv(NATS_TOKEN, "", true),
|
||||
kv(NATS_CREDENTIALS_FILE, "", true),
|
||||
kv(NATS_TLS_CA, "", true),
|
||||
kv(NATS_TLS_CLIENT_CERT, "", true),
|
||||
kv(NATS_TLS_CLIENT_KEY, "", true),
|
||||
kv(NATS_TLS_REQUIRED, EnableState::Off.to_string(), false),
|
||||
kv(NATS_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(NATS_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(NATS_JETSTREAM_ENABLE, EnableState::Off.to_string(), false),
|
||||
kv(NATS_JETSTREAM_STREAM_NAME, "", false),
|
||||
kv(
|
||||
NATS_JETSTREAM_ACK_TIMEOUT_SECS,
|
||||
NATS_JETSTREAM_ACK_TIMEOUT_DEFAULT_SECS.to_string(),
|
||||
false,
|
||||
),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the pulsar delivery target.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn pulsar_kvs() -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(PULSAR_BROKER, "", false),
|
||||
kv(PULSAR_TOPIC, "", false),
|
||||
kv(PULSAR_AUTH_TOKEN, "", true),
|
||||
kv(PULSAR_USERNAME, "", false),
|
||||
kv(PULSAR_PASSWORD, "", true),
|
||||
kv(PULSAR_TLS_CA, "", true),
|
||||
kv(PULSAR_TLS_ALLOW_INSECURE, EnableState::Off.to_string(), false),
|
||||
kv(PULSAR_TLS_HOSTNAME_VERIFICATION, EnableState::On.to_string(), false),
|
||||
kv(PULSAR_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(PULSAR_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the postgres delivery target.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn postgres_kvs() -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(POSTGRES_DSN_STRING, "", true),
|
||||
kv(POSTGRES_TABLE, "", false),
|
||||
kv(POSTGRES_FORMAT, "namespace", false),
|
||||
kv(POSTGRES_TLS_REQUIRED, EnableState::Off.to_string(), false),
|
||||
kv(POSTGRES_TLS_CA, "", true),
|
||||
kv(POSTGRES_TLS_CLIENT_CERT, "", true),
|
||||
kv(POSTGRES_TLS_CLIENT_KEY, "", true),
|
||||
kv(POSTGRES_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(POSTGRES_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the kafka delivery target.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn kafka_kvs() -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(KAFKA_BROKERS, "", false),
|
||||
kv(KAFKA_TOPIC, "", false),
|
||||
kv(KAFKA_ACKS, "1", false),
|
||||
kv(KAFKA_TLS_ENABLE, EnableState::Off.to_string(), false),
|
||||
kv(KAFKA_TLS_CA, "", true),
|
||||
kv(KAFKA_TLS_CLIENT_CERT, "", true),
|
||||
kv(KAFKA_TLS_CLIENT_KEY, "", true),
|
||||
kv(KAFKA_SASL_ENABLE, EnableState::Off.to_string(), false),
|
||||
kv(KAFKA_SASL_MECHANISM, "", false),
|
||||
kv(KAFKA_SASL_USERNAME, "", false),
|
||||
kv(KAFKA_SASL_PASSWORD, "", true),
|
||||
kv(KAFKA_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(KAFKA_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the redis delivery target. `channel` is the subsystem's default pub/sub channel,
|
||||
/// which is the only value audit and notify disagree on.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn redis_kvs(channel: &str) -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(REDIS_URL, "", false),
|
||||
kv(REDIS_CHANNEL, channel, false),
|
||||
kv(REDIS_USERNAME, "", false),
|
||||
kv(REDIS_PASSWORD, "", true),
|
||||
kv(REDIS_KEEP_ALIVE_INTERVAL, "15", false),
|
||||
kv(REDIS_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(REDIS_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(REDIS_MAX_RETRY_ATTEMPTS, "3", false),
|
||||
kv(REDIS_RECONNECT_RETRY_ATTEMPTS, "", false),
|
||||
kv(REDIS_MIN_RETRY_DELAY, "", false),
|
||||
kv(REDIS_MAX_RETRY_DELAY, "", false),
|
||||
kv(REDIS_CONNECTION_TIMEOUT, "", false),
|
||||
kv(REDIS_RESPONSE_TIMEOUT, "", false),
|
||||
kv(REDIS_PIPELINE_BUFFER_SIZE, "", false),
|
||||
kv(REDIS_TLS_POLICY, "", true),
|
||||
kv(REDIS_TLS_CA, "", true),
|
||||
kv(REDIS_TLS_CLIENT_CERT, "", true),
|
||||
kv(REDIS_TLS_CLIENT_KEY, "", true),
|
||||
kv(REDIS_TLS_ALLOW_INSECURE, EnableState::Off.to_string(), false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
/// Default KVS for the mysql delivery target. `table` is the subsystem's default destination table,
|
||||
/// which is the only value audit and notify disagree on.
|
||||
// Unused until the audit/notify tables are migrated onto these constructors.
|
||||
#[allow(dead_code)]
|
||||
pub fn mysql_kvs(table: &str) -> KVS {
|
||||
KVS(vec![
|
||||
kv(ENABLE_KEY, EnableState::Off.to_string(), false),
|
||||
kv(MYSQL_DSN_STRING, "", true),
|
||||
kv(MYSQL_TABLE, table, false),
|
||||
kv(MYSQL_FORMAT, "access", false),
|
||||
kv(MYSQL_TLS_CA, "", true),
|
||||
kv(MYSQL_TLS_CLIENT_CERT, "", true),
|
||||
kv(MYSQL_TLS_CLIENT_KEY, "", true),
|
||||
kv(MYSQL_QUEUE_DIR, EVENT_DEFAULT_DIR, false),
|
||||
kv(MYSQL_QUEUE_LIMIT, DEFAULT_LIMIT.to_string(), false),
|
||||
kv(MYSQL_MAX_OPEN_CONNECTIONS, "2", false),
|
||||
kv(COMMENT_KEY, "", false),
|
||||
])
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Expected values are spelled out as literals on purpose: they mirror the tables currently
|
||||
/// declared in `audit.rs` and `notify.rs`, so a drift in key order or in any default breaks
|
||||
/// the test instead of silently changing admin config output.
|
||||
fn assert_table(actual: &KVS, expected: &[(&str, &str, bool)]) {
|
||||
let actual: Vec<(&str, &str, bool)> = actual
|
||||
.0
|
||||
.iter()
|
||||
.map(|kv| (kv.key.as_str(), kv.value.as_str(), kv.hidden_if_empty))
|
||||
.collect();
|
||||
assert_eq!(actual, expected);
|
||||
}
|
||||
|
||||
const QUEUE_DIR: &str = "/opt/rustfs/events";
|
||||
const QUEUE_LIMIT: &str = "100000";
|
||||
|
||||
#[test]
|
||||
fn amqp_table_matches_audit_and_notify() {
|
||||
assert_table(
|
||||
&amqp_kvs(),
|
||||
&[
|
||||
("enable", "off", false),
|
||||
("url", "", false),
|
||||
("exchange", "", false),
|
||||
("routing_key", "", false),
|
||||
("mandatory", "off", false),
|
||||
("persistent", "on", false),
|
||||
("username", "", false),
|
||||
("password", "", true),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("comment", "", false),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nats_table_matches_audit_and_notify() {
|
||||
assert_table(
|
||||
&nats_kvs(),
|
||||
&[
|
||||
("enable", "off", false),
|
||||
("address", "", false),
|
||||
("subject", "", false),
|
||||
("username", "", false),
|
||||
("password", "", true),
|
||||
("token", "", true),
|
||||
("credentials_file", "", true),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("tls_required", "off", false),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("jetstream_enable", "off", false),
|
||||
("jetstream_stream_name", "", false),
|
||||
("jetstream_ack_timeout_secs", "30", false),
|
||||
("comment", "", false),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pulsar_table_matches_audit_and_notify() {
|
||||
assert_table(
|
||||
&pulsar_kvs(),
|
||||
&[
|
||||
("enable", "off", false),
|
||||
("broker", "", false),
|
||||
("topic", "", false),
|
||||
("auth_token", "", true),
|
||||
("username", "", false),
|
||||
("password", "", true),
|
||||
("tls_ca", "", true),
|
||||
("tls_allow_insecure", "off", false),
|
||||
("tls_hostname_verification", "on", false),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("comment", "", false),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn postgres_table_matches_audit_and_notify() {
|
||||
assert_table(
|
||||
&postgres_kvs(),
|
||||
&[
|
||||
("enable", "off", false),
|
||||
("dsn_string", "", true),
|
||||
("table", "", false),
|
||||
("format", "namespace", false),
|
||||
("tls_required", "off", false),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("comment", "", false),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn kafka_table_matches_audit_and_notify() {
|
||||
assert_table(
|
||||
&kafka_kvs(),
|
||||
&[
|
||||
("enable", "off", false),
|
||||
("brokers", "", false),
|
||||
("topic", "", false),
|
||||
("acks", "1", false),
|
||||
("tls_enable", "off", false),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("sasl_enable", "off", false),
|
||||
("sasl_mechanism", "", false),
|
||||
("sasl_username", "", false),
|
||||
("sasl_password", "", true),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("comment", "", false),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
fn expected_redis(channel: &str) -> Vec<(&str, &str, bool)> {
|
||||
vec![
|
||||
("enable", "off", false),
|
||||
("url", "", false),
|
||||
("channel", channel, false),
|
||||
("username", "", false),
|
||||
("password", "", true),
|
||||
("keep_alive_interval", "15", false),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("max_retry_attempts", "3", false),
|
||||
("reconnect_retry_attempts", "", false),
|
||||
("min_retry_delay", "", false),
|
||||
("max_retry_delay", "", false),
|
||||
("connection_timeout", "", false),
|
||||
("response_timeout", "", false),
|
||||
("pipeline_buffer_size", "", false),
|
||||
("tls_policy", "", true),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("tls_allow_insecure", "off", false),
|
||||
("comment", "", false),
|
||||
]
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redis_table_matches_audit() {
|
||||
assert_table(&redis_kvs("rustfs_audit_channel"), &expected_redis("rustfs_audit_channel"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redis_table_matches_notify() {
|
||||
assert_table(&redis_kvs("rustfs_notify_channel"), &expected_redis("rustfs_notify_channel"));
|
||||
}
|
||||
|
||||
fn expected_mysql(table: &str) -> Vec<(&str, &str, bool)> {
|
||||
vec![
|
||||
("enable", "off", false),
|
||||
("dsn_string", "", true),
|
||||
("table", table, false),
|
||||
("format", "access", false),
|
||||
("tls_ca", "", true),
|
||||
("tls_client_cert", "", true),
|
||||
("tls_client_key", "", true),
|
||||
("queue_dir", QUEUE_DIR, false),
|
||||
("queue_limit", QUEUE_LIMIT, false),
|
||||
("max_open_connections", "2", false),
|
||||
("comment", "", false),
|
||||
]
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mysql_table_matches_audit() {
|
||||
assert_table(&mysql_kvs("rustfs_audit_logs"), &expected_mysql("rustfs_audit_logs"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mysql_table_matches_notify() {
|
||||
assert_table(&mysql_kvs("rustfs_events"), &expected_mysql("rustfs_events"));
|
||||
}
|
||||
}
|
||||
@@ -96,8 +96,6 @@ const LOG_SUBSYSTEM_POOLS: &str = "pools";
|
||||
const EVENT_DECOMMISSION_STATE: &str = "decommission_state";
|
||||
const EVENT_DECOMMISSION_BUCKET: &str = "decommission_bucket";
|
||||
const EVENT_DECOMMISSION_ENTRY: &str = "decommission_entry";
|
||||
const POOL_ACTIVATION_FLEET_PROOF_REQUIRED: &str = "pool activation requires a live fleet capability proof";
|
||||
const POOL_ACTIVATION_FLEET_PROOF_EXPIRED: &str = "pool activation fleet capability proof expired before commit";
|
||||
const DECOMMISSION_STAGE_MIGRATE_OBJECT: &str = "migrate_object";
|
||||
const DECOMMISSION_STAGE_CLEANUP_PREFLIGHT: &str = "cleanup_preflight";
|
||||
const DECOMMISSION_STAGE_SOURCE_CLEANUP: &str = "source_cleanup";
|
||||
@@ -176,34 +174,6 @@ fn pool_meta_v3_writer_enabled() -> bool {
|
||||
)
|
||||
}
|
||||
|
||||
fn ensure_decommission_ledger_persistence_supported_for(
|
||||
version: u16,
|
||||
v2_writer_enabled: bool,
|
||||
v3_writer_enabled: bool,
|
||||
) -> Result<()> {
|
||||
if matches!(version, POOL_META_VERSION | POOL_META_GENERATION_VERSION) || v2_writer_enabled || v3_writer_enabled {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
Err(Error::InvalidArgument(
|
||||
"decommission".to_string(),
|
||||
"pool-metadata-version".to_string(),
|
||||
format!(
|
||||
"durable unresolved-entry recovery requires pool metadata V2 or V3; enable both {} and {} only after every reader and writer supports V2",
|
||||
rustfs_config::ENV_POOL_META_V2_WRITE,
|
||||
rustfs_config::ENV_POOL_META_V2_FLEET_CONFIRMED,
|
||||
),
|
||||
))
|
||||
}
|
||||
|
||||
fn ensure_decommission_ledger_persistence_supported(pool_meta: &PoolMeta) -> Result<()> {
|
||||
ensure_decommission_ledger_persistence_supported_for(
|
||||
pool_meta.version,
|
||||
pool_meta_v2_writer_enabled(),
|
||||
pool_meta_v3_writer_enabled(),
|
||||
)
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct DecommissionCanceler {
|
||||
operation: Arc<DecommissionOperation>,
|
||||
@@ -1862,13 +1832,6 @@ pub(crate) struct PoolRebalanceActivationFence {
|
||||
}
|
||||
|
||||
impl PoolRebalanceActivationFence {
|
||||
pub(crate) fn set_fleet_proof(
|
||||
&mut self,
|
||||
fleet_proof: Option<crate::services::notification_sys::CrossPoolFenceFleetProofToken>,
|
||||
) {
|
||||
self.fleet_proof = fleet_proof;
|
||||
}
|
||||
|
||||
pub(crate) fn ensure_held(&self) -> Result<()> {
|
||||
#[cfg(test)]
|
||||
let forced_lost = self.forced_lost.load(Ordering::Acquire);
|
||||
@@ -1882,7 +1845,7 @@ impl PoolRebalanceActivationFence {
|
||||
.as_ref()
|
||||
.is_some_and(|proof| !crate::services::notification_sys::cross_pool_fence_fleet_proof_matches(proof))
|
||||
{
|
||||
return Err(Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED));
|
||||
return Err(Error::other("pool activation fleet capability proof expired before commit"));
|
||||
}
|
||||
|
||||
Ok(())
|
||||
@@ -1938,17 +1901,7 @@ pub(crate) async fn acquire_pool_activation_fleet_proof(
|
||||
}
|
||||
crate::services::notification_sys::acquire_cross_pool_fence_fleet_proof()
|
||||
.map(Some)
|
||||
.ok_or_else(|| Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED))
|
||||
}
|
||||
|
||||
pub(crate) fn is_pool_activation_fleet_proof_error(err: &Error) -> bool {
|
||||
// Save-stage helpers add context by formatting the original error, so the
|
||||
// marker may be nested in the display string. Restrict matching to the
|
||||
// `Error::other` I/O shape used by this activation path.
|
||||
matches!(err, Error::Io(io_error) if io_error.kind() == std::io::ErrorKind::Other && {
|
||||
let message = io_error.to_string();
|
||||
message.contains(POOL_ACTIVATION_FLEET_PROOF_REQUIRED) || message.contains(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)
|
||||
})
|
||||
.ok_or_else(|| Error::other("pool activation requires a live fleet capability proof"))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -2055,7 +2008,6 @@ pub(crate) async fn pause_pool_activation_after_durable_save<S>(pool: &Arc<S>, f
|
||||
#[cfg(test)]
|
||||
struct PoolActivationStartProbeState {
|
||||
kind: PoolActivationStartKind,
|
||||
preflight_side_effect_attempted: std::sync::atomic::AtomicBool,
|
||||
attempted: std::sync::atomic::AtomicBool,
|
||||
notify: tokio::sync::Notify,
|
||||
}
|
||||
@@ -2074,7 +2026,6 @@ impl PoolActivationStartProbe {
|
||||
pub(crate) fn install(kind: PoolActivationStartKind) -> Self {
|
||||
let state = Arc::new(PoolActivationStartProbeState {
|
||||
kind,
|
||||
preflight_side_effect_attempted: std::sync::atomic::AtomicBool::new(false),
|
||||
attempted: std::sync::atomic::AtomicBool::new(false),
|
||||
notify: tokio::sync::Notify::new(),
|
||||
});
|
||||
@@ -2091,14 +2042,6 @@ impl PoolActivationStartProbe {
|
||||
self.state.notify.notified().await;
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn preflight_side_effect_was_attempted(&self) -> bool {
|
||||
self.state.preflight_side_effect_attempted.load(Ordering::Acquire)
|
||||
}
|
||||
|
||||
pub(crate) fn activation_was_attempted(&self) -> bool {
|
||||
self.state.attempted.load(Ordering::Acquire)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -2128,21 +2071,6 @@ pub(crate) fn observe_pool_activation_start_attempt(kind: PoolActivationStartKin
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn observe_pool_activation_preflight_side_effect_attempt(kind: PoolActivationStartKind) {
|
||||
let probes = POOL_ACTIVATION_START_PROBES
|
||||
.get_or_init(|| std::sync::Mutex::new(Vec::new()))
|
||||
.lock()
|
||||
.expect("pool activation start probe should not be poisoned")
|
||||
.iter()
|
||||
.filter(|state| state.kind == kind)
|
||||
.cloned()
|
||||
.collect::<Vec<_>>();
|
||||
for state in probes {
|
||||
state.preflight_side_effect_attempted.store(true, Ordering::Release);
|
||||
}
|
||||
}
|
||||
|
||||
fn rollback_decommission_pool_meta(pool_meta: &mut PoolMeta, previous_pool_meta: &PoolMeta, indices: &[usize]) {
|
||||
publish_pool_meta_updates(pool_meta, previous_pool_meta, indices);
|
||||
}
|
||||
@@ -6242,7 +6170,6 @@ impl ECStore {
|
||||
) -> Result<()> {
|
||||
{
|
||||
let mut pool_meta = self.pool_meta.write().await;
|
||||
ensure_decommission_ledger_persistence_supported(&pool_meta)?;
|
||||
record_decommission_unresolved_entry(&mut pool_meta, idx, generation, entry)?;
|
||||
}
|
||||
self.save_current_pool_meta(&[idx])
|
||||
@@ -6393,7 +6320,6 @@ impl ECStore {
|
||||
}
|
||||
|
||||
ensure_decommission_start_pool_states(&latest_pool_meta, indices)?;
|
||||
ensure_decommission_ledger_persistence_supported(&latest_pool_meta)?;
|
||||
|
||||
let previous_pool_meta = latest_pool_meta.clone();
|
||||
let first_idx = indices.first().copied();
|
||||
@@ -7095,14 +7021,10 @@ impl ECStore {
|
||||
save_guard.ensure_write_safe("decommission cannot be scheduled while pool metadata requires recovery")?;
|
||||
let indices = {
|
||||
let pool_meta = self.pool_meta.read().await;
|
||||
let indices = resumable_decommission_queue_indices(&pool_meta)
|
||||
resumable_decommission_queue_indices(&pool_meta)
|
||||
.into_iter()
|
||||
.filter(|idx| indices.contains(idx))
|
||||
.collect::<Vec<_>>();
|
||||
if !indices.is_empty() {
|
||||
ensure_decommission_ledger_persistence_supported(&pool_meta)?;
|
||||
}
|
||||
indices
|
||||
.collect::<Vec<_>>()
|
||||
};
|
||||
if indices.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
@@ -9358,11 +9280,8 @@ impl ECStore {
|
||||
{
|
||||
let pool_meta = self.pool_meta.read().await;
|
||||
ensure_decommission_start_pool_states(&pool_meta, &indices)?;
|
||||
ensure_decommission_ledger_persistence_supported(&pool_meta)?;
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
observe_pool_activation_preflight_side_effect_attempt(PoolActivationStartKind::Decommission);
|
||||
let decom_buckets = self.get_buckets_to_decommission().await?;
|
||||
|
||||
let mut healed_buckets = HashSet::with_capacity(decom_buckets.len());
|
||||
@@ -10909,107 +10828,10 @@ mod tests {
|
||||
use crate::bucket::replication::{ReplicationState, ReplicationStatusType};
|
||||
use serde::Serialize;
|
||||
|
||||
#[test]
|
||||
fn pool_activation_fleet_proof_error_classifier_matches_only_retryable_proof_failures() {
|
||||
assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_REQUIRED)));
|
||||
assert!(is_pool_activation_fleet_proof_error(&Error::other(POOL_ACTIVATION_FLEET_PROOF_EXPIRED)));
|
||||
let wrapped = format!("rebalance meta save failed during start_rebalance: {POOL_ACTIVATION_FLEET_PROOF_EXPIRED}");
|
||||
assert!(is_pool_activation_fleet_proof_error(&Error::other(wrapped)));
|
||||
assert!(!is_pool_activation_fleet_proof_error(&Error::ConfigNotFound));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn decommission_v1_start_preflights_reject_before_metadata_writes() {
|
||||
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(None).await;
|
||||
let baseline = load_pool_meta_replicas(store.pools.clone(), true)
|
||||
.await
|
||||
.expect("baseline pool metadata should be readable");
|
||||
assert_eq!(baseline.meta.version, POOL_META_V1_VERSION);
|
||||
*store.pool_meta.write().await = baseline.meta.clone();
|
||||
let start_probe = PoolActivationStartProbe::install(PoolActivationStartKind::Decommission);
|
||||
let err = store
|
||||
.start_decommission(vec![0])
|
||||
.await
|
||||
.expect_err("the initial V1 start preflight must reject before side effects");
|
||||
assert!(matches!(err, Error::InvalidArgument(..)));
|
||||
assert!(
|
||||
!start_probe.preflight_side_effect_was_attempted(),
|
||||
"V1 rejection must precede bucket listing, healing, and metadata-bucket creation"
|
||||
);
|
||||
assert!(
|
||||
!start_probe.activation_was_attempted(),
|
||||
"V1 rejection must not enter the authoritative activation save"
|
||||
);
|
||||
let after_early_rejection = load_pool_meta_replicas(store.pools.clone(), true)
|
||||
.await
|
||||
.expect("early rejection must leave durable pool metadata readable");
|
||||
assert_eq!(after_early_rejection.canonical, baseline.canonical);
|
||||
assert!(
|
||||
store
|
||||
.pool_meta
|
||||
.read()
|
||||
.await
|
||||
.pools
|
||||
.iter()
|
||||
.all(|pool| pool.decommission.is_none())
|
||||
);
|
||||
assert!(store.decommission_cancelers.read().await.iter().all(Option::is_none));
|
||||
store
|
||||
.ensure_pool_meta_side_effects_safe("V1 start preflight")
|
||||
.await
|
||||
.expect("a deterministic start rejection must not latch recovery");
|
||||
drop(start_probe);
|
||||
|
||||
let err = store
|
||||
.save_current_pool_meta_for_decommission_start(
|
||||
&[0],
|
||||
vec![(
|
||||
0,
|
||||
PoolSpaceInfo {
|
||||
free: 50,
|
||||
total: 100,
|
||||
used: 50,
|
||||
},
|
||||
)],
|
||||
Vec::new(),
|
||||
)
|
||||
.await
|
||||
.expect_err("the authoritative V1 start preflight must reject before saving");
|
||||
assert!(matches!(err, Error::InvalidArgument(..)));
|
||||
|
||||
let after_authoritative_rejection = load_pool_meta_replicas(store.pools.clone(), true)
|
||||
.await
|
||||
.expect("authoritative rejection must leave durable pool metadata readable");
|
||||
assert_eq!(after_authoritative_rejection.canonical, baseline.canonical);
|
||||
assert!(
|
||||
after_authoritative_rejection
|
||||
.meta
|
||||
.pools
|
||||
.iter()
|
||||
.all(|pool| pool.decommission.is_none())
|
||||
);
|
||||
assert!(
|
||||
store
|
||||
.pool_meta
|
||||
.read()
|
||||
.await
|
||||
.pools
|
||||
.iter()
|
||||
.all(|pool| pool.decommission.is_none())
|
||||
);
|
||||
assert!(store.decommission_cancelers.read().await.iter().all(Option::is_none));
|
||||
store
|
||||
.ensure_pool_meta_side_effects_safe("authoritative V1 start preflight")
|
||||
.await
|
||||
.expect("an authoritative capability rejection must not latch recovery");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn decommission_activation_fence_loss_after_durable_save_blocks_publication() {
|
||||
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(None).await;
|
||||
crate::services::rebalance::promote_test_pool_meta_to_v2(&store).await;
|
||||
let barrier = PoolActivationDurableSaveBarrier::install(&store.pools[0]);
|
||||
let start_store = Arc::clone(&store);
|
||||
let start_task = tokio::spawn(async move {
|
||||
@@ -11064,7 +10886,6 @@ mod tests {
|
||||
#[serial_test::serial]
|
||||
async fn decommission_activation_adopts_canonical_commit_after_replica_failure() {
|
||||
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(None).await;
|
||||
crate::services::rebalance::promote_test_pool_meta_to_v2(&store).await;
|
||||
let barrier = PoolActivationDurableSaveBarrier::install(&store.pools[0]);
|
||||
let start_store = Arc::clone(&store);
|
||||
let start_task = tokio::spawn(async move {
|
||||
@@ -11377,35 +11198,6 @@ mod tests {
|
||||
assert!(pool_meta_v3_writer_enabled_for(true, true));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decommission_ledger_persistence_requires_an_observed_or_confirmed_format() {
|
||||
for (version, v2_enabled, v3_enabled, expected) in [
|
||||
(POOL_META_V1_VERSION, false, false, false),
|
||||
(POOL_META_V1_VERSION, true, false, true),
|
||||
(POOL_META_V1_VERSION, false, true, true),
|
||||
(POOL_META_VERSION, false, false, true),
|
||||
(super::POOL_META_GENERATION_VERSION, false, false, true),
|
||||
] {
|
||||
let result = super::ensure_decommission_ledger_persistence_supported_for(version, v2_enabled, v3_enabled);
|
||||
assert_eq!(
|
||||
result.is_ok(),
|
||||
expected,
|
||||
"unexpected capability result for pool metadata version {version}"
|
||||
);
|
||||
}
|
||||
|
||||
let half_confirmed_v2 = pool_meta_v2_writer_enabled_for(true, false);
|
||||
let half_confirmed_v3 = pool_meta_v3_writer_enabled_for(false, true);
|
||||
let err = super::ensure_decommission_ledger_persistence_supported_for(
|
||||
POOL_META_V1_VERSION,
|
||||
half_confirmed_v2,
|
||||
half_confirmed_v3,
|
||||
)
|
||||
.expect_err("half-enabled rollout gates must not admit decommission");
|
||||
assert!(matches!(err, Error::InvalidArgument(..)));
|
||||
assert!(err.to_string().contains("durable unresolved-entry recovery"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pool_meta_stale_write_rejection_metric_is_countable() {
|
||||
let recorder = metrics_util::debugging::DebuggingRecorder::new();
|
||||
@@ -14674,113 +14466,6 @@ mod pools_tests {
|
||||
assert!(store.decommission_cancelers.read().await[0].is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_v1_unresolved_ledger_rejection_keeps_live_state_and_write_gate_safe() {
|
||||
let generation = OffsetDateTime::UNIX_EPOCH;
|
||||
let status = decommission_test_pool_status(
|
||||
0,
|
||||
Some(PoolDecommissionInfo {
|
||||
start_time: Some(generation),
|
||||
..Default::default()
|
||||
}),
|
||||
);
|
||||
let last_update = status.last_update;
|
||||
let store = decommission_worker_test_store(
|
||||
PoolMeta {
|
||||
version: POOL_META_V1_VERSION,
|
||||
pools: vec![status],
|
||||
..Default::default()
|
||||
},
|
||||
vec![None],
|
||||
);
|
||||
let entry = DecommissionUnresolvedEntry {
|
||||
bucket: "bucket-a".to_string(),
|
||||
object: "directory/".to_string(),
|
||||
pool_index: 0,
|
||||
set_index: 0,
|
||||
source_generation: generation,
|
||||
candidate_count: 1,
|
||||
disk_error_count: 0,
|
||||
observed_at: generation,
|
||||
reason: "metadata_resolution_failed".to_string(),
|
||||
};
|
||||
|
||||
let err = store
|
||||
.persist_decommission_unresolved_entry(0, generation, entry)
|
||||
.await
|
||||
.expect_err("V1 must reject the ledger before changing live state");
|
||||
|
||||
assert!(matches!(err, Error::InvalidArgument(..)));
|
||||
let pool_meta = store.pool_meta.read().await;
|
||||
let status = &pool_meta.pools[0];
|
||||
assert_eq!(status.last_update, last_update);
|
||||
assert!(
|
||||
status
|
||||
.decommission
|
||||
.as_ref()
|
||||
.expect("active decommission metadata should remain present")
|
||||
.unresolved_entries
|
||||
.is_empty()
|
||||
);
|
||||
drop(pool_meta);
|
||||
store
|
||||
.pool_meta_save_gate
|
||||
.lock()
|
||||
.await
|
||||
.ensure_write_safe("V1 unresolved-entry preflight")
|
||||
.expect("a deterministic capability rejection must not latch recovery");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_v1_runtime_recovery_rejects_worker_but_keeps_cancel_persistable() {
|
||||
let generation = OffsetDateTime::UNIX_EPOCH;
|
||||
let store = decommission_worker_test_store(
|
||||
PoolMeta {
|
||||
version: POOL_META_V1_VERSION,
|
||||
pools: vec![decommission_test_pool_status(
|
||||
0,
|
||||
Some(PoolDecommissionInfo {
|
||||
start_time: Some(generation),
|
||||
..Default::default()
|
||||
}),
|
||||
)],
|
||||
..Default::default()
|
||||
},
|
||||
vec![None],
|
||||
);
|
||||
|
||||
let err = store
|
||||
.reserve_decommission_routines(&CancellationToken::new(), &[0])
|
||||
.await
|
||||
.err()
|
||||
.expect("V1 recovery must not install a worker that cannot persist an unresolved ledger");
|
||||
assert!(matches!(err, Error::InvalidArgument(..)));
|
||||
assert!(store.decommission_cancelers.read().await[0].is_none());
|
||||
|
||||
let save_called = Arc::new(AtomicBool::new(false));
|
||||
store
|
||||
.decommission_cancel_with_owner_and_save(0, None, {
|
||||
let save_called = save_called.clone();
|
||||
move |snapshot, _| async move {
|
||||
snapshot.encode_config_data_for_v2_gate(false)?;
|
||||
save_called.store(true, Ordering::SeqCst);
|
||||
Ok(())
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("a rejected V1 recovery must remain cancelable without restart");
|
||||
|
||||
assert!(save_called.load(Ordering::SeqCst));
|
||||
let pool_meta = store.pool_meta.read().await;
|
||||
let info = pool_meta.pools[0]
|
||||
.decommission
|
||||
.as_ref()
|
||||
.expect("cancel metadata should remain present");
|
||||
assert!(info.canceled);
|
||||
assert!(!info.failed);
|
||||
assert!(!info.complete);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_decommission_transition_waits_without_registered_canceler() {
|
||||
let store = decommission_worker_test_store(PoolMeta::default(), vec![None]);
|
||||
@@ -17774,7 +17459,6 @@ mod pools_tests {
|
||||
#[tokio::test]
|
||||
async fn test_runtime_recovery_reserves_the_startup_resumable_queue() {
|
||||
let meta = PoolMeta {
|
||||
version: super::POOL_META_VERSION,
|
||||
pools: vec![
|
||||
decommission_test_pool_status(
|
||||
0,
|
||||
@@ -17832,7 +17516,6 @@ mod pools_tests {
|
||||
#[tokio::test]
|
||||
async fn test_runtime_recovery_does_not_reserve_behind_active_predecessor() {
|
||||
let meta = PoolMeta {
|
||||
version: super::POOL_META_VERSION,
|
||||
pools: vec![
|
||||
decommission_test_pool_status(
|
||||
0,
|
||||
|
||||
@@ -781,7 +781,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for Sets {
|
||||
.await
|
||||
}
|
||||
|
||||
#[tracing::instrument(skip(self, opts))]
|
||||
#[tracing::instrument(skip(self))]
|
||||
async fn delete_object(&self, bucket: &str, object: &str, opts: ObjectOptions) -> Result<ObjectInfo> {
|
||||
if opts.delete_prefix && !opts.delete_prefix_object {
|
||||
self.delete_prefix(bucket, object, &opts).await?;
|
||||
|
||||
@@ -15,8 +15,7 @@
|
||||
use crate::error::{Error, Result};
|
||||
use crate::runtime::sources::{self as runtime_sources, WorkloadSnapshotProviderRef};
|
||||
use metrics::{counter, histogram};
|
||||
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
|
||||
use rustfs_concurrency::workload::ForegroundPressure;
|
||||
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionSnapshotProvider, WorkloadClass};
|
||||
use std::time::{Duration, Instant};
|
||||
use tokio::time::sleep;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
@@ -138,6 +137,23 @@ async fn wait_for_data_movement_admission_with_provider(
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
struct ForegroundPressure {
|
||||
class: WorkloadClass,
|
||||
usage_pct: usize,
|
||||
threshold_pct: usize,
|
||||
}
|
||||
|
||||
impl ForegroundPressure {
|
||||
const fn reason(self) -> &'static str {
|
||||
match self.class {
|
||||
WorkloadClass::ForegroundRead => "foreground_read_pressure",
|
||||
WorkloadClass::ForegroundWrite => "foreground_write_pressure",
|
||||
_ => "foreground_pressure",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn foreground_pressure(
|
||||
config: &DataMovementBackpressureConfig,
|
||||
provider: Option<&(dyn WorkloadAdmissionSnapshotProvider + Send + Sync)>,
|
||||
@@ -147,11 +163,39 @@ fn foreground_pressure(
|
||||
}
|
||||
|
||||
let snapshot = provider?.workload_admission_snapshot();
|
||||
rustfs_concurrency::workload::foreground_pressure(
|
||||
&snapshot,
|
||||
config.foreground_read_high_percent,
|
||||
config.foreground_write_high_percent,
|
||||
)
|
||||
[
|
||||
(WorkloadClass::ForegroundRead, config.foreground_read_high_percent),
|
||||
(WorkloadClass::ForegroundWrite, config.foreground_write_high_percent),
|
||||
]
|
||||
.into_iter()
|
||||
.filter_map(|(class, threshold_pct)| {
|
||||
if threshold_pct == 0 {
|
||||
return None;
|
||||
}
|
||||
|
||||
let entry = snapshot.get(class)?;
|
||||
let usage_pct = if matches!(entry.state, AdmissionState::Saturated) {
|
||||
100
|
||||
} else {
|
||||
let limit = entry.limit?;
|
||||
if limit == 0 {
|
||||
return None;
|
||||
}
|
||||
entry
|
||||
.active
|
||||
.unwrap_or(0)
|
||||
.saturating_mul(100)
|
||||
.checked_div(limit)
|
||||
.unwrap_or(100)
|
||||
};
|
||||
|
||||
(usage_pct >= threshold_pct).then_some(ForegroundPressure {
|
||||
class,
|
||||
usage_pct,
|
||||
threshold_pct,
|
||||
})
|
||||
})
|
||||
.max_by_key(|pressure| pressure.usage_pct)
|
||||
}
|
||||
|
||||
fn record_delay_start(
|
||||
@@ -232,7 +276,7 @@ fn record_delay_completion(
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use rustfs_concurrency::{AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadClass};
|
||||
use rustfs_concurrency::{WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot};
|
||||
use std::sync::Arc;
|
||||
|
||||
#[derive(Debug)]
|
||||
|
||||
@@ -422,7 +422,9 @@ async fn save_data_usage_in_backend(
|
||||
if publication_epoch != expected_publication_epoch {
|
||||
return Err(Error::other("data usage publication epoch changed before save"));
|
||||
}
|
||||
crate::config::com::save_config(store.clone(), &DATA_USAGE_OBJ_NAME_PATH, data).await?;
|
||||
crate::config::com::save_config(store.clone(), &DATA_USAGE_OBJ_NAME_PATH, data)
|
||||
.await
|
||||
.map_err(Error::other)?;
|
||||
drop(publication_guard);
|
||||
|
||||
cleanup_observed_data_usage_after_authoritative_save_with_publication(store.as_ref(), &data_usage_info, Some(store.as_ref()))
|
||||
@@ -576,7 +578,7 @@ pub(crate) async fn prepare_bucket_usage_for_namespace_change(
|
||||
guard: Option<&rustfs_lock::NamespaceLockGuard>,
|
||||
) -> Result<(), Error> {
|
||||
ensure_bucket_namespace_guard(guard, bucket, "data usage cache cleanup")?;
|
||||
let _ = USAGE_MEMORY_GENERATION.try_update(Ordering::AcqRel, Ordering::Acquire, |current| Some(current.saturating_add(1)));
|
||||
let _ = USAGE_MEMORY_GENERATION.fetch_update(Ordering::AcqRel, Ordering::Acquire, |current| Some(current.saturating_add(1)));
|
||||
live_bucket_usage_cache().invalidate(bucket).await;
|
||||
clear_bucket_usage_memory(bucket, guard).await?;
|
||||
|
||||
@@ -639,7 +641,7 @@ where
|
||||
{
|
||||
Ok(reader) => reader,
|
||||
Err(Error::FileNotFound | Error::ObjectNotFound(_, _) | Error::ConfigNotFound) => return Ok(None),
|
||||
Err(err) => return Err(map_data_usage_metadata_read_error(err, object)),
|
||||
Err(err) => return Err(err),
|
||||
};
|
||||
let revision = reader
|
||||
.object_info
|
||||
@@ -654,18 +656,6 @@ where
|
||||
Ok(Some((data_usage_info, revision)))
|
||||
}
|
||||
|
||||
/// A missing usage object is harmless during bucket creation, but a missing
|
||||
/// system metadata volume is a storage outage. Keep the latter retryable and
|
||||
/// distinguishable from the user bucket not existing.
|
||||
fn map_data_usage_metadata_read_error(err: Error, object: &str) -> Error {
|
||||
match err {
|
||||
Error::BucketNotFound(_) | Error::VolumeNotFound => {
|
||||
Error::InsufficientReadQuorum(RUSTFS_META_BUCKET.to_string(), object.to_string())
|
||||
}
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
|
||||
fn data_usage_contains_bucket(data_usage_info: &DataUsageInfo, bucket: &str) -> bool {
|
||||
data_usage_info.buckets_usage.contains_key(bucket) || data_usage_info.bucket_sizes.contains_key(bucket)
|
||||
}
|
||||
@@ -922,7 +912,7 @@ where
|
||||
)
|
||||
.await;
|
||||
drop(publication_guard);
|
||||
match save_result.map_err(|err| crate::config::com::map_system_metadata_write_error(err, object)) {
|
||||
match save_result {
|
||||
Ok(_) => return Ok(()),
|
||||
Err(err) => {
|
||||
if let Some((observed, observed_revision)) = load_data_usage_for_bucket_removal(store, object).await? {
|
||||
@@ -1171,45 +1161,11 @@ fn select_admin_data_usage_snapshot(
|
||||
authoritative.usage_snapshot_converged = Some(true);
|
||||
}
|
||||
match observed {
|
||||
Some(observed)
|
||||
if observed.usage_snapshot_partial
|
||||
&& authoritative.is_complete_bucket_usage_snapshot()
|
||||
&& observed_data_usage_is_newer(&observed, &authoritative) =>
|
||||
{
|
||||
(merge_partial_observation_for_admin(authoritative, observed), true)
|
||||
}
|
||||
Some(observed) if observed_data_usage_is_newer(&observed, &authoritative) => (observed, true),
|
||||
_ => (authoritative, authoritative_format),
|
||||
}
|
||||
}
|
||||
|
||||
fn merge_partial_observation_for_admin(mut authoritative: DataUsageInfo, observed: DataUsageInfo) -> DataUsageInfo {
|
||||
for (bucket, usage) in observed.buckets_usage {
|
||||
authoritative.buckets_usage.insert(bucket, usage);
|
||||
}
|
||||
|
||||
authoritative.last_update = observed.last_update;
|
||||
authoritative.scanner_cycle = observed.scanner_cycle;
|
||||
authoritative.scanner_epoch = observed.scanner_epoch;
|
||||
authoritative.usage_snapshot_complete = false;
|
||||
authoritative.usage_snapshot_partial = true;
|
||||
authoritative.usage_snapshot_converged = Some(false);
|
||||
authoritative.usage_snapshot_authoritative_baseline = observed.usage_snapshot_authoritative_baseline;
|
||||
authoritative.usage_snapshot_set_states = observed.usage_snapshot_set_states;
|
||||
authoritative.usage_snapshot_bootstrap_pending = false;
|
||||
authoritative.buckets_count = authoritative.buckets_usage.len() as u64;
|
||||
authoritative.bucket_sizes = authoritative
|
||||
.buckets_usage
|
||||
.iter()
|
||||
.map(|(bucket, usage)| (bucket.clone(), usage.size))
|
||||
.collect();
|
||||
authoritative.replication_info.clear();
|
||||
authoritative.tier_stats = None;
|
||||
authoritative.unknown_tier_stats = None;
|
||||
authoritative.calculate_totals();
|
||||
authoritative
|
||||
}
|
||||
|
||||
async fn load_admin_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
|
||||
let (authoritative, source) = load_data_usage_snapshot(store.clone()).await?;
|
||||
let observed = load_observed_data_usage_snapshot(store).await;
|
||||
@@ -2769,7 +2725,6 @@ mod tests {
|
||||
struct UsageCacheReadStore {
|
||||
transient_failures: Mutex<usize>,
|
||||
reads: Mutex<Vec<String>>,
|
||||
terminal_error: Mutex<Option<Error>>,
|
||||
}
|
||||
|
||||
impl UsageCacheReadStore {
|
||||
@@ -2777,15 +2732,6 @@ mod tests {
|
||||
Self {
|
||||
transient_failures: Mutex::new(n),
|
||||
reads: Mutex::new(Vec::new()),
|
||||
terminal_error: Mutex::new(None),
|
||||
}
|
||||
}
|
||||
|
||||
fn with_terminal_error(error: Error) -> Self {
|
||||
Self {
|
||||
transient_failures: Mutex::new(0),
|
||||
reads: Mutex::new(Vec::new()),
|
||||
terminal_error: Mutex::new(Some(error)),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2813,9 +2759,6 @@ mod tests {
|
||||
_opts: &Self::ObjectOptions,
|
||||
) -> Result<Self::GetObjectReader, Self::Error> {
|
||||
self.reads.lock().await.push(object.to_string());
|
||||
if let Some(error) = self.terminal_error.lock().await.clone() {
|
||||
return Err(error);
|
||||
}
|
||||
let mut remaining = self.transient_failures.lock().await;
|
||||
if *remaining > 0 {
|
||||
*remaining -= 1;
|
||||
@@ -2880,22 +2823,6 @@ mod tests {
|
||||
assert!(!is_data_usage_cache_absent(&Error::DiskNotFound));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn data_usage_removal_maps_missing_system_volume_to_read_quorum() {
|
||||
for error in [Error::VolumeNotFound, Error::BucketNotFound(RUSTFS_META_BUCKET.to_string())] {
|
||||
assert_eq!(
|
||||
map_data_usage_metadata_read_error(error, "bucket-metadata/.usage.json"),
|
||||
Error::InsufficientReadQuorum(RUSTFS_META_BUCKET.to_string(), "bucket-metadata/.usage.json".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
let missing_object = Error::ObjectNotFound(RUSTFS_META_BUCKET.to_string(), "bucket-metadata/.usage.json".to_string());
|
||||
assert_eq!(
|
||||
map_data_usage_metadata_read_error(missing_object.clone(), "bucket-metadata/.usage.json"),
|
||||
missing_object
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn load_data_usage_cache_treats_absence_as_an_empty_cache_without_retrying() {
|
||||
let name = "usage-cache";
|
||||
@@ -2911,22 +2838,6 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn data_usage_removal_surfaces_missing_system_volume_as_read_quorum() {
|
||||
for cause in [Error::BucketNotFound(RUSTFS_META_BUCKET.to_string()), Error::VolumeNotFound] {
|
||||
let store = UsageCacheReadStore::with_terminal_error(cause);
|
||||
|
||||
let error = load_data_usage_for_bucket_removal(&store, "bucket-metadata/.usage.json")
|
||||
.await
|
||||
.expect_err("missing system metadata volume must not be treated as an absent usage object");
|
||||
|
||||
assert_eq!(
|
||||
error,
|
||||
Error::InsufficientReadQuorum(RUSTFS_META_BUCKET.to_string(), "bucket-metadata/.usage.json".to_string())
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn load_data_usage_cache_retries_a_transient_failure() {
|
||||
let name = "usage-cache";
|
||||
@@ -3306,108 +3217,6 @@ mod tests {
|
||||
assert_eq!(selected.buckets_usage.get("bucket").map(|usage| usage.size), Some(100));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn partial_admin_observation_preserves_authoritative_cold_buckets() {
|
||||
let baseline_time = SystemTime::UNIX_EPOCH + Duration::from_secs(10);
|
||||
let mut authoritative = data_usage_info_for_test("cold", 152_318, 80 * 1024 * 1024 * 1024, baseline_time);
|
||||
authoritative.scanner_epoch = Some(4);
|
||||
authoritative.scanner_cycle = Some(10);
|
||||
authoritative.buckets_usage.insert(
|
||||
"hot".to_string(),
|
||||
BucketUsageInfo {
|
||||
objects_count: 3_000,
|
||||
versions_count: 3_000,
|
||||
size: 400 * 1024 * 1024,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
authoritative.buckets_count = 2;
|
||||
authoritative.bucket_sizes = authoritative
|
||||
.buckets_usage
|
||||
.iter()
|
||||
.map(|(bucket, usage)| (bucket.clone(), usage.size))
|
||||
.collect();
|
||||
authoritative.calculate_totals();
|
||||
authoritative.replication_info.insert(
|
||||
"stale-target".to_string(),
|
||||
BucketTargetUsageInfo {
|
||||
replicated_size: 400 * 1024 * 1024,
|
||||
replicated_count: 3_000,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
authoritative.tier_stats = Some(rustfs_data_usage::AllTierStats {
|
||||
tiers: HashMap::from([(
|
||||
"WARM".to_string(),
|
||||
rustfs_data_usage::TierStats {
|
||||
total_size: 80 * 1024 * 1024 * 1024,
|
||||
num_versions: 152_318,
|
||||
num_objects: 152_318,
|
||||
},
|
||||
)]),
|
||||
});
|
||||
|
||||
let mut observed = DataUsageInfo {
|
||||
last_update: Some(baseline_time + Duration::from_secs(1)),
|
||||
scanner_epoch: Some(4),
|
||||
scanner_cycle: Some(11),
|
||||
usage_snapshot_complete: false,
|
||||
usage_snapshot_partial: true,
|
||||
usage_snapshot_converged: Some(false),
|
||||
usage_snapshot_authoritative_baseline: Some(authoritative.snapshot_identity()),
|
||||
usage_snapshot_set_states: vec![rustfs_data_usage::DataUsageSnapshotSetState {
|
||||
pool_index: 0,
|
||||
set_index: 0,
|
||||
scanner_cycle: Some(11),
|
||||
scanner_epoch: Some(4),
|
||||
scan_plan_digest: Some([1; 32]),
|
||||
complete: true,
|
||||
tombstone: false,
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
observed.buckets_usage.insert(
|
||||
"hot".to_string(),
|
||||
BucketUsageInfo {
|
||||
objects_count: 34,
|
||||
versions_count: 34,
|
||||
size: 8 * 1024 * 1024,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
observed.buckets_count = 1;
|
||||
observed.bucket_sizes.insert("hot".to_string(), 8 * 1024 * 1024);
|
||||
observed.calculate_totals();
|
||||
|
||||
let (selected, current_format) = select_admin_data_usage_snapshot(authoritative, true, Some(observed));
|
||||
|
||||
assert!(current_format);
|
||||
assert!(!selected.usage_snapshot_complete);
|
||||
assert!(selected.usage_snapshot_partial);
|
||||
assert!(selected.is_valid_partial_snapshot());
|
||||
assert_eq!(selected.usage_snapshot_converged, Some(false));
|
||||
assert_eq!(selected.buckets_count, 2);
|
||||
assert_eq!(
|
||||
selected
|
||||
.buckets_usage
|
||||
.get("cold")
|
||||
.map(|usage| (usage.objects_count, usage.size)),
|
||||
Some((152_318, 80 * 1024 * 1024 * 1024))
|
||||
);
|
||||
assert_eq!(
|
||||
selected
|
||||
.buckets_usage
|
||||
.get("hot")
|
||||
.map(|usage| (usage.objects_count, usage.size)),
|
||||
Some((34, 8 * 1024 * 1024))
|
||||
);
|
||||
assert_eq!(selected.objects_total_count, 152_352);
|
||||
assert_eq!(selected.objects_total_size, 80 * 1024 * 1024 * 1024 + 8 * 1024 * 1024);
|
||||
assert!(selected.replication_info.is_empty());
|
||||
assert!(selected.tier_stats.is_none());
|
||||
assert!(selected.unknown_tier_stats.is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn authoritative_save_cleanup_removes_observed_snapshot_best_effort() {
|
||||
let store = UsageCasStore::default();
|
||||
|
||||
@@ -250,40 +250,6 @@ pub(crate) trait DiskStoreRenameDataExt {
|
||||
dst_volume: &str,
|
||||
dst_path: &str,
|
||||
) -> Result<RenameDataResp>;
|
||||
|
||||
async fn rename_data_borrowed_with_guard(
|
||||
&self,
|
||||
src_volume: &str,
|
||||
src_path: &str,
|
||||
fi: &FileInfo,
|
||||
dst_volume: &str,
|
||||
dst_path: &str,
|
||||
external_guard: Option<Arc<dyn Send + Sync>>,
|
||||
) -> Result<RenameDataResp> {
|
||||
let _ = external_guard;
|
||||
self.rename_data_borrowed(src_volume, src_path, fi, dst_volume, dst_path)
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
/// Run a mutation in an owned task when a caller supplied publication guard.
|
||||
/// RPC cancellation drops only the waiter; the mutation owner keeps the guard
|
||||
/// until its operation has returned, including any detached blocking syscall.
|
||||
async fn run_owned_mutation<T, F, Fut>(external_guard: Option<Arc<dyn Send + Sync>>, operation: F) -> Result<T>
|
||||
where
|
||||
T: Send + 'static,
|
||||
F: FnOnce() -> Fut + Send + 'static,
|
||||
Fut: std::future::Future<Output = Result<T>> + Send + 'static,
|
||||
{
|
||||
if external_guard.is_none() {
|
||||
return operation().await;
|
||||
}
|
||||
tokio::spawn(async move {
|
||||
let _external_guard = external_guard;
|
||||
operation().await
|
||||
})
|
||||
.await
|
||||
.map_err(|_| Error::other("owned mutation task failed"))?
|
||||
}
|
||||
|
||||
impl DiskStoreRenameDataExt for LocalDiskWrapper {
|
||||
@@ -307,49 +273,6 @@ impl DiskStoreRenameDataExt for LocalDiskWrapper {
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn rename_data_borrowed_with_guard(
|
||||
&self,
|
||||
src_volume: &str,
|
||||
src_path: &str,
|
||||
fi: &FileInfo,
|
||||
dst_volume: &str,
|
||||
dst_path: &str,
|
||||
external_guard: Option<Arc<dyn Send + Sync>>,
|
||||
) -> Result<RenameDataResp> {
|
||||
let operation = self.clone();
|
||||
let src_volume = src_volume.to_owned();
|
||||
let src_path = src_path.to_owned();
|
||||
let fi = fi.clone();
|
||||
let dst_volume = dst_volume.to_owned();
|
||||
let dst_path = dst_path.to_owned();
|
||||
let timeout_duration = if external_guard.is_some() {
|
||||
// A fenced mutation owns the publication guard until the storage
|
||||
// operation returns. Timing out this waiter would cancel the
|
||||
// LocalDisk future while a spawn_blocking namespace syscall could
|
||||
// still be committing, reopening the movement window. The caller
|
||||
// may drop its waiter; the owned task drains the mutation.
|
||||
Duration::ZERO
|
||||
} else {
|
||||
get_max_timeout_duration()
|
||||
};
|
||||
run_owned_mutation(external_guard, move || async move {
|
||||
operation
|
||||
.track_disk_health_mutation(
|
||||
"rename_data",
|
||||
DiskMetricMutation::Write,
|
||||
|| async {
|
||||
operation
|
||||
.disk
|
||||
.rename_data_borrowed(&src_volume, &src_path, &fi, &dst_volume, &dst_path)
|
||||
.await
|
||||
},
|
||||
timeout_duration,
|
||||
)
|
||||
.await
|
||||
})
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
pub fn get_drive_walkdir_timeout() -> Duration {
|
||||
@@ -755,20 +678,17 @@ impl DiskOperationMetrics {
|
||||
let elapsed_nanos = u64::try_from(elapsed.as_nanos()).unwrap_or(u64::MAX);
|
||||
let slot = &self.last_minute[(now_sec % 60) as usize];
|
||||
loop {
|
||||
// The successful CAS below is AcqRel, so it is the publication
|
||||
// fence for the writer that owns this slot. The initial parity
|
||||
// check does not need to acquire the slot payload.
|
||||
let version = slot.version.load(Ordering::Relaxed);
|
||||
let version = slot.version.load(Ordering::Acquire);
|
||||
if !version.is_multiple_of(2) {
|
||||
std::hint::spin_loop();
|
||||
continue;
|
||||
}
|
||||
if slot
|
||||
.version
|
||||
.compare_exchange(version, version.wrapping_add(1), Ordering::AcqRel, Ordering::Relaxed)
|
||||
.compare_exchange(version, version.wrapping_add(1), Ordering::AcqRel, Ordering::Acquire)
|
||||
.is_ok()
|
||||
{
|
||||
if slot.unix_sec.load(Ordering::Relaxed) != now_sec {
|
||||
if slot.unix_sec.load(Ordering::Acquire) != now_sec {
|
||||
slot.count.store(0, Ordering::Relaxed);
|
||||
slot.acc_time.store(0, Ordering::Relaxed);
|
||||
slot.unix_sec.store(now_sec, Ordering::Release);
|
||||
@@ -784,10 +704,14 @@ impl DiskOperationMetrics {
|
||||
fn last_minute_snapshot(&self, now_sec: u64) -> TimedAction {
|
||||
let mut snapshot = TimedAction::default();
|
||||
for slot in &self.last_minute {
|
||||
let Some((slot_sec, count, acc_time)) = slot.snapshot() else {
|
||||
let version = slot.version.load(Ordering::Acquire);
|
||||
if !version.is_multiple_of(2) {
|
||||
continue;
|
||||
};
|
||||
if slot_sec <= now_sec && now_sec.saturating_sub(slot_sec) < 60 {
|
||||
}
|
||||
let slot_sec = slot.unix_sec.load(Ordering::Acquire);
|
||||
let count = slot.count.load(Ordering::Acquire);
|
||||
let acc_time = slot.acc_time.load(Ordering::Acquire);
|
||||
if slot.version.load(Ordering::Acquire) == version && slot_sec <= now_sec && now_sec.saturating_sub(slot_sec) < 60 {
|
||||
snapshot.count = snapshot.count.saturating_add(count);
|
||||
snapshot.acc_time = snapshot.acc_time.saturating_add(acc_time);
|
||||
}
|
||||
@@ -796,23 +720,6 @@ impl DiskOperationMetrics {
|
||||
}
|
||||
}
|
||||
|
||||
impl TimedActionSlot {
|
||||
fn snapshot(&self) -> Option<(u64, u64, u64)> {
|
||||
let version = self.version.load(Ordering::Acquire);
|
||||
if !version.is_multiple_of(2) {
|
||||
return None;
|
||||
}
|
||||
|
||||
// The first Acquire load publishes the payload written before the
|
||||
// matching Release store. Relaxed payload loads are sufficient while
|
||||
// the final Acquire version load validates that no writer intervened.
|
||||
let slot_sec = self.unix_sec.load(Ordering::Relaxed);
|
||||
let count = self.count.load(Ordering::Relaxed);
|
||||
let acc_time = self.acc_time.load(Ordering::Relaxed);
|
||||
(self.version.load(Ordering::Acquire) == version).then_some((slot_sec, count, acc_time))
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) struct DiskHealthWaitingGuard<'a> {
|
||||
health: &'a DiskHealthTracker,
|
||||
}
|
||||
@@ -1190,37 +1097,6 @@ impl LocalDiskWrapper {
|
||||
)
|
||||
}
|
||||
|
||||
/// Run a delete under an owned coordinator task when a publication guard
|
||||
/// is present. This keeps the guard alive if the RPC waiter is cancelled
|
||||
/// while the local namespace mutation is still in progress.
|
||||
pub(crate) async fn delete_with_publication_guard(
|
||||
&self,
|
||||
volume: &str,
|
||||
path: &str,
|
||||
options: DeleteOptions,
|
||||
external_guard: Option<Arc<dyn Send + Sync>>,
|
||||
) -> Result<()> {
|
||||
let operation = self.clone();
|
||||
let volume = volume.to_owned();
|
||||
let path = path.to_owned();
|
||||
let timeout_duration = if external_guard.is_some() {
|
||||
Duration::ZERO
|
||||
} else {
|
||||
get_max_timeout_duration()
|
||||
};
|
||||
run_owned_mutation(external_guard, move || async move {
|
||||
operation
|
||||
.track_disk_health_mutation(
|
||||
"delete",
|
||||
DiskMetricMutation::Delete,
|
||||
|| async { operation.disk.delete(&volume, &path, options).await },
|
||||
timeout_duration,
|
||||
)
|
||||
.await
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
pub(crate) fn new_with_reconnect_state(
|
||||
disk: Arc<LocalDisk>,
|
||||
health_check: bool,
|
||||
@@ -2371,44 +2247,6 @@ mod tests {
|
||||
};
|
||||
use tokio::io::AsyncWrite;
|
||||
|
||||
struct DropProbe(Arc<std::sync::atomic::AtomicUsize>);
|
||||
|
||||
impl Drop for DropProbe {
|
||||
fn drop(&mut self) {
|
||||
self.0.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn owned_mutation_keeps_publication_guard_after_waiter_cancellation() {
|
||||
let drops = Arc::new(std::sync::atomic::AtomicUsize::new(0));
|
||||
let guard: Arc<dyn Send + Sync> = Arc::new(DropProbe(Arc::clone(&drops)));
|
||||
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
|
||||
let (release_tx, release_rx) = tokio::sync::oneshot::channel();
|
||||
let (finished_tx, finished_rx) = tokio::sync::oneshot::channel();
|
||||
|
||||
let waiter = tokio::spawn(run_owned_mutation(Some(guard), move || async move {
|
||||
started_tx.send(()).expect("mutation should signal start");
|
||||
release_rx.await.expect("mutation should be released");
|
||||
finished_tx.send(()).expect("mutation should signal completion");
|
||||
Ok::<_, Error>(())
|
||||
}));
|
||||
|
||||
started_rx.await.expect("mutation owner should start");
|
||||
waiter.abort();
|
||||
assert_eq!(drops.load(std::sync::atomic::Ordering::SeqCst), 0);
|
||||
|
||||
release_tx.send(()).expect("mutation owner should still be alive");
|
||||
finished_rx.await.expect("mutation owner should finish");
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
while drops.load(std::sync::atomic::Ordering::SeqCst) == 0 {
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("publication guard should be released after mutation completion");
|
||||
}
|
||||
|
||||
struct PendingWriter;
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -22,7 +22,6 @@ pub type Error = DiskError;
|
||||
pub type Result<T> = core::result::Result<T, Error>;
|
||||
|
||||
const METACACHE_OUTPUT_STREAM_CLOSED: &str = "metacache output stream closed";
|
||||
pub(crate) const HEAL_DANGLING_DELETE_GRACE_MESSAGE: &str = "dangling object deletion deferred by heal grace window";
|
||||
|
||||
/// Marker carried by a shard-read `io::Error` when the underlying reader can
|
||||
/// no longer be realigned after a fresh remote open failed. The marker is
|
||||
@@ -34,12 +33,6 @@ pub(crate) struct TerminalReadError {
|
||||
source: DiskError,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct DanglingDeleteGraceError {
|
||||
retry_after_secs: i64,
|
||||
grace_secs: i64,
|
||||
}
|
||||
|
||||
// DiskError == StorageErr
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum DiskError {
|
||||
@@ -207,18 +200,6 @@ impl StdError for TerminalReadError {
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Display for DanglingDeleteGraceError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
write!(
|
||||
f,
|
||||
"{HEAL_DANGLING_DELETE_GRACE_MESSAGE}; retry_after_secs={}; grace_secs={}",
|
||||
self.retry_after_secs, self.grace_secs
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
impl StdError for DanglingDeleteGraceError {}
|
||||
|
||||
fn classify_internode_missing_error(error: &InternodeHttpError) -> Option<DiskError> {
|
||||
if error.is_remote_file_not_found() {
|
||||
return Some(DiskError::FileNotFound);
|
||||
@@ -272,24 +253,6 @@ impl DiskError {
|
||||
DiskError::Io(std::io::Error::other(error))
|
||||
}
|
||||
|
||||
pub(crate) fn dangling_delete_grace(retry_after_secs: i64, grace_secs: i64) -> Self {
|
||||
DiskError::other(DanglingDeleteGraceError {
|
||||
retry_after_secs,
|
||||
grace_secs,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn is_dangling_delete_grace(&self) -> bool {
|
||||
matches!(self, DiskError::Io(io_error) if Self::io_error_is_dangling_delete_grace(io_error))
|
||||
}
|
||||
|
||||
pub fn io_error_is_dangling_delete_grace(io_error: &io::Error) -> bool {
|
||||
io_error
|
||||
.get_ref()
|
||||
.is_some_and(|source| source.downcast_ref::<DanglingDeleteGraceError>().is_some())
|
||||
|| io_error.to_string().contains(HEAL_DANGLING_DELETE_GRACE_MESSAGE)
|
||||
}
|
||||
|
||||
pub(crate) fn metacache_output_stream_closed() -> Self {
|
||||
DiskError::Io(std::io::Error::new(std::io::ErrorKind::BrokenPipe, METACACHE_OUTPUT_STREAM_CLOSED))
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use crate::config::storageclass::DEFAULT_INLINE_BLOCK;
|
||||
use crate::crash_inject::{self, CrashPoint};
|
||||
use crate::data_usage::local_snapshot::ensure_data_usage_layout;
|
||||
use crate::diagnostics::get::{
|
||||
@@ -3148,10 +3149,9 @@ impl LocalIoBackend for StdBackend {
|
||||
direct_read_copy_fault_delta: MmapPageFaultDelta,
|
||||
blocking_task_duration: StdDuration,
|
||||
used_direct_io: bool,
|
||||
/// The descriptor and size snapshot opened by THIS call (None on a
|
||||
/// cache hit), handed back so the async caller can index it in the
|
||||
/// fd cache.
|
||||
opened_fd: Option<Arc<FdCacheEntry>>,
|
||||
/// The descriptor opened by THIS call (None on a cache hit), handed
|
||||
/// back so the async caller can index it in the fd cache.
|
||||
opened_fd: Option<Arc<std::fs::File>>,
|
||||
}
|
||||
|
||||
enum MmapCopyReadError {
|
||||
@@ -3198,12 +3198,12 @@ impl LocalIoBackend for StdBackend {
|
||||
(cache, key, gen_at_open)
|
||||
});
|
||||
#[cfg(target_os = "linux")]
|
||||
let cached_fd: Option<Arc<FdCacheEntry>> = match &fd_lookup {
|
||||
let cached_fd: Option<Arc<std::fs::File>> = match &fd_lookup {
|
||||
Some((cache, key, _)) => cache.get(key).await,
|
||||
None => None,
|
||||
};
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
let cached_fd: Option<Arc<FdCacheEntry>> = None;
|
||||
let cached_fd: Option<Arc<std::fs::File>> = None;
|
||||
|
||||
let blocking_wait_start = metrics_enabled.then(std::time::Instant::now);
|
||||
let read_result = tokio::task::spawn_blocking(move || {
|
||||
@@ -3225,15 +3225,8 @@ impl LocalIoBackend for StdBackend {
|
||||
// the read below is positioned (mmap offset argument / `read_exact_at`)
|
||||
// and never depends on the descriptor's current offset. `cached_fd` being
|
||||
// None also marks this call as a miss for the cache-insert side-channel.
|
||||
// The cached length is the metadata snapshot captured at open time;
|
||||
// all in-place/replacement writers invalidate this entry before
|
||||
// publishing a mutation, so cache hits avoid a redundant fstat.
|
||||
let (file, cached_len, access_check_duration) = if let Some(cached) = cached_fd.as_ref() {
|
||||
(
|
||||
cached.file.as_ref().try_clone().map_err(DiskError::from)?,
|
||||
Some(cached.len),
|
||||
StdDuration::ZERO,
|
||||
)
|
||||
let (file, access_check_duration) = if let Some(cached) = cached_fd.as_ref() {
|
||||
(cached.as_ref().try_clone().map_err(DiskError::from)?, StdDuration::ZERO)
|
||||
} else {
|
||||
// Measure the volume access probe only — the part-path resolution
|
||||
// above is accounted in `path_resolve_duration` (rustfs/backlog#1801).
|
||||
@@ -3244,27 +3237,20 @@ impl LocalIoBackend for StdBackend {
|
||||
.map_err(|e| DiskError::from(to_access_error(e, DiskError::VolumeAccessDenied)))?;
|
||||
}
|
||||
let access_check_duration = access_check_start.map_or(StdDuration::ZERO, |started_at| started_at.elapsed());
|
||||
(std::fs::File::open(&file_path).map_err(DiskError::from)?, None, access_check_duration)
|
||||
(std::fs::File::open(&file_path).map_err(DiskError::from)?, access_check_duration)
|
||||
};
|
||||
let file_open_duration = file_open_start.map_or(StdDuration::ZERO, |started_at| started_at.elapsed());
|
||||
|
||||
let (metadata_len, metadata_lookup_duration) = if let Some(len) = cached_len {
|
||||
// Reuse the open-time metadata snapshot on a cache hit. The
|
||||
// generation fence and mutation invalidation keep this value
|
||||
// tied to the inode held by `file`.
|
||||
(len, StdDuration::ZERO)
|
||||
} else {
|
||||
let metadata_lookup_start = metrics_enabled.then(StdInstant::now);
|
||||
let meta = file.metadata().map_err(DiskError::from)?;
|
||||
let duration = metadata_lookup_start.map_or(StdDuration::ZERO, |started_at| started_at.elapsed());
|
||||
(meta.len(), duration)
|
||||
};
|
||||
let metadata_lookup_start = metrics_enabled.then(StdInstant::now);
|
||||
// On a cache hit this fstats the cached descriptor — the inode it was
|
||||
// opened against, which invalidation keeps current for live entries. EC
|
||||
// shards are fixed-length, so a still-cached pre-heal length is benign.
|
||||
let meta = file.metadata().map_err(DiskError::from)?;
|
||||
let metadata_lookup_duration = metadata_lookup_start.map_or(StdDuration::ZERO, |started_at| started_at.elapsed());
|
||||
|
||||
let metadata_validate_start = metrics_enabled.then(StdInstant::now);
|
||||
if metadata_len < end_offset_u64 {
|
||||
return Err(MmapCopyReadError::OutOfBounds {
|
||||
actual_size: metadata_len,
|
||||
});
|
||||
if meta.len() < end_offset_u64 {
|
||||
return Err(MmapCopyReadError::OutOfBounds { actual_size: meta.len() });
|
||||
}
|
||||
let metadata_validate_duration =
|
||||
metadata_validate_start.map_or(StdDuration::ZERO, |started_at| started_at.elapsed());
|
||||
@@ -3410,14 +3396,9 @@ impl LocalIoBackend for StdBackend {
|
||||
// Arc; `cached_fd.is_none()` is true exactly when this call did the open.
|
||||
// Non-Linux has no fd cache, so skip the Arc allocation there.
|
||||
#[cfg(target_os = "linux")]
|
||||
let opened_fd: Option<Arc<FdCacheEntry>> = cached_fd.is_none().then(|| {
|
||||
Arc::new(FdCacheEntry {
|
||||
file: Arc::new(file),
|
||||
len: metadata_len,
|
||||
})
|
||||
});
|
||||
let opened_fd: Option<Arc<std::fs::File>> = cached_fd.is_none().then(|| Arc::new(file));
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
let opened_fd: Option<Arc<FdCacheEntry>> = None;
|
||||
let opened_fd: Option<Arc<std::fs::File>> = None;
|
||||
|
||||
Ok::<MmapCopyReadResult, MmapCopyReadError>(MmapCopyReadResult {
|
||||
bytes,
|
||||
@@ -3540,7 +3521,7 @@ impl LocalIoBackend for StdBackend {
|
||||
}
|
||||
}
|
||||
}
|
||||
// Index the freshly opened descriptor and metadata snapshot for future cache hits
|
||||
// Index the freshly opened descriptor for future cache hits
|
||||
// (rustfs/backlog#1801). `insert_if_fresh` refuses to cache if an
|
||||
// invalidation (heal/delete/rename) bumped the generation between the
|
||||
// open snapshot and now, so a stale pre-mutation inode is never served
|
||||
@@ -3892,18 +3873,6 @@ struct FdKey {
|
||||
direct: bool,
|
||||
}
|
||||
|
||||
/// Descriptor and immutable size snapshot retained for one cached shard inode.
|
||||
///
|
||||
/// The generation fence and explicit mutation invalidation keep the snapshot
|
||||
/// tied to the inode held by `file`, allowing cache hits to avoid a repeated
|
||||
/// metadata syscall without weakening replacement/heal semantics.
|
||||
struct FdCacheEntry {
|
||||
/// An independently cloneable descriptor for the immutable shard inode.
|
||||
file: Arc<std::fs::File>,
|
||||
/// File length captured together with the descriptor.
|
||||
len: u64,
|
||||
}
|
||||
|
||||
/// Per-disk cache of open descriptors for io_uring reads (backlog#1145).
|
||||
///
|
||||
/// Why this exists: `pread_uring` opened the file on the blocking pool for every
|
||||
@@ -3933,7 +3902,7 @@ struct FdCacheEntry {
|
||||
/// the descriptor once no in-flight read still holds it.
|
||||
#[cfg(target_os = "linux")]
|
||||
struct FdCache {
|
||||
cache: moka::future::Cache<FdKey, Arc<FdCacheEntry>>,
|
||||
cache: moka::future::Cache<FdKey, Arc<std::fs::File>>,
|
||||
/// Bumped by every invalidation. A miss-path open snapshots this before it
|
||||
/// opens and refuses to insert if it moved, so an fd opened before a
|
||||
/// heal/delete commit can never be resurrected into the cache after the
|
||||
@@ -3963,7 +3932,7 @@ impl FdCache {
|
||||
}
|
||||
}
|
||||
|
||||
async fn get(&self, key: &FdKey) -> Option<Arc<FdCacheEntry>> {
|
||||
async fn get(&self, key: &FdKey) -> Option<Arc<std::fs::File>> {
|
||||
self.cache.get(key).await
|
||||
}
|
||||
|
||||
@@ -3978,11 +3947,11 @@ impl FdCache {
|
||||
/// open bumped the generation, so a stale pre-heal/pre-delete inode is never
|
||||
/// cached. The post-insert re-check closes the tiny window where an
|
||||
/// invalidate races the insert itself, by removing the entry we just added.
|
||||
async fn insert_if_fresh(&self, key: FdKey, entry: Arc<FdCacheEntry>, gen_at_open: u64) {
|
||||
async fn insert_if_fresh(&self, key: FdKey, file: Arc<std::fs::File>, gen_at_open: u64) {
|
||||
if self.generation.load(Ordering::Acquire) != gen_at_open {
|
||||
return;
|
||||
}
|
||||
self.cache.insert(key.clone(), entry).await;
|
||||
self.cache.insert(key.clone(), file).await;
|
||||
if self.generation.load(Ordering::Acquire) != gen_at_open {
|
||||
self.cache.invalidate(&key).await;
|
||||
}
|
||||
@@ -4018,7 +3987,7 @@ impl FdCache {
|
||||
self.generation.fetch_add(1, Ordering::AcqRel);
|
||||
let volume = volume.to_owned();
|
||||
let prefix = prefix.trim_end_matches('/').to_owned();
|
||||
let matches = move |k: &FdKey, _: &Arc<FdCacheEntry>| {
|
||||
let matches = move |k: &FdKey, _: &Arc<std::fs::File>| {
|
||||
k.volume == volume && (k.path == prefix || k.path.strip_prefix(&prefix).is_some_and(|r| r.starts_with('/')))
|
||||
};
|
||||
if self.cache.invalidate_entries_if(matches).is_err() {
|
||||
@@ -4034,7 +4003,7 @@ impl FdCache {
|
||||
fn invalidate_volume(&self, volume: &str) {
|
||||
self.generation.fetch_add(1, Ordering::AcqRel);
|
||||
let volume = volume.to_owned();
|
||||
let matches = move |k: &FdKey, _: &Arc<FdCacheEntry>| k.volume == volume;
|
||||
let matches = move |k: &FdKey, _: &Arc<std::fs::File>| k.volume == volume;
|
||||
if self.cache.invalidate_entries_if(matches).is_err() {
|
||||
self.cache.invalidate_all();
|
||||
}
|
||||
@@ -4052,8 +4021,7 @@ impl FdCache {
|
||||
/// tests that drive the cache directly.
|
||||
#[cfg(test)]
|
||||
async fn insert(&self, key: FdKey, file: Arc<std::fs::File>) {
|
||||
let len = file.metadata().map(|metadata| metadata.len()).unwrap_or_default();
|
||||
self.cache.insert(key, Arc::new(FdCacheEntry { file, len })).await;
|
||||
self.cache.insert(key, file).await;
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -4432,12 +4400,7 @@ impl UringBackend {
|
||||
};
|
||||
|
||||
let file = match cached {
|
||||
Some(entry) => {
|
||||
if entry.len < u64::try_from(end_offset).map_err(|_| DiskError::FileCorrupt)? {
|
||||
return Err(DiskError::FileCorrupt);
|
||||
}
|
||||
Arc::clone(&entry.file)
|
||||
}
|
||||
Some(file) => file,
|
||||
None => {
|
||||
// Snapshot the cache generation BEFORE opening (rustfs/backlog#1176):
|
||||
// if a heal/delete invalidation runs while this open is in flight,
|
||||
@@ -4447,7 +4410,7 @@ impl UringBackend {
|
||||
let root = self.root.clone();
|
||||
let volume_owned = volume.to_owned();
|
||||
let path_owned = path.to_owned();
|
||||
let (file, len) = tokio::task::spawn_blocking(move || -> Result<(std::fs::File, u64)> {
|
||||
let file = tokio::task::spawn_blocking(move || -> Result<std::fs::File> {
|
||||
let file_path = resolve_uring_object_path(&root, &volume_owned, &path_owned)?;
|
||||
let file = std::fs::File::open(&file_path).map_err(DiskError::from)?;
|
||||
let meta = file.metadata().map_err(DiskError::from)?;
|
||||
@@ -4455,22 +4418,30 @@ impl UringBackend {
|
||||
if meta.len() < end_offset_u64 {
|
||||
return Err(DiskError::FileCorrupt);
|
||||
}
|
||||
Ok((file, meta.len()))
|
||||
Ok(file)
|
||||
})
|
||||
.await
|
||||
.map_err(|e| DiskError::other(format!("uring pread join error: {e}")))??;
|
||||
let file = Arc::new(FdCacheEntry {
|
||||
file: Arc::new(file),
|
||||
len,
|
||||
});
|
||||
let file = Arc::new(file);
|
||||
if let (Some((cache, key)), Some(gen_at_open)) = (cache_entry, gen_at_open) {
|
||||
cache.insert_if_fresh(key, Arc::clone(&file), gen_at_open).await;
|
||||
}
|
||||
file.file.clone()
|
||||
file
|
||||
}
|
||||
};
|
||||
|
||||
if length == 0 {
|
||||
// Parity with StdBackend and the miss path (rustfs/backlog#1173): a
|
||||
// zero-length read still rejects an offset past EOF. The miss path
|
||||
// validated `meta.len() < end_offset` (end_offset == offset here), but
|
||||
// a cache hit skipped it — so fstat the descriptor and match. This is
|
||||
// a rare path (callers do not issue zero-length reads), so the one
|
||||
// extra fstat is negligible.
|
||||
match file.metadata() {
|
||||
Ok(meta) if offset_u64 > meta.len() => return Err(DiskError::FileCorrupt),
|
||||
Ok(_) => {}
|
||||
Err(e) => return Err(DiskError::from(e)),
|
||||
}
|
||||
return Ok(Bytes::new());
|
||||
}
|
||||
|
||||
@@ -10439,14 +10410,8 @@ impl DiskAPI for LocalDisk {
|
||||
fi.data = None;
|
||||
}
|
||||
|
||||
// Keep this compatibility read-ahead decision on the same policy
|
||||
// as PUT's inline admission. In particular, do not use the old
|
||||
// fixed 128 KiB shard limit: a non-inline object in a wider EC
|
||||
// layout can have a smaller shard and would otherwise be copied
|
||||
// out of part.1 during every metadata read. Such objects remain
|
||||
// fully readable through the normal EC reader below.
|
||||
let storage_class_config = runtime_sources::storage_class_config_snapshot();
|
||||
if should_read_legacy_inline_part(&fi, storage_class_config.as_ref()) {
|
||||
let inline = fi.transition_status.is_empty() && fi.data_dir.is_some() && fi.parts.len() == 1;
|
||||
if inline && fi.shard_file_size(fi.parts[0].actual_size) < DEFAULT_INLINE_BLOCK as i64 {
|
||||
let part_path = path_join_buf(&[
|
||||
path,
|
||||
fi.data_dir.map_or_else(|| "".to_string(), |dir| dir.to_string()).as_str(),
|
||||
@@ -10948,21 +10913,6 @@ impl DiskAPI for LocalDisk {
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether a legacy object without the inline marker should have its external
|
||||
/// part materialized into `FileInfo.data` for compatibility with the old GET
|
||||
/// fast path. The marker-bearing path is handled by `read_raw`/`get_file_info`;
|
||||
/// this is only a conservative fallback for old metadata.
|
||||
fn should_read_legacy_inline_part(fi: &FileInfo, storage_class_config: &crate::config::storageclass::Config) -> bool {
|
||||
if !fi.transition_status.is_empty() || fi.data_dir.is_none() || fi.parts.len() != 1 || fi.inline_data() {
|
||||
return false;
|
||||
}
|
||||
|
||||
let part = &fi.parts[0];
|
||||
let shard_size = fi.shard_file_size(part.actual_size);
|
||||
let versioned = fi.versioned || fi.version_id.is_some_and(|version_id| !version_id.is_nil());
|
||||
storage_class_config.should_inline(shard_size, fi.erasure.data_blocks, versioned)
|
||||
}
|
||||
|
||||
impl LocalDisk {
|
||||
pub(crate) async fn rename_data_borrowed(
|
||||
&self,
|
||||
@@ -11099,46 +11049,6 @@ mod test {
|
||||
file_info
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_inline_read_ahead_matches_writer_policy_for_ec_layouts() {
|
||||
let config = crate::config::storageclass::Config::default();
|
||||
let object_sizes = [128 * 1024_i64, 256 * 1024, 512 * 1024, 1024 * 1024, 4 * 1024 * 1024];
|
||||
|
||||
for (data_shards, parity_shards) in [(8, 4), (12, 4)] {
|
||||
for object_size in object_sizes {
|
||||
let mut fi = FileInfo::new("object", data_shards, parity_shards);
|
||||
fi.data_dir = Some(Uuid::from_u128(1));
|
||||
fi.parts = vec![ObjectPartInfo {
|
||||
number: 1,
|
||||
size: usize::try_from(object_size).expect("test object size should fit usize"),
|
||||
actual_size: object_size,
|
||||
..Default::default()
|
||||
}];
|
||||
|
||||
let writer_decision = config.should_inline(fi.shard_file_size(object_size), data_shards, false);
|
||||
assert_eq!(
|
||||
should_read_legacy_inline_part(&fi, &config),
|
||||
writer_decision,
|
||||
"legacy read-ahead must match PUT for EC{data_shards}+{parity_shards}, size={object_size}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
let mut ec12 = FileInfo::new("object", 12, 4);
|
||||
ec12.data_dir = Some(Uuid::from_u128(1));
|
||||
ec12.parts = vec![ObjectPartInfo {
|
||||
number: 1,
|
||||
size: 1024 * 1024,
|
||||
actual_size: 1024 * 1024,
|
||||
..Default::default()
|
||||
}];
|
||||
assert!(
|
||||
ec12.shard_file_size(1024 * 1024) < crate::config::storageclass::DEFAULT_INLINE_BLOCK as i64,
|
||||
"the regression guard must exercise the old fixed 128 KiB read-ahead boundary"
|
||||
);
|
||||
assert!(!should_read_legacy_inline_part(&ec12, &config));
|
||||
}
|
||||
|
||||
fn test_meta(fi: FileInfo) -> Vec<u8> {
|
||||
let mut meta = FileMeta::default();
|
||||
meta.add_version(fi).expect("test metadata should accept file info");
|
||||
@@ -21437,10 +21347,11 @@ mod test {
|
||||
|
||||
/// Zero-length read bounds parity on the cache-HIT path (backlog#1173/#1180).
|
||||
/// A `length == 0` read past EOF must be rejected identically whether the
|
||||
/// descriptor is freshly opened (miss path) or served from the cache. Seeds
|
||||
/// the cache with a normal read so the zero-length reads reuse the same
|
||||
/// open-time size snapshot, then pins that UringBackend and StdBackend agree
|
||||
/// on every case.
|
||||
/// descriptor is freshly opened (miss path) or served from the cache: the
|
||||
/// cache-hit branch fstats the descriptor to reproduce the miss path's
|
||||
/// `offset > len` check instead of returning empty unconditionally. Seeds
|
||||
/// the cache with a normal read so the zero-length reads are hits, then pins
|
||||
/// that UringBackend and StdBackend agree on every case.
|
||||
#[cfg(target_os = "linux")]
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn uring_zero_length_read_bounds_match_std_on_cache_hit() {
|
||||
|
||||
@@ -677,20 +677,15 @@ impl DiskAPI for Disk {
|
||||
}
|
||||
|
||||
impl Disk {
|
||||
pub async fn delete_with_scanner_publication_lease_and_guard(
|
||||
pub(crate) async fn delete_with_scanner_publication_lease(
|
||||
&self,
|
||||
volume: &str,
|
||||
path: &str,
|
||||
opts: DeleteOptions,
|
||||
scanner_publication_lease_token: Option<Uuid>,
|
||||
external_guard: Option<Arc<dyn Send + Sync>>,
|
||||
) -> Result<()> {
|
||||
match self {
|
||||
Disk::Local(local_disk) => {
|
||||
local_disk
|
||||
.delete_with_publication_guard(volume, path, opts, external_guard)
|
||||
.await
|
||||
}
|
||||
Disk::Local(local_disk) => local_disk.delete(volume, path, opts).await,
|
||||
Disk::Remote(remote_disk) => {
|
||||
remote_disk
|
||||
.delete_with_scanner_publication_lease(volume, path, opts, scanner_publication_lease_token)
|
||||
@@ -719,34 +714,11 @@ impl Disk {
|
||||
dst_volume: &str,
|
||||
dst_path: &str,
|
||||
scanner_publication_lease_token: Option<Uuid>,
|
||||
) -> Result<RenameDataResp> {
|
||||
self.rename_data_borrowed_with_fence_and_guard(
|
||||
src_volume,
|
||||
src_path,
|
||||
fi,
|
||||
dst_volume,
|
||||
dst_path,
|
||||
scanner_publication_lease_token,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub async fn rename_data_borrowed_with_fence_and_guard(
|
||||
&self,
|
||||
src_volume: &str,
|
||||
src_path: &str,
|
||||
fi: &FileInfo,
|
||||
dst_volume: &str,
|
||||
dst_path: &str,
|
||||
scanner_publication_lease_token: Option<Uuid>,
|
||||
external_guard: Option<Arc<dyn Send + Sync>>,
|
||||
) -> Result<RenameDataResp> {
|
||||
match self {
|
||||
Disk::Local(local_disk) => {
|
||||
local_disk
|
||||
.rename_data_borrowed_with_guard(src_volume, src_path, fi, dst_volume, dst_path, external_guard)
|
||||
.rename_data_borrowed(src_volume, src_path, fi, dst_volume, dst_path)
|
||||
.await
|
||||
}
|
||||
Disk::Remote(remote_disk) => {
|
||||
|
||||
@@ -27,8 +27,6 @@ use std::io;
|
||||
use std::io::ErrorKind;
|
||||
use std::pin::Pin;
|
||||
use std::sync::Mutex;
|
||||
#[cfg(test)]
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::task::{Context, Poll, ready};
|
||||
use std::time::Instant;
|
||||
use tokio::io::{AsyncRead, ReadBuf};
|
||||
@@ -40,14 +38,6 @@ const DEFAULT_RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT: usize = 2;
|
||||
const FILL_POLICY_SINGLE_INFLIGHT: &str = "single_inflight";
|
||||
const FILL_POLICY_DUAL_INFLIGHT: &str = "dual_inflight";
|
||||
|
||||
#[cfg(test)]
|
||||
static SINGLE_INFLIGHT_CONSTRUCTIONS: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn test_single_inflight_construction_count() -> u64 {
|
||||
SINGLE_INFLIGHT_CONSTRUCTIONS.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
type FillTask = oneshot::Receiver<FillResult>;
|
||||
|
||||
struct FillWorker {
|
||||
@@ -165,23 +155,6 @@ where
|
||||
Self::new_with_fill_policy_inner(source, engine, total_length, metrics_path, FillPolicy::from_env())
|
||||
}
|
||||
|
||||
/// Construct the bounded reader without lookahead.
|
||||
///
|
||||
/// Mid-size GETs are latency-sensitive and are already gated to a single
|
||||
/// plain part. Keeping one stripe in flight avoids retaining a second
|
||||
/// decoded output buffer while preserving the same source, reconstruction,
|
||||
/// bitrot and cancellation semantics as the general streaming reader.
|
||||
pub(crate) fn new_single_inflight_with_metrics_path(
|
||||
source: S,
|
||||
engine: E,
|
||||
total_length: usize,
|
||||
metrics_path: &'static str,
|
||||
) -> io::Result<Self> {
|
||||
#[cfg(test)]
|
||||
SINGLE_INFLIGHT_CONSTRUCTIONS.fetch_add(1, Ordering::Relaxed);
|
||||
Self::new_with_fill_policy_inner(source, engine, total_length, metrics_path, FillPolicy::SingleInFlight)
|
||||
}
|
||||
|
||||
fn new_with_fill_policy_inner(
|
||||
source: S,
|
||||
engine: E,
|
||||
@@ -629,8 +602,7 @@ where
|
||||
|
||||
loop {
|
||||
if self.output_pos < self.output_buf.len() {
|
||||
if self.fill_policy == FillPolicy::DualInFlight
|
||||
&& self.prefetched_bufs.len() < self.fill_policy.max_inflight()
|
||||
if self.prefetched_bufs.len() < self.fill_policy.max_inflight()
|
||||
&& self.prefetch_error.is_none()
|
||||
&& self.remaining > 0
|
||||
&& let Poll::Ready(result) = self.poll_prefetch(cx)
|
||||
@@ -1648,149 +1620,6 @@ mod tests {
|
||||
assert_eq!(decoded, data);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_inflight_reader_reads_full_body_without_lookahead() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
let data = (0..96u8).collect::<Vec<_>>();
|
||||
let read_count = Arc::new(AtomicUsize::new(0));
|
||||
let mut source = source_from_data(&erasure, &data, &[]);
|
||||
source.read_count = Some(Arc::clone(&read_count));
|
||||
let engine = LegacyEcDecodeEngine::new(erasure);
|
||||
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source,
|
||||
engine,
|
||||
data.len(),
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
)
|
||||
.expect("single-inflight reader should be constructed");
|
||||
let mut decoded = Vec::new();
|
||||
|
||||
reader
|
||||
.read_to_end(&mut decoded)
|
||||
.await
|
||||
.expect("single-inflight reader should decode the complete body");
|
||||
|
||||
assert_eq!(decoded, data);
|
||||
assert_eq!(
|
||||
read_count.load(Ordering::SeqCst),
|
||||
3,
|
||||
"single-inflight must not read ahead after the final stripe"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_inflight_reader_preserves_partial_reads() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
let data = (0..83u8).collect::<Vec<_>>();
|
||||
let engine = LegacyEcDecodeEngine::new(erasure.clone());
|
||||
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source_from_data(&erasure, &data, &[]),
|
||||
engine,
|
||||
data.len(),
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
)
|
||||
.expect("single-inflight reader should be constructed");
|
||||
let mut decoded = Vec::with_capacity(data.len());
|
||||
let mut chunk = [0u8; 3];
|
||||
|
||||
loop {
|
||||
let read = reader.read(&mut chunk).await.expect("partial read should succeed");
|
||||
if read == 0 {
|
||||
break;
|
||||
}
|
||||
decoded.extend_from_slice(&chunk[..read]);
|
||||
}
|
||||
|
||||
assert_eq!(decoded, data);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_inflight_reader_does_not_prefetch_before_output_is_drained() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
let data = (0..96u8).collect::<Vec<_>>();
|
||||
let read_count = Arc::new(AtomicUsize::new(0));
|
||||
let mut source = source_from_data(&erasure, &data, &[]);
|
||||
source.read_count = Some(Arc::clone(&read_count));
|
||||
let engine = LegacyEcDecodeEngine::new(erasure);
|
||||
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source,
|
||||
engine,
|
||||
data.len(),
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
)
|
||||
.expect("single-inflight reader should be constructed");
|
||||
let mut first = [0u8; 3];
|
||||
|
||||
reader
|
||||
.read_exact(&mut first)
|
||||
.await
|
||||
.expect("first partial read should succeed");
|
||||
|
||||
assert_eq!(
|
||||
read_count.load(Ordering::SeqCst),
|
||||
1,
|
||||
"single-inflight must not prefetch while output remains"
|
||||
);
|
||||
assert_eq!(&first, &data[..3]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_inflight_reader_reconstructs_degraded_body() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
let data = (0..97u16).map(|value| value as u8).collect::<Vec<_>>();
|
||||
let engine = LegacyEcDecodeEngine::new(erasure.clone());
|
||||
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source_from_data(&erasure, &data, &[1]),
|
||||
engine,
|
||||
data.len(),
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
)
|
||||
.expect("single-inflight reader should be constructed");
|
||||
let mut decoded = Vec::new();
|
||||
|
||||
reader
|
||||
.read_to_end(&mut decoded)
|
||||
.await
|
||||
.expect("a readable degraded stripe should be reconstructed");
|
||||
|
||||
assert_eq!(decoded, data);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_inflight_reader_surfaces_error_after_buffered_body() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
let first_stripe = (0..32u8).collect::<Vec<_>>();
|
||||
let first_state = source_from_data(&erasure, &first_stripe, &[])
|
||||
.stripes
|
||||
.pop_front()
|
||||
.expect("first stripe should exist");
|
||||
let source = VecStripeSource {
|
||||
stripes: VecDeque::from([
|
||||
first_state,
|
||||
StripeReadState::from_parts(Vec::new(), Vec::new(), erasure.data_shards),
|
||||
]),
|
||||
read_quorum: erasure.data_shards,
|
||||
read_count: None,
|
||||
};
|
||||
let engine = LegacyEcDecodeEngine::new(erasure);
|
||||
let mut reader = ErasureDecodeReader::new_single_inflight_with_metrics_path(
|
||||
source,
|
||||
engine,
|
||||
first_stripe.len() + 1,
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
)
|
||||
.expect("single-inflight reader should be constructed");
|
||||
let mut decoded = Vec::new();
|
||||
|
||||
let error = reader
|
||||
.read_to_end(&mut decoded)
|
||||
.await
|
||||
.expect_err("short source error should be returned after buffered bytes");
|
||||
|
||||
assert_eq!(error.kind(), ErrorKind::Other);
|
||||
assert_eq!(decoded, first_stripe);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn erasure_decode_reader_stops_at_eof_for_empty_object() {
|
||||
let erasure = Erasure::new(4, 2, 32);
|
||||
@@ -2053,9 +1882,14 @@ mod tests {
|
||||
};
|
||||
let engine = LegacyEcDecodeEngine::new(Erasure::new(1, 0, 32));
|
||||
let task = tokio::spawn(async move {
|
||||
let mut reader =
|
||||
ErasureDecodeReader::new_single_inflight_with_metrics_path(source, engine, 1, GET_OBJECT_PATH_CODEC_STREAMING)
|
||||
.expect("reader should be constructed");
|
||||
let mut reader = ErasureDecodeReader::new_with_fill_policy(
|
||||
source,
|
||||
engine,
|
||||
1,
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
FillPolicy::SingleInFlight,
|
||||
)
|
||||
.expect("reader should be constructed");
|
||||
let mut first_read = [0u8; 1];
|
||||
let _ = reader.read(&mut first_read).await;
|
||||
});
|
||||
@@ -2392,7 +2226,7 @@ mod tests {
|
||||
engine,
|
||||
data.len(),
|
||||
GET_OBJECT_PATH_CODEC_STREAMING,
|
||||
FillPolicy::DualInFlight,
|
||||
FillPolicy::SingleInFlight,
|
||||
)
|
||||
.expect("reader should be constructed");
|
||||
let mut first_read = [0u8; 1];
|
||||
@@ -2401,11 +2235,13 @@ mod tests {
|
||||
|
||||
assert_eq!(read, first_read.len());
|
||||
assert_eq!(first_read[0], data[0]);
|
||||
assert_eq!(
|
||||
read_count.load(Ordering::SeqCst),
|
||||
2,
|
||||
"dual-inflight reader should prefetch the next stripe before returning the first byte"
|
||||
);
|
||||
timeout(Duration::from_secs(1), async {
|
||||
while read_count.load(Ordering::SeqCst) < 2 {
|
||||
yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("reader should start reading the next stripe before the current output buffer is fully consumed");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -157,8 +157,6 @@ pub enum StorageError {
|
||||
InvalidPartNumber(usize),
|
||||
#[error("Your proposed upload is smaller than the minimum allowed size. Part {0} size {1} is less than minimum {2}")]
|
||||
EntityTooSmall(usize, i64, i64),
|
||||
#[error("multipart upload size {0} exceeds the configured limit {1}")]
|
||||
EntityTooLarge(u64, u64),
|
||||
|
||||
// ── Erasure / Quorum ─────────────────────────────────────────────
|
||||
#[error("erasure read quorum")]
|
||||
@@ -277,10 +275,6 @@ impl StorageError {
|
||||
| StorageError::NamespaceLockQuorumUnavailable { .. }
|
||||
)
|
||||
}
|
||||
|
||||
pub fn is_dangling_delete_grace(&self) -> bool {
|
||||
matches!(self, StorageError::Io(io_error) if DiskError::io_error_is_dangling_delete_grace(io_error))
|
||||
}
|
||||
}
|
||||
|
||||
impl From<HTTPRangeError> for StorageError {
|
||||
@@ -560,7 +554,6 @@ impl Clone for StorageError {
|
||||
StorageError::DecommissionNotStarted => StorageError::DecommissionNotStarted,
|
||||
StorageError::InvalidPart(a, b, c) => StorageError::InvalidPart(*a, b.clone(), c.clone()),
|
||||
StorageError::EntityTooSmall(a, b, c) => StorageError::EntityTooSmall(*a, *b, *c),
|
||||
StorageError::EntityTooLarge(a, b) => StorageError::EntityTooLarge(*a, *b),
|
||||
StorageError::DoneForNow => StorageError::DoneForNow,
|
||||
StorageError::DecommissionAlreadyRunning => StorageError::DecommissionAlreadyRunning,
|
||||
StorageError::RebalanceAlreadyRunning => StorageError::RebalanceAlreadyRunning,
|
||||
@@ -680,7 +673,6 @@ impl StorageError {
|
||||
StorageError::InsufficientWriteQuorum(_, _) => StorageErrorCode::InsufficientWriteQuorum,
|
||||
StorageError::PreconditionFailed => StorageErrorCode::PreconditionFailed,
|
||||
StorageError::EntityTooSmall(_, _, _) => StorageErrorCode::EntityTooSmall,
|
||||
StorageError::EntityTooLarge(_, _) => StorageErrorCode::EntityTooLarge,
|
||||
StorageError::InvalidRangeSpec(_) => StorageErrorCode::InvalidRangeSpec,
|
||||
StorageError::NotModified => StorageErrorCode::NotModified,
|
||||
StorageError::InvalidPartNumber(_) => StorageErrorCode::InvalidPartNumber,
|
||||
@@ -803,7 +795,6 @@ impl StorageError {
|
||||
StorageErrorCode::EntityTooSmall => {
|
||||
Some(StorageError::EntityTooSmall(Default::default(), Default::default(), Default::default()))
|
||||
}
|
||||
StorageErrorCode::EntityTooLarge => Some(StorageError::EntityTooLarge(Default::default(), Default::default())),
|
||||
StorageErrorCode::InvalidRangeSpec => Some(StorageError::InvalidRangeSpec(Default::default())),
|
||||
StorageErrorCode::NotModified => Some(StorageError::NotModified),
|
||||
StorageErrorCode::InvalidPartNumber => Some(StorageError::InvalidPartNumber(Default::default())),
|
||||
|
||||
@@ -730,18 +730,7 @@ impl ReadPlan {
|
||||
})
|
||||
.await
|
||||
.map_err(Error::other)?
|
||||
.ok_or_else(|| {
|
||||
// The resolver saw no encryption it recognizes, yet the
|
||||
// object's markers say it is encrypted. Keep failing closed,
|
||||
// but as a typed, non-retryable error: the condition is a
|
||||
// permanent property of the stored metadata, not a fault a
|
||||
// retry can fix.
|
||||
Error::other(EncryptionResolutionError::new(
|
||||
EncryptionResolutionErrorKind::InvalidMetadata,
|
||||
"object is marked encrypted, but no decryption material could be resolved from its metadata; \
|
||||
the encryption metadata is incomplete or in a format this server cannot read",
|
||||
))
|
||||
})?;
|
||||
.ok_or_else(|| Error::other("encrypted object metadata is incomplete"))?;
|
||||
let material = resolved;
|
||||
#[cfg(feature = "rio-v2")]
|
||||
let uses_legacy_encryption = matches!(material.mode, ReadEncryptionMode::Direct { .. });
|
||||
|
||||
@@ -19,9 +19,6 @@ use crate::storage_api_contracts::{
|
||||
HTTPPreconditions, ObjectLockRetentionOptions, ObjectPreconditionError, ObjectPreconditionPart, ObjectPreconditionState,
|
||||
},
|
||||
};
|
||||
use std::sync::atomic::{AtomicBool, AtomicU8, Ordering};
|
||||
use tokio::sync::{Mutex, Notify, OwnedRwLockReadGuard};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct NamespaceLockFence {
|
||||
@@ -350,338 +347,7 @@ impl QuotaAdmission {
|
||||
}
|
||||
}
|
||||
|
||||
const SCANNER_PUBLICATION_SCOPE_ADMITTED: u8 = 0;
|
||||
const SCANNER_PUBLICATION_SCOPE_IN_FLIGHT: u8 = 1;
|
||||
const SCANNER_PUBLICATION_SCOPE_COMMITTED: u8 = 2;
|
||||
const SCANNER_PUBLICATION_SCOPE_ABORTED_BEFORE_COMMIT: u8 = 3;
|
||||
const SCANNER_PUBLICATION_SCOPE_INDETERMINATE: u8 = 4;
|
||||
|
||||
/// The terminal result of a storage-owned scanner publication mutation.
|
||||
///
|
||||
/// This state is deliberately not serialized. It is the ownership hand-off
|
||||
/// between the scanner coordinator and the storage mutation task, so a
|
||||
/// detached rename/cleanup task can retain the movement permit until it has
|
||||
/// reported a definitive result.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ScannerPublicationCommitState {
|
||||
Admitted,
|
||||
InFlight,
|
||||
Committed,
|
||||
AbortedBeforeCommit,
|
||||
Indeterminate,
|
||||
}
|
||||
|
||||
impl ScannerPublicationCommitState {
|
||||
fn as_u8(self) -> u8 {
|
||||
match self {
|
||||
Self::Admitted => SCANNER_PUBLICATION_SCOPE_ADMITTED,
|
||||
Self::InFlight => SCANNER_PUBLICATION_SCOPE_IN_FLIGHT,
|
||||
Self::Committed => SCANNER_PUBLICATION_SCOPE_COMMITTED,
|
||||
Self::AbortedBeforeCommit => SCANNER_PUBLICATION_SCOPE_ABORTED_BEFORE_COMMIT,
|
||||
Self::Indeterminate => SCANNER_PUBLICATION_SCOPE_INDETERMINATE,
|
||||
}
|
||||
}
|
||||
|
||||
fn from_u8(value: u8) -> Self {
|
||||
match value {
|
||||
SCANNER_PUBLICATION_SCOPE_IN_FLIGHT => Self::InFlight,
|
||||
SCANNER_PUBLICATION_SCOPE_COMMITTED => Self::Committed,
|
||||
SCANNER_PUBLICATION_SCOPE_ABORTED_BEFORE_COMMIT => Self::AbortedBeforeCommit,
|
||||
SCANNER_PUBLICATION_SCOPE_INDETERMINATE => Self::Indeterminate,
|
||||
_ => Self::Admitted,
|
||||
}
|
||||
}
|
||||
|
||||
/// A caller may release its remote lease only after one of these states.
|
||||
/// `Indeterminate` is intentionally excluded: the mutation may have
|
||||
/// committed after cancellation or a transport failure.
|
||||
pub fn permits_lease_release(self) -> bool {
|
||||
matches!(self, Self::Committed | Self::AbortedBeforeCommit)
|
||||
}
|
||||
}
|
||||
|
||||
/// Why a storage-owned publication scope could not start its mutation.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ScannerPublicationCommitStartError {
|
||||
Cancelled,
|
||||
DeadlineExceeded,
|
||||
AlreadyStarted,
|
||||
Terminal,
|
||||
}
|
||||
|
||||
struct ScannerPublicationCommitScopeInner {
|
||||
expected_movement_epoch: u64,
|
||||
safe_deadline: tokio::time::Instant,
|
||||
remote_lease_tokens: Arc<[Uuid]>,
|
||||
cancellation: CancellationToken,
|
||||
state: AtomicU8,
|
||||
completed: Notify,
|
||||
/// Set once a storage mutation task has taken ownership of the scope.
|
||||
/// The caller-side RAII guard must not classify cancellation as
|
||||
/// indeterminate while that owner can still report a definitive result.
|
||||
owner_attached: AtomicBool,
|
||||
/// The permit is storage-owned rather than borrowed from the scanner
|
||||
/// future. A detached mutation task keeps the scope alive and therefore
|
||||
/// keeps this guard alive until it reports a terminal state.
|
||||
movement_permit: Mutex<Option<OwnedRwLockReadGuard<()>>>,
|
||||
lease_release_safe: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
/// Storage-owned ownership scope for one fenced scanner metadata mutation.
|
||||
///
|
||||
/// The scope is an in-memory capability. It is intentionally carried through
|
||||
/// [`ObjectOptions`] as a hidden field and never participates in serde, object
|
||||
/// metadata, RPC wire structures, or on-disk formats.
|
||||
#[derive(Clone)]
|
||||
pub struct ScannerPublicationCommitScope {
|
||||
inner: Arc<ScannerPublicationCommitScopeInner>,
|
||||
}
|
||||
|
||||
/// RAII fallback for storage paths that return before their commit closure
|
||||
/// takes ownership. An in-flight scope is never guessed to be aborted: it is
|
||||
/// marked indeterminate so remote lease release remains blocked.
|
||||
pub(crate) struct ScannerPublicationCommitScopeGuard {
|
||||
scope: Option<ScannerPublicationCommitScope>,
|
||||
}
|
||||
|
||||
impl ScannerPublicationCommitScopeGuard {
|
||||
pub(crate) fn new(scope: ScannerPublicationCommitScope) -> Self {
|
||||
Self { scope: Some(scope) }
|
||||
}
|
||||
|
||||
pub(crate) fn disarm(&mut self) {
|
||||
self.scope = None;
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ScannerPublicationCommitScopeGuard {
|
||||
fn drop(&mut self) {
|
||||
let Some(scope) = self.scope.as_ref() else {
|
||||
return;
|
||||
};
|
||||
if scope.owner_attached() {
|
||||
return;
|
||||
}
|
||||
match scope.state() {
|
||||
ScannerPublicationCommitState::Admitted => {
|
||||
let _ = scope.mark_aborted_before_commit();
|
||||
}
|
||||
ScannerPublicationCommitState::InFlight => {
|
||||
let _ = scope.mark_indeterminate();
|
||||
}
|
||||
ScannerPublicationCommitState::Committed
|
||||
| ScannerPublicationCommitState::AbortedBeforeCommit
|
||||
| ScannerPublicationCommitState::Indeterminate => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Debug for ScannerPublicationCommitScope {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("ScannerPublicationCommitScope")
|
||||
.field("expected_movement_epoch", &self.expected_movement_epoch())
|
||||
.field("safe_deadline", &self.safe_deadline())
|
||||
.field("remote_lease_token_count", &self.remote_lease_tokens().len())
|
||||
.field("state", &self.state())
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
impl ScannerPublicationCommitScope {
|
||||
/// Construct a scope after the storage layer has acquired its movement
|
||||
/// read permit. Callers must keep the scope attached to the actual
|
||||
/// mutation owner until [`Self::wait_for_completion`] has resolved.
|
||||
pub(crate) fn new_storage_owned(
|
||||
expected_movement_epoch: u64,
|
||||
safe_deadline: tokio::time::Instant,
|
||||
remote_lease_tokens: Vec<Uuid>,
|
||||
movement_permit: OwnedRwLockReadGuard<()>,
|
||||
) -> Self {
|
||||
Self::new_storage_owned_with_release_flag(
|
||||
expected_movement_epoch,
|
||||
safe_deadline,
|
||||
remote_lease_tokens,
|
||||
movement_permit,
|
||||
Arc::new(AtomicBool::new(true)),
|
||||
)
|
||||
}
|
||||
|
||||
pub(crate) fn new_storage_owned_with_release_flag(
|
||||
expected_movement_epoch: u64,
|
||||
safe_deadline: tokio::time::Instant,
|
||||
remote_lease_tokens: Vec<Uuid>,
|
||||
movement_permit: OwnedRwLockReadGuard<()>,
|
||||
lease_release_safe: Arc<AtomicBool>,
|
||||
) -> Self {
|
||||
lease_release_safe.store(false, Ordering::Release);
|
||||
Self {
|
||||
inner: Arc::new(ScannerPublicationCommitScopeInner {
|
||||
expected_movement_epoch,
|
||||
safe_deadline,
|
||||
remote_lease_tokens: remote_lease_tokens.into(),
|
||||
cancellation: CancellationToken::new(),
|
||||
state: AtomicU8::new(SCANNER_PUBLICATION_SCOPE_ADMITTED),
|
||||
completed: Notify::new(),
|
||||
owner_attached: AtomicBool::new(false),
|
||||
movement_permit: Mutex::new(Some(movement_permit)),
|
||||
lease_release_safe,
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn expected_movement_epoch(&self) -> u64 {
|
||||
self.inner.expected_movement_epoch
|
||||
}
|
||||
|
||||
pub fn safe_deadline(&self) -> tokio::time::Instant {
|
||||
self.inner.safe_deadline
|
||||
}
|
||||
|
||||
pub fn is_expired(&self) -> bool {
|
||||
tokio::time::Instant::now() >= self.safe_deadline()
|
||||
}
|
||||
|
||||
pub fn remote_lease_tokens(&self) -> &[Uuid] {
|
||||
&self.inner.remote_lease_tokens
|
||||
}
|
||||
|
||||
pub fn cancellation_token(&self) -> CancellationToken {
|
||||
self.inner.cancellation.clone()
|
||||
}
|
||||
|
||||
pub fn is_cancelled(&self) -> bool {
|
||||
self.inner.cancellation.is_cancelled()
|
||||
}
|
||||
|
||||
/// Whether a mutation that has already begun may still enter its durable
|
||||
/// commit boundary. The storage owner must check this immediately before
|
||||
/// starting each irreversible fan-out/rename operation.
|
||||
pub fn can_commit(&self) -> bool {
|
||||
self.state() == ScannerPublicationCommitState::InFlight && !self.is_cancelled() && !self.is_expired()
|
||||
}
|
||||
|
||||
/// Transfer terminal-state responsibility from the caller to a detached
|
||||
/// storage mutation owner. Once set, dropping a scanner waiter leaves the
|
||||
/// scope in-flight until that owner reports committed or indeterminate.
|
||||
pub fn attach_mutation_owner(&self) {
|
||||
self.inner.owner_attached.store(true, Ordering::Release);
|
||||
}
|
||||
|
||||
fn owner_attached(&self) -> bool {
|
||||
self.inner.owner_attached.load(Ordering::Acquire)
|
||||
}
|
||||
|
||||
pub fn state(&self) -> ScannerPublicationCommitState {
|
||||
ScannerPublicationCommitState::from_u8(self.inner.state.load(Ordering::Acquire))
|
||||
}
|
||||
|
||||
/// Request cancellation without claiming that a mutation has stopped.
|
||||
/// The owner must still report `AbortedBeforeCommit` or `Indeterminate`.
|
||||
pub fn cancel(&self) {
|
||||
self.inner.cancellation.cancel();
|
||||
}
|
||||
|
||||
pub fn try_begin(&self) -> std::result::Result<(), ScannerPublicationCommitStartError> {
|
||||
if self.inner.cancellation.is_cancelled() {
|
||||
return Err(ScannerPublicationCommitStartError::Cancelled);
|
||||
}
|
||||
if self.is_expired() {
|
||||
return Err(ScannerPublicationCommitStartError::DeadlineExceeded);
|
||||
}
|
||||
self.inner
|
||||
.state
|
||||
.compare_exchange(
|
||||
SCANNER_PUBLICATION_SCOPE_ADMITTED,
|
||||
SCANNER_PUBLICATION_SCOPE_IN_FLIGHT,
|
||||
Ordering::AcqRel,
|
||||
Ordering::Acquire,
|
||||
)
|
||||
.map(|_| ())
|
||||
.map_err(|state| {
|
||||
if ScannerPublicationCommitState::from_u8(state).permits_lease_release() {
|
||||
ScannerPublicationCommitStartError::Terminal
|
||||
} else {
|
||||
ScannerPublicationCommitStartError::AlreadyStarted
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
pub fn mark_committed(&self) -> bool {
|
||||
self.mark_terminal(ScannerPublicationCommitState::Committed)
|
||||
}
|
||||
|
||||
pub fn mark_aborted_before_commit(&self) -> bool {
|
||||
if self
|
||||
.inner
|
||||
.state
|
||||
.compare_exchange(
|
||||
SCANNER_PUBLICATION_SCOPE_ADMITTED,
|
||||
SCANNER_PUBLICATION_SCOPE_ABORTED_BEFORE_COMMIT,
|
||||
Ordering::AcqRel,
|
||||
Ordering::Acquire,
|
||||
)
|
||||
.is_ok()
|
||||
{
|
||||
self.inner.lease_release_safe.store(true, Ordering::Release);
|
||||
self.inner.completed.notify_waiters();
|
||||
return true;
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
pub fn mark_indeterminate(&self) -> bool {
|
||||
self.mark_terminal(ScannerPublicationCommitState::Indeterminate)
|
||||
}
|
||||
|
||||
fn mark_terminal(&self, terminal: ScannerPublicationCommitState) -> bool {
|
||||
self.inner
|
||||
.state
|
||||
.compare_exchange(SCANNER_PUBLICATION_SCOPE_IN_FLIGHT, terminal.as_u8(), Ordering::AcqRel, Ordering::Acquire)
|
||||
.is_ok()
|
||||
.then(|| {
|
||||
if terminal.permits_lease_release() {
|
||||
self.inner.lease_release_safe.store(true, Ordering::Release);
|
||||
}
|
||||
self.inner.completed.notify_waiters()
|
||||
})
|
||||
.is_some()
|
||||
}
|
||||
|
||||
/// Wait until the mutation owner has reported a definitive terminal
|
||||
/// state. The permit remains owned by this scope until all scope clones are
|
||||
/// dropped or [`Self::release_movement_permit`] is called safely.
|
||||
pub async fn wait_for_completion(&self) -> ScannerPublicationCommitState {
|
||||
loop {
|
||||
let notified = self.inner.completed.notified();
|
||||
tokio::pin!(notified);
|
||||
notified.as_mut().enable();
|
||||
let state = self.state();
|
||||
if state != ScannerPublicationCommitState::Admitted && state != ScannerPublicationCommitState::InFlight {
|
||||
return state;
|
||||
}
|
||||
notified.await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Release the storage-owned movement permit only after a known-safe
|
||||
/// terminal result. Returns `false` for in-flight or indeterminate work.
|
||||
pub async fn release_movement_permit(&self) -> bool {
|
||||
if !self.state().permits_lease_release() {
|
||||
return false;
|
||||
}
|
||||
self.inner.movement_permit.lock().await.take().is_some()
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ScannerPublicationCommitScopeInner {
|
||||
fn drop(&mut self) {
|
||||
if !ScannerPublicationCommitState::from_u8(self.state.load(Ordering::Acquire)).permits_lease_release() {
|
||||
self.lease_release_safe.store(false, Ordering::Release);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default, Clone)]
|
||||
#[derive(Debug, Default, Clone)]
|
||||
pub struct ObjectOptions {
|
||||
// Use the maximum parity (N/2), used when saving server configuration files
|
||||
pub max_parity: bool,
|
||||
@@ -718,11 +384,6 @@ pub struct ObjectOptions {
|
||||
#[doc(hidden)]
|
||||
pub put_object_cancellation: Option<tokio_util::sync::CancellationToken>,
|
||||
|
||||
/// Storage-owned scanner publication capability. This field is an
|
||||
/// in-memory hand-off only; it is never copied into object metadata.
|
||||
#[doc(hidden)]
|
||||
pub scanner_publication_commit_scope: Option<ScannerPublicationCommitScope>,
|
||||
|
||||
pub data_movement: bool,
|
||||
pub raw_data_movement_read: bool,
|
||||
/// Materialize the data-movement per-part checksum sidecar for APIs that
|
||||
@@ -790,76 +451,6 @@ pub struct ObjectOptions {
|
||||
pub tier_delete_journal_api: Option<Arc<crate::store::ECStore>>,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for ObjectOptions {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("ObjectOptions")
|
||||
.field("max_parity", &self.max_parity)
|
||||
.field("mod_time", &self.mod_time)
|
||||
.field("part_number", &self.part_number)
|
||||
.field("delete_prefix", &self.delete_prefix)
|
||||
.field("delete_prefix_object", &self.delete_prefix_object)
|
||||
.field("version_id", &self.version_id.is_some())
|
||||
.field("lifecycle_delete_all", &self.lifecycle_delete_all.is_some())
|
||||
.field("lifecycle_delete_all_journal", &self.lifecycle_delete_all_journal.is_some())
|
||||
.field("expected_current_version_id", &self.expected_current_version_id.is_some())
|
||||
.field("expected_bucket_incarnation_id", &self.expected_bucket_incarnation_id)
|
||||
.field("no_lock", &self.no_lock)
|
||||
.field("metadata_cache_safe", &self.metadata_cache_safe)
|
||||
.field("versioned", &self.versioned)
|
||||
.field("version_suspended", &self.version_suspended)
|
||||
.field("incl_free_versions", &self.incl_free_versions)
|
||||
.field("skip_decommissioned", &self.skip_decommissioned)
|
||||
.field("skip_rebalancing", &self.skip_rebalancing)
|
||||
.field("skip_free_version", &self.skip_free_version)
|
||||
.field("put_object_cancellation", &self.put_object_cancellation.is_some())
|
||||
.field("scanner_publication_commit_scope", &self.scanner_publication_commit_scope)
|
||||
.field("data_movement", &self.data_movement)
|
||||
.field("raw_data_movement_read", &self.raw_data_movement_read)
|
||||
.field("include_part_checksums", &self.include_part_checksums)
|
||||
.field("src_pool_idx", &self.src_pool_idx)
|
||||
.field("user_defined_count", &self.user_defined.len())
|
||||
.field("preserve_etag", &self.preserve_etag.is_some())
|
||||
.field("metadata_chg", &self.metadata_chg)
|
||||
.field("http_preconditions", &self.http_preconditions.is_some())
|
||||
.field("delete_replication", &self.delete_replication.is_some())
|
||||
.field("delete_replication_config_snapshot", &self.delete_replication_config_snapshot)
|
||||
.field("namespace_lock_fence", &self.namespace_lock_fence.is_some())
|
||||
.field("bucket_lifecycle_lock_fence", &self.bucket_lifecycle_lock_fence.is_some())
|
||||
.field("replication_request", &self.replication_request)
|
||||
.field("proxy_request", &self.proxy_request)
|
||||
.field("proxy_header_set", &self.proxy_header_set)
|
||||
.field("replication_tagging_timestamp", &self.replication_tagging_timestamp)
|
||||
.field("replication_retention_timestamp", &self.replication_retention_timestamp)
|
||||
.field("replication_legalhold_timestamp", &self.replication_legalhold_timestamp)
|
||||
.field("preserve_ciphertext", &self.preserve_ciphertext)
|
||||
.field("delete_marker", &self.delete_marker)
|
||||
.field("synthetic_version_id", &self.synthetic_version_id)
|
||||
.field(
|
||||
"transition",
|
||||
&(self.data_movement
|
||||
|| !self.transition.status.is_empty()
|
||||
|| !self.transition.tier.is_empty()
|
||||
|| self.transition.expected_data_dir.is_some()),
|
||||
)
|
||||
.field("expiration", &self.expiration)
|
||||
.field(
|
||||
"lifecycle_audit_event",
|
||||
&(!self.lifecycle_audit_event.event.rule_id.is_empty()
|
||||
|| !self.lifecycle_audit_event.event.storage_class.is_empty()),
|
||||
)
|
||||
.field("eval_metadata_count", &self.eval_metadata.as_ref().map(HashMap::len))
|
||||
.field("object_lock_retention", &self.object_lock_retention.is_some())
|
||||
.field("object_lock_delete", &self.object_lock_delete)
|
||||
.field("object_lock_config_snapshot", &self.object_lock_config_snapshot.is_some())
|
||||
.field("want_checksum", &self.want_checksum)
|
||||
.field("skip_verify_bitrot", &self.skip_verify_bitrot)
|
||||
.field("capacity_scope_token", &self.capacity_scope_token)
|
||||
.field("quota_admission", &self.quota_admission)
|
||||
.field("tier_delete_journal_api", &self.tier_delete_journal_api.is_some())
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
/// Transient scanner-only carrier for target-side publication lease tokens.
|
||||
/// SetDisks consumes and removes this key before constructing durable
|
||||
/// FileInfo metadata; it must never appear in an S3-visible object.
|
||||
@@ -1161,18 +752,12 @@ impl ObjectInfo {
|
||||
.any(|key| rustfs_utils::http::is_object_encryption_marker(key))
|
||||
}
|
||||
|
||||
/// Historical non-versioned inline size reference.
|
||||
///
|
||||
/// Inline admission is layout-specific now; callers must not use this
|
||||
/// constant to decide whether an object is eligible for the fast path.
|
||||
#[deprecated(note = "inline eligibility is layout-specific; use persisted metadata and the read-path policy")]
|
||||
/// Maximum inline size for non-versioned objects (128 KiB).
|
||||
/// Matches `DEFAULT_INLINE_BLOCK` in `storageclass.rs`.
|
||||
pub const INLINE_MAX_SIZE: i64 = 128 * 1024;
|
||||
|
||||
/// Historical versioned inline size reference.
|
||||
///
|
||||
/// Inline admission is layout-specific now; callers must not use this
|
||||
/// constant to decide whether an object is eligible for the fast path.
|
||||
#[deprecated(note = "inline eligibility is layout-specific; use persisted metadata and the read-path policy")]
|
||||
/// Maximum inline size for versioned objects (16 KiB).
|
||||
/// Matches `DEFAULT_INLINE_BLOCK / 8` in `storageclass.rs`.
|
||||
pub const INLINE_MAX_SIZE_VERSIONED: i64 = 16 * 1024;
|
||||
|
||||
/// Returns `true` when this object qualifies for the inline data fast path.
|
||||
@@ -1180,12 +765,10 @@ impl ObjectInfo {
|
||||
/// The inline fast path decodes erasure-coded data entirely in memory,
|
||||
/// bypassing disk I/O, duplex pipes, and the disk-read semaphore.
|
||||
///
|
||||
/// The persisted `inlined` flag is the canonical size-policy decision. PUT
|
||||
/// sets it through the captured storage-class snapshot's effective policy,
|
||||
/// which is layout- and version-aware. Reapplying a fixed object-size limit
|
||||
/// here would disagree with that policy for wider EC layouts and explicit
|
||||
/// inline configurations. The direct-memory reader retains its own bounded
|
||||
/// 128 KiB allocation gate at the call site.
|
||||
/// The `inlined` flag is the primary signal — PUT sets it through the
|
||||
/// captured storage-class snapshot's `Config::should_inline`, which applies
|
||||
/// the correct version-aware threshold (128 KiB non-versioned, 16 KiB versioned).
|
||||
/// The size check below is a safety net using the same thresholds.
|
||||
///
|
||||
/// Additional conditions:
|
||||
/// - Single part
|
||||
@@ -1196,8 +779,14 @@ impl ObjectInfo {
|
||||
if !self.inlined {
|
||||
return false;
|
||||
}
|
||||
// Apply the same version-aware threshold as PUT (storageclass.rs).
|
||||
let max_size = if self.version_id.is_some() {
|
||||
Self::INLINE_MAX_SIZE_VERSIONED
|
||||
} else {
|
||||
Self::INLINE_MAX_SIZE
|
||||
};
|
||||
self.parts.len() == 1
|
||||
&& self.size >= 0
|
||||
&& self.size <= max_size
|
||||
&& !self.is_encrypted()
|
||||
&& !self.is_compressed()
|
||||
&& self.transitioned_object.tier.is_empty()
|
||||
@@ -1793,14 +1382,14 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_fast_path_eligibility_follows_persisted_marker() {
|
||||
fn inline_fast_path_eligibility_preserves_exact_versioned_boundaries() {
|
||||
for (case, size, versioned, expected) in [
|
||||
("unversioned below", 128 * 1024 - 1, false, true),
|
||||
("unversioned exact", 128 * 1024, false, true),
|
||||
("unversioned above", 128 * 1024 + 1, false, true),
|
||||
("unversioned above", 128 * 1024 + 1, false, false),
|
||||
("versioned below", 16 * 1024 - 1, true, true),
|
||||
("versioned exact", 16 * 1024, true, true),
|
||||
("versioned above", 16 * 1024 + 1, true, true),
|
||||
("versioned above", 16 * 1024 + 1, true, false),
|
||||
] {
|
||||
assert_eq!(
|
||||
inline_fast_path_object(size, versioned).is_inline_fast_path_eligible(),
|
||||
@@ -1810,29 +1399,9 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_fast_path_marker_allows_ec8_and_ec12_layout_specific_256kib_objects() {
|
||||
for data_blocks in [8, 12] {
|
||||
let object = ObjectInfo {
|
||||
size: 256 * 1024,
|
||||
data_blocks,
|
||||
parity_blocks: 4,
|
||||
inlined: true,
|
||||
version_id: Some(Uuid::from_u128(1)),
|
||||
parts: Arc::new(vec![ObjectPartInfo::default()]),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
assert!(
|
||||
object.is_inline_fast_path_eligible(),
|
||||
"the persisted inline marker must be authoritative for EC{data_blocks}+4"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_fast_path_eligibility_rejects_incompatible_object_shapes() {
|
||||
let mut object = inline_fast_path_object(128 * 1024, false);
|
||||
let mut object = inline_fast_path_object(ObjectInfo::INLINE_MAX_SIZE, false);
|
||||
|
||||
object.inlined = false;
|
||||
assert!(!object.is_inline_fast_path_eligible(), "non-inline objects must fall back");
|
||||
|
||||
@@ -391,7 +391,7 @@ impl InstanceContext {
|
||||
let previous = self.data_movement_operation_epoch.load(Ordering::Acquire);
|
||||
let _ = self
|
||||
.data_movement_operation_epoch
|
||||
.try_update(Ordering::AcqRel, Ordering::Acquire, |epoch| Some(epoch.saturating_add(1)));
|
||||
.fetch_update(Ordering::AcqRel, Ordering::Acquire, |epoch| Some(epoch.saturating_add(1)));
|
||||
let result = self.data_movement_operation_epoch.load(Ordering::Acquire);
|
||||
if result == u64::MAX {
|
||||
self.data_movement_operation_epoch_exhausted.store(true, Ordering::Release);
|
||||
@@ -412,7 +412,7 @@ impl InstanceContext {
|
||||
}
|
||||
let updated = self
|
||||
.data_movement_generation
|
||||
.try_update(Ordering::AcqRel, Ordering::Acquire, |generation| generation.checked_add(1));
|
||||
.fetch_update(Ordering::AcqRel, Ordering::Acquire, |generation| generation.checked_add(1));
|
||||
match updated {
|
||||
Ok(previous) => {
|
||||
let Some(generation) = previous.checked_add(1) else {
|
||||
|
||||
@@ -234,39 +234,6 @@ pub(crate) fn install_cross_pool_fence_fleet_proof_for_test() {
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) struct CrossPoolFenceFleetProofGuard {
|
||||
previous_proof: Option<FleetCapabilityProof>,
|
||||
previous_topology_conflict: bool,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl Drop for CrossPoolFenceFleetProofGuard {
|
||||
fn drop(&mut self) {
|
||||
let mut state = cross_pool_fence_fleet_proof_slot()
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
state.proof = self.previous_proof.take();
|
||||
state.topology_conflict = self.previous_topology_conflict;
|
||||
}
|
||||
}
|
||||
|
||||
/// Temporarily revoke the test proof so activation paths can exercise their
|
||||
/// fail-closed behavior without changing the process-wide topology binding.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn without_cross_pool_fence_fleet_proof_for_test() -> CrossPoolFenceFleetProofGuard {
|
||||
let mut state = cross_pool_fence_fleet_proof_slot()
|
||||
.write()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
let guard = CrossPoolFenceFleetProofGuard {
|
||||
previous_proof: state.proof.clone(),
|
||||
previous_topology_conflict: state.topology_conflict,
|
||||
};
|
||||
state.proof = None;
|
||||
state.topology_conflict = true;
|
||||
guard
|
||||
}
|
||||
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
pub fn rotate_cross_pool_fence_fleet_proof_for_test() -> bool {
|
||||
let mut state = cross_pool_fence_fleet_proof_slot()
|
||||
@@ -1483,7 +1450,7 @@ impl NotificationSys {
|
||||
futures.push(async move {
|
||||
let client = client.ok_or_else(|| Error::other(format!("scanner activity peer[{idx}] is unreachable")))?;
|
||||
let host = client.grid_host.clone();
|
||||
scanner_activity_with_retry(&client, &host)
|
||||
scanner_activity_with_timeout(SCANNER_ACTIVITY_PROBE_TIMEOUT, &host, client.scanner_activity())
|
||||
.await
|
||||
.map(|activity| (host, activity))
|
||||
});
|
||||
@@ -1962,50 +1929,6 @@ where
|
||||
.map_err(|_| Error::other(format!("scanner activity peer {host} timed out after {timeout_duration:?}")))?
|
||||
}
|
||||
|
||||
/// Classify transport-only activity failures without treating an answered
|
||||
/// peer's application error as an outage.
|
||||
pub fn scanner_peer_transport_error_message_is_retryable(error: &str) -> bool {
|
||||
crate::cluster::rpc::client::message_has_network_needle(error)
|
||||
}
|
||||
|
||||
fn scanner_activity_should_retry(first_error: Option<&Error>, timed_out: bool) -> bool {
|
||||
timed_out || first_error.is_some_and(PeerRestClient::is_network_like_error)
|
||||
}
|
||||
|
||||
/// Retry one activity probe after a bounded reconnect when the first attempt
|
||||
/// failed at the transport boundary. A peer that answered with an invalid or
|
||||
/// incompatible activity response is not retried here: it must remain a hard
|
||||
/// fail-closed result for the all-peer publication proof.
|
||||
async fn scanner_activity_with_retry(client: &PeerRestClient, host: &str) -> Result<ScannerPeerActivity> {
|
||||
let first = timeout(SCANNER_ACTIVITY_PROBE_TIMEOUT, client.scanner_activity()).await;
|
||||
let should_retry = match &first {
|
||||
Ok(Ok(_)) => false,
|
||||
Ok(Err(err)) => scanner_activity_should_retry(Some(err), false),
|
||||
Err(_) => scanner_activity_should_retry(None, true),
|
||||
};
|
||||
|
||||
match first {
|
||||
Ok(Ok(activity)) => return Ok(activity),
|
||||
Ok(Err(err)) if !should_retry => return Err(err),
|
||||
Ok(Err(err)) => {
|
||||
debug!(peer = host, error = %err, "scanner activity probe failed on first transport attempt; reconnecting");
|
||||
client.prepare_retry().await;
|
||||
}
|
||||
Err(_) => {
|
||||
debug!(peer = host, timeout = ?SCANNER_ACTIVITY_PROBE_TIMEOUT, "scanner activity probe timed out on first attempt; reconnecting");
|
||||
client.prepare_retry().await;
|
||||
}
|
||||
}
|
||||
|
||||
match timeout(SCANNER_ACTIVITY_PROBE_TIMEOUT, client.scanner_activity()).await {
|
||||
Ok(result) => result,
|
||||
Err(_) => {
|
||||
client.evict_connection().await;
|
||||
Err(Error::Timeout)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
|
||||
async fn call_peer_with_timeout<F, Fut>(
|
||||
timeout_dur: Duration,
|
||||
@@ -2926,20 +2849,6 @@ mod tests {
|
||||
assert!(err.to_string().contains("peer-1"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scanner_activity_retry_only_reconnects_transport_failures() {
|
||||
assert!(scanner_activity_should_retry(None, true));
|
||||
assert!(scanner_activity_should_retry(Some(&Error::other("connection refused")), false));
|
||||
assert!(!scanner_activity_should_retry(
|
||||
Some(&Error::other("peer returned an invalid scanner activity response proof")),
|
||||
false
|
||||
));
|
||||
assert!(!scanner_activity_should_retry(
|
||||
Some(&Error::from(tonic::Status::internal("peer rejected activity"))),
|
||||
false
|
||||
));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn scanner_dirty_usage_acknowledgement_rejects_missing_and_duplicate_targets() {
|
||||
let sys = NotificationSys {
|
||||
|
||||
@@ -570,13 +570,10 @@ impl ECStore {
|
||||
where
|
||||
S: EcstoreObjectIO + StorageNamespaceLocking<Error = Error, NamespaceLock = rustfs_lock::NamespaceLockWrapper>,
|
||||
{
|
||||
// Lock order: pool_meta_save_gate -> pool.bin -> rebalance.bin.
|
||||
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
|
||||
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
|
||||
pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
|
||||
// Classify the durable rebalance record while holding both namespace
|
||||
// fences. A terminal record is a no-op and must not depend on the
|
||||
// notification subsystem having published a fleet proof yet.
|
||||
let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
|
||||
let activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), fleet_proof).await?;
|
||||
let pool_meta = self
|
||||
.load_runtime_pool_meta_under_activation_fence(&mut pool_meta_guard, &activation_fence, "rebalance worker activation")
|
||||
.await?;
|
||||
@@ -600,17 +597,10 @@ impl ECStore {
|
||||
}
|
||||
|
||||
activation_fence.ensure_held()?;
|
||||
if !crate::services::rebalance::rebalance_requires_worker_activation(&persisted) {
|
||||
if !is_rebalance_conflicting_with_decommission(&persisted) {
|
||||
return Ok(RebalanceWorkerActivationFence::NotStartedTerminal);
|
||||
}
|
||||
|
||||
// Active worker admission still requires the fail-closed fleet proof.
|
||||
// Attach it immediately before the final fence validation so expiry or
|
||||
// topology changes are checked again at every later commit boundary.
|
||||
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
|
||||
activation_fence.set_fleet_proof(fleet_proof);
|
||||
activation_fence.ensure_held()?;
|
||||
|
||||
Ok(RebalanceWorkerActivationFence::Ready(Box::new(activation_fence)))
|
||||
}
|
||||
|
||||
@@ -1486,64 +1476,6 @@ mod tests {
|
||||
assert_activation_locks_released(&store).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn rebalance_worker_skips_terminal_metadata_without_fleet_proof() {
|
||||
let rebalance_id = "terminal-metadata-without-proof";
|
||||
let completed = RebalanceMeta {
|
||||
id: rebalance_id.to_string(),
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Completed,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(completed)).await;
|
||||
let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
|
||||
|
||||
let activation = store
|
||||
.fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
|
||||
.await
|
||||
.expect("terminal metadata should not require a fleet proof");
|
||||
assert!(matches!(activation, RebalanceWorkerActivationFence::NotStartedTerminal));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn rebalance_worker_still_requires_fleet_proof_for_active_metadata() {
|
||||
let rebalance_id = "active-metadata-without-proof";
|
||||
let active = RebalanceMeta {
|
||||
id: rebalance_id.to_string(),
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let (_temp_dirs, store, _other_store) = crate::services::rebalance::test_two_pool_stores(Some(active)).await;
|
||||
let _proof_guard = crate::services::notification_sys::without_cross_pool_fence_fleet_proof_for_test();
|
||||
|
||||
let err = match store
|
||||
.fence_rebalance_worker_activation(store.pools[0].clone(), rebalance_id)
|
||||
.await
|
||||
{
|
||||
Ok(_) => panic!("active metadata must not be admitted without a fleet proof"),
|
||||
Err(err) => err,
|
||||
};
|
||||
assert!(
|
||||
err.to_string()
|
||||
.contains("pool activation requires a live fleet capability proof")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn rebalance_activation_adopts_commit_after_post_save_fence_loss() {
|
||||
@@ -1667,8 +1599,6 @@ mod tests {
|
||||
async fn assert_real_activation_start_race(paused_kind: PoolActivationStartKind) {
|
||||
let (_temp_dirs, rebalance_store, decommission_store) =
|
||||
crate::services::rebalance::test_two_pool_stores_with_isolated_node_contexts(None).await;
|
||||
crate::services::rebalance::promote_test_pool_meta_to_v2(&rebalance_store).await;
|
||||
crate::services::rebalance::promote_test_pool_meta_to_v2(&decommission_store).await;
|
||||
let disk_stats = vec![
|
||||
DiskStat {
|
||||
total_space: 100,
|
||||
|
||||
@@ -214,14 +214,6 @@ pub(super) fn is_rebalance_in_progress(meta: &RebalanceMeta) -> bool {
|
||||
meta.pool_stats.iter().any(is_rebalance_pool_active)
|
||||
}
|
||||
|
||||
/// Persisted rebalance metadata requires worker activation only while it has
|
||||
/// not reached a durable terminal marker and at least one pool is still marked
|
||||
/// active. Merely finding `rebalance.bin` is not evidence that admission is
|
||||
/// required: terminal metadata is retained for status reporting.
|
||||
pub(crate) fn rebalance_requires_worker_activation(meta: &RebalanceMeta) -> bool {
|
||||
meta.stopped_at.is_none() && is_rebalance_in_progress(meta)
|
||||
}
|
||||
|
||||
pub(crate) fn is_rebalance_conflicting_with_decommission(meta: &RebalanceMeta) -> bool {
|
||||
is_rebalance_in_progress(meta)
|
||||
}
|
||||
|
||||
@@ -49,8 +49,8 @@ mod worker;
|
||||
|
||||
#[cfg(feature = "test-util")]
|
||||
pub use entry::test_util::PausedRebalanceEntryTestFixture;
|
||||
pub(crate) use meta::is_rebalance_conflicting_with_decommission;
|
||||
pub use meta::{decode_rebalance_stop_propagation_record, encode_rebalance_stop_propagation_record};
|
||||
pub(crate) use meta::{is_rebalance_conflicting_with_decommission, rebalance_requires_worker_activation};
|
||||
pub use types::{
|
||||
DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, RebalanceMeta,
|
||||
RebalanceStats, RebalanceStopPropagationRecord,
|
||||
@@ -111,17 +111,6 @@ pub(crate) async fn test_two_pool_stores_with_isolated_node_contexts(
|
||||
test_two_pool_stores_with_contexts(rebalance_meta, true).await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) async fn promote_test_pool_meta_to_v2(store: &std::sync::Arc<crate::store::ECStore>) {
|
||||
let mut pool_meta = store.pool_meta.read().await.clone();
|
||||
pool_meta.version = crate::core::pools::POOL_META_VERSION;
|
||||
pool_meta
|
||||
.save(store.pools.clone())
|
||||
.await
|
||||
.expect("test pool metadata should be promoted to V2");
|
||||
*store.pool_meta.write().await = pool_meta;
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
async fn test_two_pool_stores_with_contexts(
|
||||
rebalance_meta: Option<RebalanceMeta>,
|
||||
|
||||
@@ -22,11 +22,11 @@ use super::meta::{
|
||||
is_rebalance_in_progress, is_rebalance_meta_replaceable_for_new_id, is_rebalance_stopped_terminal_event,
|
||||
mark_rebalance_bucket_done, merge_rebalance_bucket_lists, merge_rebalance_meta, next_rebal_bucket_from_stat,
|
||||
percent_free_ratio, rebalance_goal_reached, rebalance_meta_load_no_data_error, rebalance_meta_load_unknown_format_error,
|
||||
rebalance_meta_load_unknown_version_error, rebalance_requires_worker_activation, record_rebalance_cleanup_warning_in_meta,
|
||||
remove_rebalanced_buckets_from_queue, resolve_next_rebalance_bucket, resolve_rebalance_participants,
|
||||
should_accept_rebalance_stats_update, should_ignore_rebalance_data_usage_cache, should_pool_participate,
|
||||
should_preserve_rebalance_stopped_state, should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state,
|
||||
take_bucket_from_rebalance_queue, validate_init_rebalance_state, validate_start_rebalance_state,
|
||||
rebalance_meta_load_unknown_version_error, record_rebalance_cleanup_warning_in_meta, remove_rebalanced_buckets_from_queue,
|
||||
resolve_next_rebalance_bucket, resolve_rebalance_participants, should_accept_rebalance_stats_update,
|
||||
should_ignore_rebalance_data_usage_cache, should_pool_participate, should_preserve_rebalance_stopped_state,
|
||||
should_skip_start_rebalance, stop_rebalance_meta_snapshot, stop_rebalance_state, take_bucket_from_rebalance_queue,
|
||||
validate_init_rebalance_state, validate_start_rebalance_state,
|
||||
};
|
||||
use super::migration::{
|
||||
MigrationBackend, MigrationVersionResult, migrate_entry_version, migrate_entry_version_with_retry_wait,
|
||||
@@ -3386,52 +3386,6 @@ fn test_is_rebalance_in_progress_only_started_participants() {
|
||||
assert!(is_rebalance_in_progress(&meta));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_rebalance_requires_worker_activation_only_for_active_non_stopped_metadata() {
|
||||
let now = OffsetDateTime::now_utc();
|
||||
let active = RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let stopped_active = RebalanceMeta {
|
||||
stopped_at: Some(now),
|
||||
pool_stats: active.pool_stats.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
assert!(rebalance_requires_worker_activation(&active));
|
||||
for status in [
|
||||
RebalStatus::Completed,
|
||||
RebalStatus::Stopped,
|
||||
RebalStatus::Failed,
|
||||
RebalStatus::None,
|
||||
] {
|
||||
let terminal = RebalanceMeta {
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
info: RebalanceInfo {
|
||||
status,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
assert!(
|
||||
!rebalance_requires_worker_activation(&terminal),
|
||||
"terminal status {status:?} must not resume"
|
||||
);
|
||||
}
|
||||
assert!(!rebalance_requires_worker_activation(&stopped_active));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_is_rebalance_conflicting_with_decommission_true_when_in_progress() {
|
||||
let now = OffsetDateTime::now_utc();
|
||||
|
||||
@@ -625,7 +625,7 @@ impl WarmBackend for MockWarmBackend {
|
||||
let reject_once = self
|
||||
.inner
|
||||
.reject_non_empty_remote_version_validations
|
||||
.try_update(Ordering::AcqRel, Ordering::Acquire, |remaining| remaining.checked_sub(1))
|
||||
.fetch_update(Ordering::AcqRel, Ordering::Acquire, |remaining| remaining.checked_sub(1))
|
||||
.is_ok();
|
||||
if reject_once || self.inner.reject_non_empty_remote_versions.load(Ordering::Acquire) {
|
||||
return Err(std::io::Error::other("mock warm backend requires an unversioned remote object"));
|
||||
|
||||
@@ -65,7 +65,6 @@ use crate::storage_api_contracts::{
|
||||
};
|
||||
use crate::{
|
||||
bucket::lifecycle::{
|
||||
get_lifecycle_config,
|
||||
tier_delete_journal::{TIER_DELETE_JOURNAL_PREFIX, decode_tier_delete_journal_entry},
|
||||
transition_transaction::{TRANSITION_TRANSACTION_RECORD_PREFIX, decode_transition_transaction_record},
|
||||
},
|
||||
@@ -82,7 +81,7 @@ use rustfs_filemeta::FileInfo;
|
||||
use rustfs_rio::HashReader;
|
||||
use rustfs_s3_client::{admin_handler_utils::AdminError, provider_versions::ProviderVersionCapabilities};
|
||||
use rustfs_utils::path::{SLASH_SEPARATOR, path_join};
|
||||
use s3s::{S3ErrorCode, dto::BucketLifecycleConfiguration};
|
||||
use s3s::S3ErrorCode;
|
||||
|
||||
use super::{
|
||||
tier_handlers::{ERR_TIER_BUCKET_NOT_FOUND, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_PERM_ERR},
|
||||
@@ -545,16 +544,6 @@ impl TierCandidateMutation {
|
||||
}
|
||||
}
|
||||
|
||||
/// `force` as supplied by the caller, or `false` for `Edit` (credential rebind never
|
||||
/// accepts a force override). Used to gate the lifecycle-config reference check, which
|
||||
/// unlike the persisted/physical-object reference checks is meant to be force-bypassable.
|
||||
fn force(&self) -> bool {
|
||||
match self {
|
||||
Self::Add(_, force) | Self::Remove(_, force) | Self::Clear(force) => *force,
|
||||
Self::Edit(_, _) => false,
|
||||
}
|
||||
}
|
||||
|
||||
fn explicit_tier_name(&self) -> Option<&str> {
|
||||
match self {
|
||||
Self::Add(config, _) => Some(&config.name),
|
||||
@@ -705,7 +694,6 @@ fn tier_backend_identity_admin_error(err: io::Error) -> AdminError {
|
||||
admin_err
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
trait TierReferenceProofStore:
|
||||
EcstoreObjectIO
|
||||
+ BucketOperations<Error = Error>
|
||||
@@ -717,27 +705,28 @@ trait TierReferenceProofStore:
|
||||
WalkOptions = TierReferenceProofWalkOptions,
|
||||
WalkCancellation = tokio_util::sync::CancellationToken,
|
||||
WalkResultSender = tokio::sync::mpsc::Sender<StorageObjectInfoOrErr<ObjectInfo, Error>>,
|
||||
> + Send
|
||||
+ Sync
|
||||
>
|
||||
{
|
||||
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>>;
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl TierReferenceProofStore for ECStore {
|
||||
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
|
||||
match get_lifecycle_config(bucket).await {
|
||||
Ok((config, _updated_at)) => Ok(Some(config)),
|
||||
Err(Error::ConfigNotFound) => Ok(None),
|
||||
Err(err) => Err(err),
|
||||
}
|
||||
}
|
||||
impl<T> TierReferenceProofStore for T where
|
||||
T: EcstoreObjectIO
|
||||
+ BucketOperations<Error = Error>
|
||||
+ ListOperations<
|
||||
Error = Error,
|
||||
ListObjectsV2Info = StorageListObjectsV2Info<ObjectInfo>,
|
||||
ListObjectVersionsInfo = StorageListObjectVersionsInfo<ObjectInfo>,
|
||||
ObjectInfoOrErr = StorageObjectInfoOrErr<ObjectInfo, Error>,
|
||||
WalkOptions = TierReferenceProofWalkOptions,
|
||||
WalkCancellation = tokio_util::sync::CancellationToken,
|
||||
WalkResultSender = tokio::sync::mpsc::Sender<StorageObjectInfoOrErr<ObjectInfo, Error>>,
|
||||
>
|
||||
{
|
||||
}
|
||||
|
||||
async fn ensure_no_authoritative_tier_object_references<S>(
|
||||
api: Arc<S>,
|
||||
affected_targets: &[TierMutationIntentTarget],
|
||||
force: bool,
|
||||
) -> std::result::Result<(), AdminError>
|
||||
where
|
||||
S: TierReferenceProofStore,
|
||||
@@ -750,13 +739,12 @@ where
|
||||
if targets.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
ensure_no_authoritative_target_references(api, &targets, force).await
|
||||
ensure_no_authoritative_target_references(api, &targets).await
|
||||
}
|
||||
|
||||
async fn ensure_no_authoritative_target_references<S>(
|
||||
api: Arc<S>,
|
||||
targets: &[TierMutationIntentTarget],
|
||||
force: bool,
|
||||
) -> std::result::Result<(), AdminError>
|
||||
where
|
||||
S: TierReferenceProofStore,
|
||||
@@ -766,13 +754,6 @@ where
|
||||
.await
|
||||
.map_err(tier_reference_proof_admin_error)?;
|
||||
for bucket in buckets {
|
||||
// The lifecycle-config check only warns about a *future* transition attempt against
|
||||
// this tier name, not an existing object/journal/transaction reference — unlike those,
|
||||
// it is meant to be bypassable with `force`, mirroring `TierConfigMgr::remove()`'s
|
||||
// `!force` gate on its own `driver.in_use()` probe (rustfs/backlog#2077).
|
||||
if !force {
|
||||
ensure_no_authoritative_lifecycle_references(api.as_ref(), &bucket.name, targets).await?;
|
||||
}
|
||||
let mut marker = None;
|
||||
let mut version_marker = None;
|
||||
loop {
|
||||
@@ -808,66 +789,6 @@ where
|
||||
ensure_no_authoritative_persisted_references(api, targets).await
|
||||
}
|
||||
|
||||
async fn ensure_no_authoritative_lifecycle_references(
|
||||
api: &impl TierReferenceProofStore,
|
||||
bucket: &str,
|
||||
targets: &[TierMutationIntentTarget],
|
||||
) -> std::result::Result<(), AdminError> {
|
||||
match api.lifecycle_config_for_reference_proof(bucket).await {
|
||||
Ok(Some(config)) => {
|
||||
if let Some(reference) = lifecycle_config_target_reference(&config, targets) {
|
||||
return Err(tier_reference_proof_lifecycle_in_use_error(
|
||||
reference.tier_name,
|
||||
bucket,
|
||||
reference.rule_id,
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(None) => {}
|
||||
Err(err) => {
|
||||
return Err(tier_reference_proof_admin_error(format!("bucket {bucket} lifecycle config: {err}")));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
struct TierLifecycleReference<'a> {
|
||||
tier_name: &'a str,
|
||||
rule_id: Option<&'a str>,
|
||||
}
|
||||
|
||||
fn lifecycle_config_target_reference<'a>(
|
||||
config: &'a BucketLifecycleConfiguration,
|
||||
targets: &[TierMutationIntentTarget],
|
||||
) -> Option<TierLifecycleReference<'a>> {
|
||||
for rule in &config.rules {
|
||||
let rule_id = rule.id.as_deref();
|
||||
if let Some(transitions) = &rule.transitions {
|
||||
for transition in transitions {
|
||||
let Some(storage_class) = &transition.storage_class else {
|
||||
continue;
|
||||
};
|
||||
let tier_name = storage_class.as_str();
|
||||
if !tier_name.is_empty() && targets.iter().any(|target| target.tier_name == tier_name) {
|
||||
return Some(TierLifecycleReference { tier_name, rule_id });
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(noncurrent_version_transitions) = &rule.noncurrent_version_transitions {
|
||||
for transition in noncurrent_version_transitions {
|
||||
let Some(storage_class) = &transition.storage_class else {
|
||||
continue;
|
||||
};
|
||||
let tier_name = storage_class.as_str();
|
||||
if !tier_name.is_empty() && targets.iter().any(|target| target.tier_name == tier_name) {
|
||||
return Some(TierLifecycleReference { tier_name, rule_id });
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
async fn ensure_no_authoritative_persisted_references<S>(
|
||||
api: Arc<S>,
|
||||
targets: &[TierMutationIntentTarget],
|
||||
@@ -999,15 +920,6 @@ fn tier_reference_proof_persisted_in_use_error(tier_name: &str, object: &str) ->
|
||||
err
|
||||
}
|
||||
|
||||
fn tier_reference_proof_lifecycle_in_use_error(tier_name: &str, bucket: &str, rule_id: Option<&str>) -> AdminError {
|
||||
let mut err = ERR_TIER_BACKEND_IN_USE.clone();
|
||||
err.message = match rule_id {
|
||||
Some(rule_id) => format!("Remote tier {tier_name} is still referenced by lifecycle rule {rule_id} in bucket {bucket}"),
|
||||
None => format!("Remote tier {tier_name} is still referenced by a lifecycle rule in bucket {bucket}"),
|
||||
};
|
||||
err
|
||||
}
|
||||
|
||||
fn tier_reference_proof_admin_error(err: impl std::fmt::Display) -> AdminError {
|
||||
let mut admin_err = ERR_TIER_INVALID_CONFIG.clone();
|
||||
admin_err.message = format!("Remote tier reference proof failed: {err}");
|
||||
@@ -1669,7 +1581,7 @@ impl TierOperationLease {
|
||||
) -> std::result::Result<Self, AdminError> {
|
||||
inner
|
||||
.active_leases
|
||||
.try_update(Ordering::AcqRel, Ordering::Acquire, |active| active.checked_add(1))
|
||||
.fetch_update(Ordering::AcqRel, Ordering::Acquire, |active| active.checked_add(1))
|
||||
.map_err(|_| {
|
||||
let mut err = ERR_TIER_INVALID_CONFIG.clone();
|
||||
err.message = "Remote tier operation lease capacity exhausted".to_string();
|
||||
@@ -1688,7 +1600,7 @@ impl Drop for TierOperationLease {
|
||||
let result = self
|
||||
.inner
|
||||
.active_leases
|
||||
.try_update(Ordering::AcqRel, Ordering::Acquire, |active| active.checked_sub(1));
|
||||
.fetch_update(Ordering::AcqRel, Ordering::Acquire, |active| active.checked_sub(1));
|
||||
match result {
|
||||
Ok(1) => self.inner.drained.notify_one(),
|
||||
Ok(_) => {}
|
||||
@@ -2563,24 +2475,6 @@ impl TierConfigMgr {
|
||||
})?;
|
||||
}
|
||||
|
||||
// The Azure warm backend goes through the same S3-compatible TransitionClient as every
|
||||
// other provider (backlog#2055): it has no Azure Blob-native client and no Azure AD
|
||||
// dependency, so `storage_class` and `sp_auth` cannot be honored today even though the
|
||||
// config type carries them. Reject them explicitly here instead of silently accepting
|
||||
// and then dropping them at the WarmBackendAzure construction boundary.
|
||||
if matches!(&tier_config.tier_type, TierType::Azure)
|
||||
&& let Some(azure) = tier_config.azure.as_ref()
|
||||
{
|
||||
let sp_auth_set = !azure.sp_auth.tenant_id.is_empty()
|
||||
|| !azure.sp_auth.client_id.is_empty()
|
||||
|| !azure.sp_auth.client_secret.is_empty();
|
||||
if !azure.storage_class.is_empty() || sp_auth_set {
|
||||
let mut err = ERR_TIER_INVALID_CONFIG.clone();
|
||||
err.message = "Azure remote tiers do not support storageClass or spAuth yet; leave both unset".to_string();
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
|
||||
let d = new_warm_backend(&tier_config, true).await?;
|
||||
|
||||
if !force {
|
||||
@@ -3343,7 +3237,6 @@ impl TierConfigMgr {
|
||||
)));
|
||||
}
|
||||
let explicit_tier_name = mutation.explicit_tier_name().map(str::to_string);
|
||||
let mutation_force = mutation.force();
|
||||
let current_for_targets = TierConfigMgr {
|
||||
driver_cache: HashMap::new(),
|
||||
tiers: candidate
|
||||
@@ -3376,7 +3269,7 @@ impl TierConfigMgr {
|
||||
let affected_targets =
|
||||
build_tier_mutation_affected_targets(mutation_kind, proof_targets, ¤t_for_targets, &candidate)
|
||||
.map_err(TierConfigUpdateError::Publish)?;
|
||||
ensure_no_authoritative_tier_object_references(api.clone(), &affected_targets, mutation_force)
|
||||
ensure_no_authoritative_tier_object_references(api.clone(), &affected_targets)
|
||||
.await
|
||||
.map_err(TierConfigUpdateError::Publish)?;
|
||||
let coordinator_intent =
|
||||
@@ -5415,10 +5308,6 @@ mod tests {
|
||||
endpoints::{Endpoints, PoolEndpoints, SetupType},
|
||||
};
|
||||
use crate::services::tier::tier_mutation_intent::TIER_MUTATION_INTENT_RECORD_PREFIX;
|
||||
use s3s::dto::{
|
||||
BucketLifecycleConfiguration, ExpirationStatus, LifecycleRule, NoncurrentVersionTransition, Transition,
|
||||
TransitionStorageClass,
|
||||
};
|
||||
|
||||
struct SetupTypeGuard {
|
||||
previous: SetupType,
|
||||
@@ -6204,13 +6093,6 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl TierReferenceProofStore for LockingTierConfigStore {
|
||||
async fn lifecycle_config_for_reference_proof(&self, _bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn tier_config_update_path_acquires_meta_namespace_sidecar_lock_before_save() {
|
||||
let manager = TierConfigMgr::new();
|
||||
@@ -6421,57 +6303,6 @@ mod tests {
|
||||
assert!(mgr.tiers.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_add_rejects_azure_storage_class_before_backend_setup() {
|
||||
let mut mgr = empty_mgr();
|
||||
let mut tier = build_azure_tier("account-a");
|
||||
tier.azure.as_mut().expect("Azure payload should exist").storage_class = "HOT".to_string();
|
||||
|
||||
let err = mgr
|
||||
.add(tier, true)
|
||||
.await
|
||||
.expect_err("a non-empty Azure storageClass must be rejected before backend setup");
|
||||
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
|
||||
assert!(err.message.contains("storageClass"), "{}", err.message);
|
||||
assert!(mgr.tiers.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_add_rejects_azure_partial_sp_auth_before_backend_setup() {
|
||||
// Only `tenant_id` is set: `TierAzure::is_sp_enabled()`-style "all three fields"
|
||||
// logic would miss this, so the check must reject on *any* sp_auth sub-field
|
||||
// being non-empty rather than requiring all three.
|
||||
let mut mgr = empty_mgr();
|
||||
let mut tier = build_azure_tier("account-a");
|
||||
tier.azure.as_mut().expect("Azure payload should exist").sp_auth.tenant_id = "tenant".to_string();
|
||||
|
||||
let err = mgr
|
||||
.add(tier, true)
|
||||
.await
|
||||
.expect_err("a partially-filled Azure spAuth must be rejected before backend setup");
|
||||
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
|
||||
assert!(err.message.contains("spAuth"), "{}", err.message);
|
||||
assert!(mgr.tiers.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_add_does_not_reject_azure_config_without_storage_class_or_sp_auth() {
|
||||
// A plain Azure config (the common case: static access/secret key, no
|
||||
// storageClass, no spAuth) must sail past the new gate. `new_warm_backend`
|
||||
// builds the S3-compatible client lazily (no eager DNS/connect), so with
|
||||
// `force: true` (which also skips the `in_use` probe) this succeeds even
|
||||
// against a fake endpoint — the point here is only that the gate itself
|
||||
// does not fire.
|
||||
let mut mgr = empty_mgr();
|
||||
let tier = build_azure_tier("account-a");
|
||||
let tier_name = tier.name.clone();
|
||||
|
||||
mgr.add(tier, true)
|
||||
.await
|
||||
.expect("a config with no storageClass/spAuth must not trip the new gate");
|
||||
assert!(mgr.tiers.contains_key(&tier_name));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_add_rejects_reserved_names() {
|
||||
// Supersedes the former `test_add_does_not_reserve_standard_name_regression_anchor`
|
||||
@@ -11465,7 +11296,6 @@ mod tests {
|
||||
lock_manager: Arc<rustfs_lock::GlobalLockManager>,
|
||||
lock_requests: Mutex<Vec<(String, String)>>,
|
||||
listed_versions: Mutex<Vec<ObjectInfo>>,
|
||||
lifecycle_configs: Mutex<HashMap<String, BucketLifecycleConfiguration>>,
|
||||
}
|
||||
|
||||
impl Default for CasConfigStore {
|
||||
@@ -11486,7 +11316,6 @@ mod tests {
|
||||
lock_manager: Arc::new(rustfs_lock::GlobalLockManager::new()),
|
||||
lock_requests: Mutex::new(Vec::new()),
|
||||
listed_versions: Mutex::new(Vec::new()),
|
||||
lifecycle_configs: Mutex::new(HashMap::new()),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -11513,13 +11342,6 @@ mod tests {
|
||||
.push(object);
|
||||
}
|
||||
|
||||
fn add_lifecycle_config(&self, bucket: &str, config: BucketLifecycleConfiguration) {
|
||||
self.lifecycle_configs
|
||||
.lock()
|
||||
.expect("tier reference fixture should not poison")
|
||||
.insert(bucket.to_string(), config);
|
||||
}
|
||||
|
||||
async fn insert_config_object(&self, object: String, data: Vec<u8>) {
|
||||
self.objects
|
||||
.lock()
|
||||
@@ -11928,7 +11750,7 @@ mod tests {
|
||||
let should_pause =
|
||||
match barrier
|
||||
.matches_before_pause
|
||||
.try_update(Ordering::SeqCst, Ordering::SeqCst, |remaining| remaining.checked_sub(1))
|
||||
.fetch_update(Ordering::SeqCst, Ordering::SeqCst, |remaining| remaining.checked_sub(1))
|
||||
{
|
||||
Ok(_) => false,
|
||||
Err(_) => barrier.armed.swap(false, Ordering::SeqCst),
|
||||
@@ -12095,18 +11917,6 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl TierReferenceProofStore for CasConfigStore {
|
||||
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
|
||||
Ok(self
|
||||
.lifecycle_configs
|
||||
.lock()
|
||||
.expect("tier reference fixture should not poison")
|
||||
.get(bucket)
|
||||
.cloned())
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn remove_and_clear_full_update_paths_preserve_force() {
|
||||
let remove_store = Arc::new(CasConfigStore::default());
|
||||
@@ -12198,70 +12008,6 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn force_remove_and_save_bypasses_lifecycle_only_reference() {
|
||||
// rustfs/rustfs#6832: reproduces the admin RemoveTier path (not just the lower-level
|
||||
// reference-proof function) for a tier with zero transitioned objects but a lifecycle
|
||||
// rule still pointing at it — the exact shape of
|
||||
// `test_manual_transition_async_tier_failure_reports_terminal_partial` in e2e_test,
|
||||
// which force-removes a tier a lifecycle rule still references to simulate a
|
||||
// decommissioned backend.
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
let tier = build_rustfs_tier("COLD-A");
|
||||
let mut persisted = empty_mgr();
|
||||
persisted.tiers.insert("COLD-A".to_string(), tier.clone_with_credentials());
|
||||
persisted
|
||||
.save_tiering_config_if_current(store.clone(), None)
|
||||
.await
|
||||
.expect("reference proof fixture should persist");
|
||||
let config = BucketLifecycleConfiguration {
|
||||
expiry_updated_at: None,
|
||||
rules: vec![LifecycleRule {
|
||||
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
|
||||
expiration: None,
|
||||
abort_incomplete_multipart_upload: None,
|
||||
del_marker_expiration: None,
|
||||
filter: None,
|
||||
id: Some("move-current".to_string()),
|
||||
noncurrent_version_expiration: None,
|
||||
noncurrent_version_transitions: None,
|
||||
prefix: None,
|
||||
transitions: Some(vec![Transition {
|
||||
days: Some(1),
|
||||
date: None,
|
||||
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
|
||||
}]),
|
||||
}],
|
||||
};
|
||||
// CasConfigStore::list_bucket derives its fake bucket listing from listed_versions, so
|
||||
// a bucket with a lifecycle rule but zero objects still needs a decoy entry to be
|
||||
// visible to the reference-proof walk at all (mirrors production, where list_bucket
|
||||
// enumerates real buckets independent of their contents).
|
||||
store.add_listed_version(ObjectInfo {
|
||||
bucket: "photos".to_string(),
|
||||
name: "safe.txt".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
store.add_lifecycle_config("photos", config);
|
||||
|
||||
let manager = TierConfigMgr::new();
|
||||
manager.write().await.tiers.insert("COLD-A".to_string(), tier);
|
||||
TierConfigMgr::remove_and_save_with(&manager, store.clone(), "COLD-A", true)
|
||||
.await
|
||||
.expect("force remove must bypass a lifecycle-config-only reference");
|
||||
|
||||
assert!(!manager.read().await.tiers.contains_key("COLD-A"));
|
||||
assert!(
|
||||
!load_tier_config_for_update(store)
|
||||
.await
|
||||
.expect("config should still reload")
|
||||
.0
|
||||
.tiers
|
||||
.contains_key("COLD-A"),
|
||||
"force removal must persist the empty candidate"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn zero_reference_proof_blocks_clear_before_config_save() {
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
@@ -12327,7 +12073,7 @@ mod tests {
|
||||
"COLD-A",
|
||||
Some(replacement_identity),
|
||||
));
|
||||
ensure_no_authoritative_tier_object_references(store.clone(), std::slice::from_ref(&target), false)
|
||||
ensure_no_authoritative_tier_object_references(store.clone(), std::slice::from_ref(&target))
|
||||
.await
|
||||
.expect("references already written with the new destination identity should not block rebind");
|
||||
|
||||
@@ -12337,154 +12083,15 @@ mod tests {
|
||||
"COLD-A",
|
||||
Some(current_identity),
|
||||
));
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target], false)
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target])
|
||||
.await
|
||||
.expect_err("references to the old destination identity must block rebind");
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
assert!(err.message.contains("photos/2026/old-destination.jpg"), "{}", err.message);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn zero_reference_proof_blocks_lifecycle_transition_references() {
|
||||
let current = build_rustfs_tier("COLD-A");
|
||||
let current_identity = tier_backend_identity(¤t).expect("current identity should encode");
|
||||
let target = TierMutationIntentTarget {
|
||||
tier_name: "COLD-A".to_string(),
|
||||
old_backend_identity: Some(current_identity),
|
||||
new_backend_identity: None,
|
||||
};
|
||||
let config = BucketLifecycleConfiguration {
|
||||
expiry_updated_at: None,
|
||||
rules: vec![LifecycleRule {
|
||||
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
|
||||
expiration: None,
|
||||
abort_incomplete_multipart_upload: None,
|
||||
del_marker_expiration: None,
|
||||
filter: None,
|
||||
id: Some("move-current".to_string()),
|
||||
noncurrent_version_expiration: None,
|
||||
noncurrent_version_transitions: None,
|
||||
prefix: None,
|
||||
transitions: Some(vec![Transition {
|
||||
days: Some(1),
|
||||
date: None,
|
||||
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
|
||||
}]),
|
||||
}],
|
||||
};
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
store.add_listed_version(ObjectInfo {
|
||||
bucket: "photos".to_string(),
|
||||
name: "safe.txt".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
store.add_lifecycle_config("photos", config);
|
||||
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target], false)
|
||||
.await
|
||||
.expect_err("lifecycle rule should block deletion");
|
||||
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
assert!(err.message.contains("move-current"), "{}", err.message);
|
||||
assert!(err.message.contains("photos"), "{}", err.message);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn zero_reference_proof_blocks_lifecycle_noncurrent_transition_references() {
|
||||
let current = build_rustfs_tier("COLD-A");
|
||||
let current_identity = tier_backend_identity(¤t).expect("current identity should encode");
|
||||
let target = TierMutationIntentTarget {
|
||||
tier_name: "COLD-A".to_string(),
|
||||
old_backend_identity: Some(current_identity),
|
||||
new_backend_identity: None,
|
||||
};
|
||||
let config = BucketLifecycleConfiguration {
|
||||
expiry_updated_at: None,
|
||||
rules: vec![LifecycleRule {
|
||||
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
|
||||
expiration: None,
|
||||
abort_incomplete_multipart_upload: None,
|
||||
del_marker_expiration: None,
|
||||
filter: None,
|
||||
id: Some("move-noncurrent".to_string()),
|
||||
noncurrent_version_expiration: None,
|
||||
noncurrent_version_transitions: Some(vec![NoncurrentVersionTransition {
|
||||
noncurrent_days: Some(1),
|
||||
newer_noncurrent_versions: None,
|
||||
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
|
||||
}]),
|
||||
prefix: None,
|
||||
transitions: None,
|
||||
}],
|
||||
};
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
store.add_listed_version(ObjectInfo {
|
||||
bucket: "photos".to_string(),
|
||||
name: "safe.txt".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
store.add_lifecycle_config("photos", config);
|
||||
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target], false)
|
||||
.await
|
||||
.expect_err("lifecycle rule should block deletion");
|
||||
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
assert!(err.message.contains("move-noncurrent"), "{}", err.message);
|
||||
assert!(err.message.contains("photos"), "{}", err.message);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn zero_reference_proof_force_true_skips_lifecycle_reference_check() {
|
||||
// rustfs/rustfs#6832: force=true must bypass a lifecycle-config-only reference,
|
||||
// mirroring the existing `!force` gate on `TierConfigMgr::remove()`'s own
|
||||
// `driver.in_use()` probe. It must NOT bypass real object/journal/transaction
|
||||
// references — those stay force-immune and are covered separately below.
|
||||
let current = build_rustfs_tier("COLD-A");
|
||||
let current_identity = tier_backend_identity(¤t).expect("current identity should encode");
|
||||
let target = TierMutationIntentTarget {
|
||||
tier_name: "COLD-A".to_string(),
|
||||
old_backend_identity: Some(current_identity),
|
||||
new_backend_identity: None,
|
||||
};
|
||||
let config = BucketLifecycleConfiguration {
|
||||
expiry_updated_at: None,
|
||||
rules: vec![LifecycleRule {
|
||||
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
|
||||
expiration: None,
|
||||
abort_incomplete_multipart_upload: None,
|
||||
del_marker_expiration: None,
|
||||
filter: None,
|
||||
id: Some("move-current".to_string()),
|
||||
noncurrent_version_expiration: None,
|
||||
noncurrent_version_transitions: None,
|
||||
prefix: None,
|
||||
transitions: Some(vec![Transition {
|
||||
days: Some(1),
|
||||
date: None,
|
||||
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
|
||||
}]),
|
||||
}],
|
||||
};
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
store.add_listed_version(ObjectInfo {
|
||||
bucket: "photos".to_string(),
|
||||
name: "safe.txt".to_string(),
|
||||
..Default::default()
|
||||
});
|
||||
store.add_lifecycle_config("photos", config);
|
||||
|
||||
ensure_no_authoritative_tier_object_references(store, &[target], true)
|
||||
.await
|
||||
.expect("force=true must bypass a lifecycle-config-only reference");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn zero_reference_proof_blocks_persisted_journal_transaction_and_free_version_references() {
|
||||
// All three sub-checks below pass `force: true` to pin that physical/persisted
|
||||
// references stay force-immune post rustfs/rustfs#6832 (only the lifecycle-config
|
||||
// check, covered by `zero_reference_proof_force_true_skips_lifecycle_reference_check`
|
||||
// above, is meant to be force-bypassable).
|
||||
let current = build_rustfs_tier("COLD-A");
|
||||
let current_identity = tier_backend_identity(¤t).expect("current identity should encode");
|
||||
let replacement = build_azure_tier("account-a");
|
||||
@@ -12513,9 +12120,9 @@ mod tests {
|
||||
.expect("journal record should encode"),
|
||||
)
|
||||
.await;
|
||||
let err = ensure_no_authoritative_tier_object_references(journal_store, std::slice::from_ref(&target), true)
|
||||
let err = ensure_no_authoritative_tier_object_references(journal_store, std::slice::from_ref(&target))
|
||||
.await
|
||||
.expect_err("unfinished delete journal for the old backend must block rebind even with force");
|
||||
.expect_err("unfinished delete journal for the old backend must block rebind");
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
assert!(err.message.contains(TIER_DELETE_JOURNAL_PREFIX), "{}", err.message);
|
||||
|
||||
@@ -12551,9 +12158,9 @@ mod tests {
|
||||
transaction.encode().expect("transaction record should encode"),
|
||||
)
|
||||
.await;
|
||||
let err = ensure_no_authoritative_tier_object_references(transaction_store, std::slice::from_ref(&target), true)
|
||||
let err = ensure_no_authoritative_tier_object_references(transaction_store, std::slice::from_ref(&target))
|
||||
.await
|
||||
.expect_err("unfinished transition transaction for the old backend must block rebind even with force");
|
||||
.expect_err("unfinished transition transaction for the old backend must block rebind");
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
assert!(err.message.contains(TRANSITION_TRANSACTION_RECORD_PREFIX), "{}", err.message);
|
||||
|
||||
@@ -12561,13 +12168,13 @@ mod tests {
|
||||
let mut free_version = transitioned_tier_object("photos", "2026/free-version.jpg", "COLD-A", Some(current_identity));
|
||||
free_version.transitioned_object.status = "pending".to_string();
|
||||
free_version.transitioned_object.free_version = true;
|
||||
ensure_no_authoritative_tier_object_references(free_version_store.clone(), std::slice::from_ref(&target), true)
|
||||
ensure_no_authoritative_tier_object_references(free_version_store.clone(), std::slice::from_ref(&target))
|
||||
.await
|
||||
.expect("empty free-version fixture should permit rebind");
|
||||
free_version_store.add_listed_version(free_version);
|
||||
let err = ensure_no_authoritative_tier_object_references(free_version_store, &[target], true)
|
||||
let err = ensure_no_authoritative_tier_object_references(free_version_store, &[target])
|
||||
.await
|
||||
.expect_err("recoverable free version for the old backend must block rebind even with force");
|
||||
.expect_err("recoverable free version for the old backend must block rebind");
|
||||
assert_eq!(err.code, ERR_TIER_BACKEND_IN_USE.code);
|
||||
assert!(err.message.contains("photos/2026/free-version.jpg"), "{}", err.message);
|
||||
}
|
||||
@@ -12591,7 +12198,7 @@ mod tests {
|
||||
}
|
||||
store.omit_truncated_reference_marker();
|
||||
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target], false)
|
||||
let err = ensure_no_authoritative_tier_object_references(store, &[target])
|
||||
.await
|
||||
.expect_err("truncated authoritative reference scan without a marker must fail closed");
|
||||
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
|
||||
|
||||
@@ -646,6 +646,12 @@ pub struct TierAzure {
|
||||
pub sp_auth: ServicePrincipalAuth,
|
||||
}
|
||||
|
||||
impl TierAzure {
|
||||
pub fn is_sp_enabled(&self) -> bool {
|
||||
!self.sp_auth.tenant_id.is_empty() && !self.sp_auth.client_id.is_empty() && !self.sp_auth.client_secret.is_empty()
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
fn AzureServicePrincipal(tenantID, clientID, clientSecret string) func(az *TierAzure) error {
|
||||
return func(az *TierAzure) error {
|
||||
|
||||
@@ -36,14 +36,11 @@ use crate::services::tier::{
|
||||
};
|
||||
use bytes::Bytes;
|
||||
use http::StatusCode;
|
||||
use rustfs_s3_client::credentials::{Credentials, SignatureType, Static, Value};
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, Options, TransitionClient, TransitionCore};
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::{AdvancedPutOptions, PutObjectOptions},
|
||||
transition_api::{ReadCloser, ReaderImpl},
|
||||
};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
use rustfs_utils::http::headers::{
|
||||
CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _,
|
||||
};
|
||||
@@ -53,7 +50,6 @@ use s3s::header::{
|
||||
X_AMZ_STORAGE_CLASS,
|
||||
};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use time::OffsetDateTime;
|
||||
use time::format_description::well_known::{Rfc2822, Rfc3339};
|
||||
use tracing::{info, warn};
|
||||
@@ -62,11 +58,6 @@ pub type WarmBackendImpl = Box<dyn WarmBackend + Send + Sync + 'static>;
|
||||
|
||||
const PROBE_OBJECT: &str = "probeobject";
|
||||
|
||||
/// Largest object the S3-compatible warm backends accept for a multipart put.
|
||||
pub(crate) const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
/// Part-count ceiling S3-compatible services impose on a multipart upload.
|
||||
pub(crate) const MAX_PARTS_COUNT: i64 = 10000;
|
||||
|
||||
#[derive(Default)]
|
||||
pub struct WarmBackendGetOpts {
|
||||
pub start_offset: i64,
|
||||
@@ -231,121 +222,6 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
|
||||
opts
|
||||
}
|
||||
|
||||
/// Connection parameters every S3-compatible warm backend provider supplies.
|
||||
///
|
||||
/// The Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, and RustFS backends all
|
||||
/// wrap [`WarmBackendS3`] around a statically-credentialed [`TransitionClient`]
|
||||
/// built from exactly these values. `bucket_lookup` is a parameter rather than a
|
||||
/// constant because the providers split into two families: Aliyun, Azure,
|
||||
/// Huaweicloud, and Tencent pin [`BucketLookupType::BucketLookupDNS`], while
|
||||
/// MinIO, R2, and RustFS leave it at [`BucketLookupType::BucketLookupAuto`].
|
||||
pub(crate) struct S3CompatibleWarmBackendParams<'a> {
|
||||
pub endpoint: &'a str,
|
||||
pub access_key: &'a str,
|
||||
pub secret_key: &'a str,
|
||||
pub bucket: &'a str,
|
||||
pub prefix: &'a str,
|
||||
pub region: &'a str,
|
||||
pub bucket_lookup: BucketLookupType,
|
||||
/// Tag handed to [`TransitionClient::new`] so per-provider client behavior
|
||||
/// and metrics stay attributable.
|
||||
pub provider_tag: &'a str,
|
||||
/// SSRF guard run against the parsed endpoint once it's known to have a
|
||||
/// host. Almost every provider passes [`rustfs_utils::egress::validate_outbound_url`]
|
||||
/// unchanged; RustFS passes its own wrapper that adds a debug-only,
|
||||
/// env-gated loopback exception for its e2e tier tests (see
|
||||
/// rustfs/rustfs#6773) — the shared constructor stays the single call
|
||||
/// site either way, so no provider can silently end up unvalidated.
|
||||
pub validate_endpoint: fn(&url::Url) -> Result<(), rustfs_utils::egress::OutboundUrlError>,
|
||||
}
|
||||
|
||||
/// Build the [`WarmBackendS3`] shared by the S3-compatible warm backend providers.
|
||||
///
|
||||
/// Credential, bucket, and endpoint validation run in this order because the
|
||||
/// existing provider constructors report the first failure they hit, and their
|
||||
/// error texts are user-visible through the tier admin API.
|
||||
pub(crate) async fn new_s3_compatible_warm_backend(
|
||||
params: S3CompatibleWarmBackendParams<'_>,
|
||||
) -> Result<WarmBackendS3, std::io::Error> {
|
||||
if params.access_key.is_empty() || params.secret_key.is_empty() {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if params.bucket.is_empty() {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(params.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: params.access_key.to_string(),
|
||||
secret_access_key: params.secret_key.to_string(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: params.region.to_string(),
|
||||
bucket_lookup: params.bucket_lookup,
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
// Runs after the host-presence check above (not immediately after Url::parse) so a
|
||||
// host-less endpoint still reports this constructor's own "missing host" text instead of
|
||||
// validate_endpoint's differently-worded rejection for the same input.
|
||||
(params.validate_endpoint)(&u).map_err(|err| std::io::Error::other(format!("tier endpoint is not allowed: {err}")))?;
|
||||
let client =
|
||||
TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, params.provider_tag).await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: params.bucket.to_string(),
|
||||
prefix: params.prefix.strip_suffix("/").unwrap_or(params.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Round the multipart part size up to a whole multiple of `min_part_size` that
|
||||
/// keeps the upload within [`MAX_PARTS_COUNT`] parts.
|
||||
///
|
||||
/// `object_size == -1` means "length unknown", so the caller is charged the
|
||||
/// worst case of a full [`MAX_MULTIPART_PUT_OBJECT_SIZE`] object.
|
||||
pub(crate) fn optimal_part_size(object_size: i64, min_part_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
|
||||
let configured_part_size = min_part_size;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(min_part_size);
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
pub async fn check_warm_backend(w: Option<&WarmBackendImpl>) -> Result<(), AdminError> {
|
||||
let w = w.ok_or_else(|| ERR_TIER_NOT_FOUND.clone())?;
|
||||
w.validate().await.map_err(|_| ERR_TIER_INVALID_CONFIG.clone())?;
|
||||
@@ -927,194 +803,6 @@ mod tests {
|
||||
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
|
||||
}
|
||||
|
||||
/// Every S3-compatible provider file pins this same floor today.
|
||||
const PROVIDER_MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
fn s3_compatible_params(endpoint: &str) -> S3CompatibleWarmBackendParams<'_> {
|
||||
S3CompatibleWarmBackendParams {
|
||||
endpoint,
|
||||
access_key: "access",
|
||||
secret_key: "secret",
|
||||
bucket: "tier-bucket",
|
||||
prefix: "archive",
|
||||
region: "us-east-1",
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
provider_tag: "aliyun",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
}
|
||||
}
|
||||
|
||||
/// `WarmBackendS3` has no `Debug`, so `Result::expect_err` is unavailable.
|
||||
async fn init_error(params: S3CompatibleWarmBackendParams<'_>, must_fail_because: &str) -> std::io::Error {
|
||||
match new_s3_compatible_warm_backend(params).await {
|
||||
Ok(_) => panic!("{must_fail_because}"),
|
||||
Err(err) => err,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_rejects_missing_credentials_before_parsing_the_endpoint() {
|
||||
let mut params = s3_compatible_params("://not-a-url");
|
||||
params.access_key = "";
|
||||
let err = init_error(params, "an empty access key must be rejected").await;
|
||||
assert_eq!(err.to_string(), "both access and secret keys are required");
|
||||
|
||||
let mut params = s3_compatible_params("://not-a-url");
|
||||
params.secret_key = "";
|
||||
let err = init_error(params, "an empty secret key must be rejected").await;
|
||||
assert_eq!(err.to_string(), "both access and secret keys are required");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_rejects_an_empty_bucket_before_parsing_the_endpoint() {
|
||||
let mut params = s3_compatible_params("://not-a-url");
|
||||
params.bucket = "";
|
||||
|
||||
let err = init_error(params, "an empty bucket must be rejected").await;
|
||||
|
||||
assert_eq!(err.to_string(), "no bucket name was provided");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_rejects_an_unparsable_endpoint() {
|
||||
let err = init_error(s3_compatible_params("://not-a-url"), "an endpoint that is not a URL must be rejected").await;
|
||||
|
||||
assert_eq!(err.to_string(), url::ParseError::RelativeUrlWithoutBase.to_string());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_rejects_an_endpoint_without_a_host() {
|
||||
let err = init_error(s3_compatible_params("rustfs://"), "an endpoint without a host must be rejected").await;
|
||||
|
||||
assert_eq!(err.to_string(), "Invalid endpoint URL: missing host");
|
||||
}
|
||||
|
||||
/// Every migrated provider that uses `validate_outbound_url` directly (all
|
||||
/// but RustFS, which injects its own debug-only, env-gated wrapper — see
|
||||
/// rustfs/rustfs#6773) goes through this one construction path, so the
|
||||
/// SSRF guard only needs to be pinned here rather than once per provider
|
||||
/// file (see backlog#2040's migrate steps and rustfs/rustfs#6764).
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_rejects_a_loopback_endpoint_before_any_network_setup() {
|
||||
let err = init_error(s3_compatible_params("https://127.0.0.1:9000"), "a loopback endpoint must be rejected").await;
|
||||
|
||||
assert!(err.to_string().contains("not allowed"), "unexpected error: {err}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_carries_provider_options_to_the_transition_client() {
|
||||
let backend = new_s3_compatible_warm_backend(s3_compatible_params("http://tier.example.com:9000"))
|
||||
.await
|
||||
.expect("a well-formed S3-compatible tier config should initialize offline");
|
||||
|
||||
assert_eq!(backend.bucket, "tier-bucket");
|
||||
assert_eq!(backend.prefix, "archive");
|
||||
assert_eq!(backend.storage_class, "");
|
||||
assert!(!backend.client.secure);
|
||||
assert_eq!(backend.client.endpoint_url.scheme(), "http");
|
||||
assert_eq!(backend.client.endpoint_url.host_str(), Some("tier.example.com"));
|
||||
assert_eq!(backend.client.endpoint_url.port(), Some(9000));
|
||||
assert_eq!(backend.client.region, "us-east-1");
|
||||
assert_eq!(backend.client.lookup, BucketLookupType::BucketLookupDNS);
|
||||
// The provider constructors all request `trailing_headers: true`, but
|
||||
// `TransitionClient` gates the feature on an explicitly overridden SigV4
|
||||
// signer, which none of them set. Pin the resulting `false` so migrating
|
||||
// a provider onto this constructor cannot silently flip wire behavior.
|
||||
assert!(!backend.client.trailing_header_support);
|
||||
assert_eq!(backend.client.tier_type, "aliyun");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_derives_tls_and_the_default_port_from_the_scheme() {
|
||||
let secure = new_s3_compatible_warm_backend(s3_compatible_params("https://tier.example.com"))
|
||||
.await
|
||||
.expect("an https endpoint should initialize offline");
|
||||
assert!(secure.client.secure);
|
||||
assert_eq!(secure.client.endpoint_url.scheme(), "https");
|
||||
assert_eq!(secure.client.endpoint_url.port_or_known_default(), Some(443));
|
||||
|
||||
let insecure = new_s3_compatible_warm_backend(s3_compatible_params("http://tier.example.com"))
|
||||
.await
|
||||
.expect("an http endpoint should initialize offline");
|
||||
assert!(!insecure.client.secure);
|
||||
assert_eq!(insecure.client.endpoint_url.scheme(), "http");
|
||||
assert_eq!(insecure.client.endpoint_url.port_or_known_default(), Some(80));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_strips_only_a_trailing_prefix_separator() {
|
||||
let mut params = s3_compatible_params("http://tier.example.com:9000");
|
||||
params.prefix = "archive/";
|
||||
let trimmed = new_s3_compatible_warm_backend(params)
|
||||
.await
|
||||
.expect("a prefix with a trailing separator should initialize offline");
|
||||
assert_eq!(trimmed.prefix, "archive");
|
||||
|
||||
let mut params = s3_compatible_params("http://tier.example.com:9000");
|
||||
params.prefix = "archive/nested";
|
||||
let untouched = new_s3_compatible_warm_backend(params)
|
||||
.await
|
||||
.expect("a nested prefix should initialize offline");
|
||||
assert_eq!(untouched.prefix, "archive/nested");
|
||||
|
||||
let mut params = s3_compatible_params("http://tier.example.com:9000");
|
||||
params.prefix = "";
|
||||
let empty = new_s3_compatible_warm_backend(params)
|
||||
.await
|
||||
.expect("an empty prefix should initialize offline");
|
||||
assert_eq!(empty.prefix, "");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn s3_compatible_backend_honors_the_auto_bucket_lookup_family() {
|
||||
let mut params = s3_compatible_params("http://tier.example.com:9000");
|
||||
params.bucket_lookup = BucketLookupType::BucketLookupAuto;
|
||||
params.provider_tag = "minio";
|
||||
|
||||
let backend = new_s3_compatible_warm_backend(params)
|
||||
.await
|
||||
.expect("the auto-lookup provider family should initialize offline");
|
||||
|
||||
assert_eq!(backend.client.lookup, BucketLookupType::BucketLookupAuto);
|
||||
assert_eq!(backend.client.tier_type, "minio");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimal_part_size_charges_an_unknown_length_the_multipart_ceiling() {
|
||||
let unknown = optimal_part_size(-1, PROVIDER_MIN_PART_SIZE).expect("an unknown length must be accepted");
|
||||
let ceiling =
|
||||
optimal_part_size(MAX_MULTIPART_PUT_OBJECT_SIZE, PROVIDER_MIN_PART_SIZE).expect("the exact ceiling must be accepted");
|
||||
|
||||
assert_eq!(unknown, ceiling);
|
||||
assert_eq!(unknown, 5 * PROVIDER_MIN_PART_SIZE);
|
||||
assert!(unknown * MAX_PARTS_COUNT >= MAX_MULTIPART_PUT_OBJECT_SIZE);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimal_part_size_rejects_an_object_above_the_multipart_ceiling() {
|
||||
let err = optimal_part_size(MAX_MULTIPART_PUT_OBJECT_SIZE + 1, PROVIDER_MIN_PART_SIZE)
|
||||
.expect_err("an object past the multipart ceiling must fail closed");
|
||||
|
||||
assert_eq!(err.to_string(), "entity too large");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn optimal_part_size_never_returns_less_than_one_part() {
|
||||
assert_eq!(
|
||||
optimal_part_size(0, PROVIDER_MIN_PART_SIZE).expect("a zero-length object must be accepted"),
|
||||
PROVIDER_MIN_PART_SIZE
|
||||
);
|
||||
assert_eq!(
|
||||
optimal_part_size(1024, PROVIDER_MIN_PART_SIZE).expect("a tiny object must be accepted"),
|
||||
PROVIDER_MIN_PART_SIZE
|
||||
);
|
||||
assert_eq!(
|
||||
optimal_part_size(PROVIDER_MIN_PART_SIZE, PROVIDER_MIN_PART_SIZE)
|
||||
.expect("an object of exactly one part must be accepted"),
|
||||
PROVIDER_MIN_PART_SIZE
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_transition_put_options_preserves_content_headers() {
|
||||
let mut metadata = HashMap::new();
|
||||
|
||||
@@ -19,38 +19,76 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierAliyun,
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
pub struct WarmBackendAliyun(WarmBackendS3);
|
||||
|
||||
impl WarmBackendAliyun {
|
||||
pub async fn new(conf: &TierAliyun, tier: &str) -> Result<Self, std::io::Error> {
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
provider_tag: "aliyun",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "aliyun").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -63,7 +101,7 @@ impl WarmBackend for WarmBackendAliyun {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -94,29 +132,23 @@ impl WarmBackend for WarmBackendAliyun {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierAliyun;
|
||||
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierAliyun {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match WarmBackendAliyun::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -19,38 +19,76 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierAzure,
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
pub struct WarmBackendAzure(WarmBackendS3);
|
||||
|
||||
impl WarmBackendAzure {
|
||||
pub async fn new(conf: &TierAzure, tier: &str) -> Result<Self, std::io::Error> {
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
provider_tag: "azure",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "azure").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -63,7 +101,7 @@ impl WarmBackend for WarmBackendAzure {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -94,29 +132,23 @@ impl WarmBackend for WarmBackendAzure {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierAzure;
|
||||
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierAzure {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match WarmBackendAzure::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -39,7 +39,6 @@ use rustfs_s3_client::{
|
||||
api_put_object::PutObjectOptions,
|
||||
transition_api::{Options, ReadCloser, ReaderImpl},
|
||||
};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
use tracing::warn;
|
||||
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
@@ -74,12 +73,6 @@ impl WarmBackendGCS {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
if !conf.endpoint.is_empty() {
|
||||
let endpoint_url = url::Url::parse(&conf.endpoint).map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
validate_outbound_url(&endpoint_url)
|
||||
.map_err(|err| std::io::Error::other(format!("tier endpoint is not allowed: {err}")))?;
|
||||
}
|
||||
|
||||
let authorized_user = serde_json::from_str(&conf.creds)?;
|
||||
let credentials = Builder::new(authorized_user)
|
||||
//.with_retry_policy(AlwaysRetry.with_attempt_limit(3))
|
||||
@@ -218,9 +211,7 @@ impl WarmBackend for WarmBackendGCS {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::WarmBackendGCS;
|
||||
use super::parse_generation;
|
||||
use crate::services::tier::tier_config::TierGCS;
|
||||
use std::io::ErrorKind;
|
||||
|
||||
#[test]
|
||||
@@ -240,21 +231,6 @@ mod tests {
|
||||
assert_eq!(err.kind(), ErrorKind::InvalidData, "{value}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_credential_setup() {
|
||||
let conf = TierGCS {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
creds: "not-json".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match WarmBackendGCS::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed"), "unexpected error: {err}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/*fn gcs_to_object_error(err: Error, params: Vec<String>) -> Option<Error> {
|
||||
|
||||
@@ -19,38 +19,77 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierHuaweicloud,
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
pub struct WarmBackendHuaweicloud(WarmBackendS3);
|
||||
|
||||
impl WarmBackendHuaweicloud {
|
||||
pub async fn new(conf: &TierHuaweicloud, tier: &str) -> Result<Self, std::io::Error> {
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
provider_tag: "huaweicloud",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client =
|
||||
TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "huaweicloud").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -63,7 +102,7 @@ impl WarmBackend for WarmBackendHuaweicloud {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -94,29 +133,23 @@ impl WarmBackend for WarmBackendHuaweicloud {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierHuaweicloud;
|
||||
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierHuaweicloud {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match WarmBackendHuaweicloud::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -19,18 +19,23 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierMinIO,
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend::{TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
@@ -38,22 +43,51 @@ pub struct WarmBackendMinIO(WarmBackendS3);
|
||||
|
||||
impl WarmBackendMinIO {
|
||||
pub async fn new(conf: &TierMinIO, tier: &str) -> Result<Self, std::io::Error> {
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
// MinIO tier endpoints are commonly path-style, so bucket addressing stays on
|
||||
// `BucketLookupAuto`; pinning DNS here would break those deployments.
|
||||
bucket_lookup: BucketLookupType::BucketLookupAuto,
|
||||
provider_tag: "minio",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "minio").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -66,7 +100,7 @@ impl WarmBackend for WarmBackendMinIO {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -115,29 +149,23 @@ impl crate::services::tier::warm_backend::TransitionCandidateReconciler for Warm
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierMinIO;
|
||||
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2043 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierMinIO {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match WarmBackendMinIO::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -19,18 +19,23 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierR2,
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend::{TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
@@ -38,22 +43,51 @@ pub struct WarmBackendR2(WarmBackendS3);
|
||||
|
||||
impl WarmBackendR2 {
|
||||
pub async fn new(conf: &TierR2, tier: &str) -> Result<Self, std::io::Error> {
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
// R2 tier endpoints are commonly path-style, so bucket addressing stays on
|
||||
// `BucketLookupAuto`; pinning DNS here would break those deployments.
|
||||
bucket_lookup: BucketLookupType::BucketLookupAuto,
|
||||
provider_tag: "r2",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "r2").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -66,7 +100,7 @@ impl WarmBackend for WarmBackendR2 {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -115,29 +149,23 @@ impl crate::services::tier::warm_backend::TransitionCandidateReconciler for Warm
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierR2;
|
||||
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2043 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierR2 {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match WarmBackendR2::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -19,55 +19,34 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierRustFS,
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend::{TransitionCandidateProbe, WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::{OutboundUrlError, validate_outbound_url};
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
// Debug-only opt-in for single-host test/dev setups; release builds always reject loopback.
|
||||
const ALLOW_LOOPBACK_TIER_ENDPOINT_ENV: &str = "RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT";
|
||||
|
||||
fn validate_rustfs_tier_endpoint(url: &url::Url) -> Result<(), OutboundUrlError> {
|
||||
let allow_loopback = cfg!(debug_assertions)
|
||||
&& std::env::var(ALLOW_LOOPBACK_TIER_ENDPOINT_ENV)
|
||||
.map(|value| value == "1" || value.eq_ignore_ascii_case("true"))
|
||||
.unwrap_or(false);
|
||||
validate_rustfs_tier_endpoint_inner(url, allow_loopback)
|
||||
}
|
||||
|
||||
fn validate_rustfs_tier_endpoint_inner(url: &url::Url, allow_loopback: bool) -> Result<(), OutboundUrlError> {
|
||||
match validate_outbound_url(url) {
|
||||
Err(OutboundUrlError::ForbiddenHost {
|
||||
reason: "loopback address" | "loopback host",
|
||||
..
|
||||
}) if allow_loopback => Ok(()),
|
||||
result => result,
|
||||
}
|
||||
}
|
||||
|
||||
pub struct WarmBackendRustFS(WarmBackendS3);
|
||||
|
||||
impl WarmBackendRustFS {
|
||||
pub async fn new(conf: &TierRustFS, tier: &str) -> Result<Self, std::io::Error> {
|
||||
// This provider reports endpoint problems with its own wording (and keeps the
|
||||
// `url::ParseError` as the io::Error source) while the shared constructor carries the
|
||||
// MinIO-derived texts. Unifying the two is a separate change, so the endpoint is
|
||||
// pre-validated here, after the credential and bucket checks so the order in which the
|
||||
// shared constructor would report the same failures is preserved.
|
||||
if conf.access_key.is_empty() || conf.secret_key.is_empty() {
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket.is_empty() {
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
@@ -76,29 +55,36 @@ impl WarmBackendRustFS {
|
||||
Err(e) => return Err(std::io::Error::other(e)),
|
||||
};
|
||||
|
||||
if u.host_str().is_none() {
|
||||
return Err(std::io::Error::other("endpoint URL must include a host"));
|
||||
}
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("endpoint URL must include a host"))?;
|
||||
let client = TransitionClient::new(&format!("{host}:{}", u.port().unwrap_or(default_port)), opts, "rustfs").await?;
|
||||
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
// RustFS tier endpoints are path-style, so bucket addressing stays on
|
||||
// `BucketLookupAuto`; pinning DNS here would break those endpoints.
|
||||
bucket_lookup: BucketLookupType::BucketLookupAuto,
|
||||
provider_tag: "rustfs",
|
||||
// Debug-only, env-gated loopback exception for this provider's own e2e tier
|
||||
// tests (rustfs/rustfs#6773); every other provider passes plain
|
||||
// `validate_outbound_url`.
|
||||
validate_endpoint: validate_rustfs_tier_endpoint,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -111,7 +97,7 @@ impl WarmBackend for WarmBackendRustFS {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -160,6 +146,27 @@ impl crate::services::tier::warm_backend::TransitionCandidateReconciler for Warm
|
||||
}
|
||||
}
|
||||
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use futures::FutureExt;
|
||||
@@ -190,23 +197,4 @@ mod tests {
|
||||
};
|
||||
assert!(err.to_string().contains("host"), "expected host validation error, got: {err}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = rustfs_tier("https://127.0.0.1:9000");
|
||||
|
||||
match WarmBackendRustFS::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn loopback_opt_in_does_not_allow_other_private_endpoints() {
|
||||
let loopback = url::Url::parse("https://127.0.0.1:9000").unwrap();
|
||||
assert!(validate_rustfs_tier_endpoint_inner(&loopback, true).is_ok());
|
||||
|
||||
let private = url::Url::parse("https://10.0.0.1:9000").unwrap();
|
||||
assert!(validate_rustfs_tier_endpoint_inner(&private, true).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,38 +19,76 @@
|
||||
#![allow(clippy::all)]
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::services::tier::{
|
||||
tier_config::TierTencent,
|
||||
warm_backend::{
|
||||
S3CompatibleWarmBackendParams, WarmBackend, WarmBackendGetOpts, build_transition_put_options,
|
||||
new_s3_compatible_warm_backend, optimal_part_size,
|
||||
},
|
||||
warm_backend::{WarmBackend, WarmBackendGetOpts, build_transition_put_options},
|
||||
warm_backend_s3::WarmBackendS3,
|
||||
};
|
||||
use rustfs_s3_client::transition_api::{BucketLookupType, ReadCloser, ReaderImpl};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
use rustfs_s3_client::{
|
||||
admin_handler_utils::AdminError,
|
||||
api_put_object::PutObjectOptions,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
transition_api::{BucketLookupType, Options, ReadCloser, ReaderImpl, TransitionClient, TransitionCore},
|
||||
};
|
||||
use tracing::warn;
|
||||
|
||||
const MAX_MULTIPART_PUT_OBJECT_SIZE: i64 = 1024 * 1024 * 1024 * 1024 * 5;
|
||||
const MAX_PARTS_COUNT: i64 = 10000;
|
||||
const _MAX_PART_SIZE: i64 = 1024 * 1024 * 1024 * 5;
|
||||
const MIN_PART_SIZE: i64 = 1024 * 1024 * 128;
|
||||
|
||||
pub struct WarmBackendTencent(WarmBackendS3);
|
||||
|
||||
impl WarmBackendTencent {
|
||||
pub async fn new(conf: &TierTencent, tier: &str) -> Result<Self, std::io::Error> {
|
||||
Ok(Self(
|
||||
new_s3_compatible_warm_backend(S3CompatibleWarmBackendParams {
|
||||
endpoint: &conf.endpoint,
|
||||
access_key: &conf.access_key,
|
||||
secret_key: &conf.secret_key,
|
||||
bucket: &conf.bucket,
|
||||
prefix: &conf.prefix,
|
||||
region: &conf.region,
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
provider_tag: "tencent",
|
||||
validate_endpoint: validate_outbound_url,
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
if conf.access_key == "" || conf.secret_key == "" {
|
||||
return Err(std::io::Error::other("both access and secret keys are required"));
|
||||
}
|
||||
|
||||
if conf.bucket == "" {
|
||||
return Err(std::io::Error::other("no bucket name was provided"));
|
||||
}
|
||||
|
||||
let u = match url::Url::parse(&conf.endpoint) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
return Err(std::io::Error::other(e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
let creds = Credentials::new(Static(Value {
|
||||
access_key_id: conf.access_key.clone(),
|
||||
secret_access_key: conf.secret_key.clone(),
|
||||
session_token: "".to_string(),
|
||||
signer_type: SignatureType::SignatureV4,
|
||||
..Default::default()
|
||||
}));
|
||||
let opts = Options {
|
||||
creds,
|
||||
secure: u.scheme() == "https",
|
||||
trailing_headers: true,
|
||||
region: conf.region.clone(),
|
||||
bucket_lookup: BucketLookupType::BucketLookupDNS,
|
||||
..Default::default()
|
||||
};
|
||||
let scheme = u.scheme();
|
||||
let default_port = if scheme == "https" { 443 } else { 80 };
|
||||
let host = u
|
||||
.host_str()
|
||||
.ok_or_else(|| std::io::Error::other("Invalid endpoint URL: missing host"))?;
|
||||
let client = TransitionClient::new(&format!("{}:{}", host, u.port().unwrap_or(default_port)), opts, "tencent").await?;
|
||||
|
||||
let client = Arc::new(client);
|
||||
let core = TransitionCore(Arc::clone(&client));
|
||||
Ok(Self(WarmBackendS3 {
|
||||
client,
|
||||
core,
|
||||
bucket: conf.bucket.clone(),
|
||||
prefix: conf.prefix.strip_suffix("/").unwrap_or(&conf.prefix).to_owned(),
|
||||
storage_class: "".to_string(),
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -63,7 +101,7 @@ impl WarmBackend for WarmBackendTencent {
|
||||
length: i64,
|
||||
meta: HashMap<String, String>,
|
||||
) -> Result<String, std::io::Error> {
|
||||
let part_size = optimal_part_size(length, MIN_PART_SIZE)?;
|
||||
let part_size = optimal_part_size(length)?;
|
||||
let client = self.0.client.clone();
|
||||
let res = client
|
||||
.put_object(&self.0.bucket, &self.0.get_dest(object), r, length, &{
|
||||
@@ -94,29 +132,23 @@ impl WarmBackend for WarmBackendTencent {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::services::tier::tier_config::TierTencent;
|
||||
|
||||
/// The SSRF guard itself is exercised once, generically, in
|
||||
/// `warm_backend::tests` (see backlog#2040/backlog#2041 and
|
||||
/// rustfs/rustfs#6764) — this test only pins that this provider's
|
||||
/// production constructor really is wired through that shared path.
|
||||
#[tokio::test]
|
||||
async fn new_rejects_loopback_endpoint_before_network_setup() {
|
||||
let conf = TierTencent {
|
||||
endpoint: "https://127.0.0.1:9000".to_string(),
|
||||
bucket: "tier-bucket".to_string(),
|
||||
access_key: "access".to_string(),
|
||||
secret_key: "secret".to_string(),
|
||||
region: "us-east-1".to_string(),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
match WarmBackendTencent::new(&conf, "tier").await {
|
||||
Ok(_) => panic!("loopback endpoint should be rejected"),
|
||||
Err(err) => assert!(err.to_string().contains("not allowed")),
|
||||
}
|
||||
fn optimal_part_size(object_size: i64) -> Result<i64, std::io::Error> {
|
||||
let mut object_size = object_size;
|
||||
if object_size == -1 {
|
||||
object_size = MAX_MULTIPART_PUT_OBJECT_SIZE;
|
||||
}
|
||||
|
||||
if object_size > MAX_MULTIPART_PUT_OBJECT_SIZE {
|
||||
return Err(std::io::Error::other("entity too large"));
|
||||
}
|
||||
|
||||
let configured_part_size = MIN_PART_SIZE;
|
||||
let mut part_size_flt = object_size as f64 / MAX_PARTS_COUNT as f64;
|
||||
part_size_flt = (part_size_flt as f64 / configured_part_size as f64).ceil() * configured_part_size as f64;
|
||||
|
||||
let part_size = part_size_flt as i64;
|
||||
if part_size == 0 {
|
||||
return Ok(MIN_PART_SIZE);
|
||||
}
|
||||
Ok(part_size)
|
||||
}
|
||||
|
||||
@@ -3657,7 +3657,6 @@ pub(in crate::set_disk) struct RenameTailOutcome {
|
||||
pub(in crate::set_disk) struct RenameDataFenceOptions<'a> {
|
||||
write_quorum: usize,
|
||||
scanner_publication_lease_tokens: Option<&'a HashMap<String, Uuid>>,
|
||||
scanner_publication_commit_scope: Option<crate::object_api::ScannerPublicationCommitScope>,
|
||||
}
|
||||
|
||||
impl<'a> RenameDataFenceOptions<'a> {
|
||||
@@ -3668,17 +3667,8 @@ impl<'a> RenameDataFenceOptions<'a> {
|
||||
Self {
|
||||
write_quorum,
|
||||
scanner_publication_lease_tokens,
|
||||
scanner_publication_commit_scope: None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(in crate::set_disk) fn with_publication_scope(
|
||||
mut self,
|
||||
scanner_publication_commit_scope: Option<crate::object_api::ScannerPublicationCommitScope>,
|
||||
) -> Self {
|
||||
self.scanner_publication_commit_scope = scanner_publication_commit_scope;
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
|
||||
@@ -3788,37 +3778,6 @@ pub(in crate::set_disk) async fn finish_rename_tail_heal<
|
||||
}
|
||||
}
|
||||
|
||||
async fn run_scanner_publication_delete_owner<F, Fut>(
|
||||
scope: Option<crate::object_api::ScannerPublicationCommitScope>,
|
||||
operation: F,
|
||||
) -> disk::error::Result<()>
|
||||
where
|
||||
F: FnOnce() -> Fut + Send + 'static,
|
||||
Fut: Future<Output = disk::error::Result<()>> + Send + 'static,
|
||||
{
|
||||
if scope.is_none() {
|
||||
return operation().await;
|
||||
}
|
||||
if let Some(scope) = scope.as_ref() {
|
||||
scope.attach_mutation_owner();
|
||||
}
|
||||
tokio::spawn(async move {
|
||||
let result = operation().await;
|
||||
if let Some(scope) = scope.as_ref() {
|
||||
if result.is_ok() {
|
||||
let _ = scope.mark_committed();
|
||||
} else {
|
||||
// A failed quorum does not prove that no replica committed;
|
||||
// keep the permit indeterminate for supervisor reconciliation.
|
||||
let _ = scope.mark_indeterminate();
|
||||
}
|
||||
}
|
||||
result
|
||||
})
|
||||
.await
|
||||
.map_err(|_| DiskError::other("scanner publication delete owner failed"))?
|
||||
}
|
||||
|
||||
impl SetDisks {
|
||||
pub(in crate::set_disk) fn default_read_quorum(&self) -> usize {
|
||||
self.set_drive_count - self.default_parity_count
|
||||
@@ -4036,7 +3995,6 @@ impl SetDisks {
|
||||
let RenameDataFenceOptions {
|
||||
write_quorum,
|
||||
scanner_publication_lease_tokens,
|
||||
scanner_publication_commit_scope: _scanner_publication_commit_scope,
|
||||
} = fence_options;
|
||||
if let Some(file_info) = disks
|
||||
.iter()
|
||||
@@ -4394,7 +4352,6 @@ impl SetDisks {
|
||||
let RenameDataFenceOptions {
|
||||
write_quorum,
|
||||
scanner_publication_lease_tokens,
|
||||
scanner_publication_commit_scope,
|
||||
} = fence_options;
|
||||
if let Some(file_info) = disks
|
||||
.iter()
|
||||
@@ -4426,15 +4383,11 @@ impl SetDisks {
|
||||
let fanout_src_object = src_object.clone();
|
||||
let fanout_dst_bucket = dst_bucket.clone();
|
||||
let fanout_dst_object = dst_object.clone();
|
||||
let fanout_publication_scope = scanner_publication_commit_scope.clone();
|
||||
// Keep one coordinator task so a cancelled caller cannot drop partially
|
||||
// completed disk mutations. Per-disk futures stay ordered in `join_all`,
|
||||
// preserving slot-indexed quorum and convergence accounting without a
|
||||
// scheduler task for every disk.
|
||||
let fanout = tokio::spawn(async move {
|
||||
// Keep the storage-owned movement permit attached to the actual
|
||||
// fan-out owner, even if the caller future is cancelled.
|
||||
let _fanout_publication_scope = fanout_publication_scope;
|
||||
let successful_rename_completion_rank =
|
||||
rustfs_io_metrics::put_stage_metrics_enabled().then(|| Arc::new(AtomicUsize::new(0)));
|
||||
let futures = fanout_disks
|
||||
@@ -4448,7 +4401,6 @@ impl SetDisks {
|
||||
let dst_object = fanout_dst_object.clone();
|
||||
let dst_bucket = fanout_dst_bucket.clone();
|
||||
let successful_rename_completion_rank = successful_rename_completion_rank.clone();
|
||||
let publication_scope = scanner_publication_commit_scope.clone();
|
||||
|
||||
std::panic::AssertUnwindSafe(async move {
|
||||
// Test-only introspection guard: counts this operation as
|
||||
@@ -4481,13 +4433,6 @@ impl SetDisks {
|
||||
return Err(err);
|
||||
}
|
||||
|
||||
if let Some(scope) = publication_scope.as_ref()
|
||||
&& !scope.can_commit()
|
||||
{
|
||||
let _ = scope.mark_indeterminate();
|
||||
return Err(DiskError::other("scanner publication commit scope deadline or cancellation reached"));
|
||||
}
|
||||
|
||||
let disk_wait_started = rustfs_io_metrics::put_stage_timer();
|
||||
let result = disk
|
||||
.rename_data_borrowed_with_fence(
|
||||
@@ -5751,8 +5696,6 @@ impl SetDisks {
|
||||
{
|
||||
let grace = dangling_delete_grace();
|
||||
if !grace.is_zero() && OffsetDateTime::now_utc() - mod_time < grace {
|
||||
let elapsed = OffsetDateTime::now_utc() - mod_time;
|
||||
let retry_after_secs = grace.saturating_sub(elapsed).whole_seconds().max(0);
|
||||
info!(
|
||||
bucket = bucket,
|
||||
object = object,
|
||||
@@ -5760,7 +5703,7 @@ impl SetDisks {
|
||||
grace_secs = grace.whole_seconds(),
|
||||
"skipping dangling-object deletion within grace window"
|
||||
);
|
||||
return Err(DiskError::dangling_delete_grace(retry_after_secs, grace.whole_seconds()));
|
||||
return Err(DiskError::ErasureReadQuorum);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5896,8 +5839,7 @@ impl SetDisks {
|
||||
|
||||
#[cfg(test)]
|
||||
pub(in crate::set_disk) async fn delete_prefix(&self, bucket: &str, prefix: &str) -> disk::error::Result<()> {
|
||||
self.delete_prefix_with_scanner_publication_lease(bucket, prefix, None, None)
|
||||
.await
|
||||
self.delete_prefix_with_scanner_publication_lease(bucket, prefix, None).await
|
||||
}
|
||||
|
||||
/// Delete a prefix with an optional per-remote-disk scanner publication
|
||||
@@ -5908,7 +5850,6 @@ impl SetDisks {
|
||||
bucket: &str,
|
||||
prefix: &str,
|
||||
scanner_publication_lease_tokens: Option<&HashMap<String, Uuid>>,
|
||||
scanner_publication_commit_scope: Option<crate::object_api::ScannerPublicationCommitScope>,
|
||||
) -> disk::error::Result<()> {
|
||||
let disks = self.get_disks_internal().await;
|
||||
let write_quorum = disks.len() / 2 + 1;
|
||||
@@ -5917,21 +5858,11 @@ impl SetDisks {
|
||||
let mut futures = Vec::with_capacity(disks.len());
|
||||
|
||||
for (disk_op, scanner_publication_lease_token) in disks.iter().zip(fanout_fence_tokens) {
|
||||
let disk_op = disk_op.clone();
|
||||
let bucket = bucket.to_string();
|
||||
let prefix = prefix.to_string();
|
||||
let scanner_publication_commit_scope = scanner_publication_commit_scope.clone();
|
||||
futures.push(async move {
|
||||
if let Some(disk) = disk_op {
|
||||
if let Some(scope) = scanner_publication_commit_scope.as_ref()
|
||||
&& !scope.can_commit()
|
||||
{
|
||||
return Err(DiskError::other("scanner publication delete scope cannot commit"));
|
||||
}
|
||||
let external_guard = scanner_publication_commit_scope
|
||||
.as_ref()
|
||||
.map(|scope| Arc::new(scope.clone()) as Arc<dyn Send + Sync>);
|
||||
disk.delete_with_scanner_publication_lease_and_guard(
|
||||
disk.delete_with_scanner_publication_lease(
|
||||
&bucket,
|
||||
&prefix,
|
||||
DeleteOptions {
|
||||
@@ -5940,7 +5871,6 @@ impl SetDisks {
|
||||
..Default::default()
|
||||
},
|
||||
scanner_publication_lease_token,
|
||||
external_guard,
|
||||
)
|
||||
.await
|
||||
} else {
|
||||
@@ -5949,10 +5879,7 @@ impl SetDisks {
|
||||
});
|
||||
}
|
||||
|
||||
run_scanner_publication_delete_owner(scanner_publication_commit_scope, move || async move {
|
||||
Self::reduce_delete_prefix_results(join_all(futures).await, write_quorum)
|
||||
})
|
||||
.await
|
||||
Self::reduce_delete_prefix_results(join_all(futures).await, write_quorum)
|
||||
}
|
||||
|
||||
/// Scan a single disk's copy of `prefix` and decide whether it is an orphan
|
||||
@@ -6872,7 +6799,6 @@ pub(in crate::set_disk) mod rename_fanout_barrier {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use crate::disk::error::HEAL_DANGLING_DELETE_GRACE_MESSAGE;
|
||||
use crate::disk::local::{DurabilityMode, durability_mode_override};
|
||||
|
||||
use super::*;
|
||||
@@ -6880,63 +6806,6 @@ mod tests {
|
||||
use tempfile::TempDir;
|
||||
use tokio::io::AsyncReadExt;
|
||||
|
||||
#[tokio::test]
|
||||
async fn scanner_delete_owner_survives_waiter_cancellation() {
|
||||
let movement_gate = Arc::new(tokio::sync::RwLock::new(()));
|
||||
let movement_permit = movement_gate.clone().read_owned().await;
|
||||
let scope = crate::object_api::ScannerPublicationCommitScope::new_storage_owned(
|
||||
7,
|
||||
tokio::time::Instant::now() + std::time::Duration::from_secs(30),
|
||||
Vec::new(),
|
||||
movement_permit,
|
||||
);
|
||||
scope.try_begin().expect("delete scope should enter flight");
|
||||
let scope_guard = crate::object_api::ScannerPublicationCommitScopeGuard::new(scope.clone());
|
||||
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
|
||||
let (release_tx, release_rx) = tokio::sync::oneshot::channel();
|
||||
let (finished_tx, finished_rx) = tokio::sync::oneshot::channel();
|
||||
|
||||
let waiter = tokio::spawn(run_scanner_publication_delete_owner(Some(scope.clone()), move || async move {
|
||||
started_tx.send(()).expect("delete owner should start");
|
||||
release_rx.await.expect("delete owner should be released");
|
||||
finished_tx.send(()).expect("delete owner should finish");
|
||||
Ok(())
|
||||
}));
|
||||
started_rx.await.expect("delete owner should run");
|
||||
drop(scope_guard);
|
||||
waiter.abort();
|
||||
assert_eq!(
|
||||
scope.state(),
|
||||
crate::object_api::ScannerPublicationCommitState::InFlight,
|
||||
"caller cancellation must not classify an owned delete as indeterminate"
|
||||
);
|
||||
|
||||
let mut movement_writer = Box::pin(movement_gate.write_owned());
|
||||
assert!(
|
||||
tokio::time::timeout(std::time::Duration::from_millis(20), &mut movement_writer)
|
||||
.await
|
||||
.is_err(),
|
||||
"movement transition must remain fenced while delete owner drains"
|
||||
);
|
||||
release_tx.send(()).expect("delete owner should remain alive");
|
||||
finished_rx.await.expect("delete owner should drain");
|
||||
tokio::time::timeout(std::time::Duration::from_secs(1), async {
|
||||
loop {
|
||||
if scope.state() == crate::object_api::ScannerPublicationCommitState::Committed {
|
||||
break;
|
||||
}
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("delete owner should report a terminal result");
|
||||
assert!(
|
||||
scope.release_movement_permit().await,
|
||||
"terminal delete should release its movement permit"
|
||||
);
|
||||
movement_writer.await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn write_precondition_lookup_errors_fail_closed_unless_absence_is_known() {
|
||||
let create_only = HTTPPreconditions {
|
||||
@@ -10877,13 +10746,7 @@ mod tests {
|
||||
let object = "object";
|
||||
let (_dir, disk) = read_multiple_test_disk(bucket, &[]).await;
|
||||
let set = io_primitives_test_set(vec![Some(disk.clone()), None, None], 1).await;
|
||||
let mut fi = FileInfo::new(object, 2, 1);
|
||||
fi.volume = bucket.to_string();
|
||||
fi.name = object.to_string();
|
||||
fi.size = 1;
|
||||
fi.erasure.index = 1;
|
||||
fi.metadata.insert("etag".to_string(), "etag-1".to_string());
|
||||
fi.add_object_part(1, "part-etag-1".to_string(), 1, None, 1, None, None);
|
||||
let mut fi = metadata_test_fileinfo(object);
|
||||
fi.mod_time = Some(OffsetDateTime::now_utc());
|
||||
disk.write_metadata(bucket, bucket, object, fi.clone())
|
||||
.await
|
||||
@@ -10901,15 +10764,7 @@ mod tests {
|
||||
.await
|
||||
.expect_err("recent dangling metadata must stay protected by grace");
|
||||
|
||||
let message = err.to_string();
|
||||
assert!(
|
||||
message.contains(HEAL_DANGLING_DELETE_GRACE_MESSAGE),
|
||||
"grace-protected dangling cleanup must explain the deferred delete: {message}"
|
||||
);
|
||||
assert!(
|
||||
message.contains("retry_after_secs="),
|
||||
"grace-protected dangling cleanup must include retry timing: {message}"
|
||||
);
|
||||
assert_eq!(err, DiskError::ErasureReadQuorum);
|
||||
disk.read_all(bucket, &path_join_buf(&[object, STORAGE_FORMAT_FILE]))
|
||||
.await
|
||||
.expect("metadata should remain during dangling grace");
|
||||
|
||||
@@ -575,12 +575,18 @@ impl SetDisks {
|
||||
meta.metadata.keys().any(|name| http::is_object_encryption_marker(name))
|
||||
}
|
||||
|
||||
fn starts_with_ignore_ascii_case(value: &str, prefix: &str) -> bool {
|
||||
value
|
||||
.get(..prefix.len())
|
||||
.is_some_and(|value_prefix| value_prefix.eq_ignore_ascii_case(prefix))
|
||||
}
|
||||
|
||||
fn internal_metadata_suffix(name: &str) -> Option<&str> {
|
||||
name.get(http::RUSTFS_INTERNAL_PREFIX.len()..)
|
||||
.filter(|_| http::starts_with_ignore_ascii_case(name, http::RUSTFS_INTERNAL_PREFIX))
|
||||
.filter(|_| Self::starts_with_ignore_ascii_case(name, http::RUSTFS_INTERNAL_PREFIX))
|
||||
.or_else(|| {
|
||||
name.get(http::MINIO_INTERNAL_PREFIX.len()..)
|
||||
.filter(|_| http::starts_with_ignore_ascii_case(name, http::MINIO_INTERNAL_PREFIX))
|
||||
.filter(|_| Self::starts_with_ignore_ascii_case(name, http::MINIO_INTERNAL_PREFIX))
|
||||
})
|
||||
}
|
||||
|
||||
@@ -598,9 +604,9 @@ impl SetDisks {
|
||||
|| suffix.eq_ignore_ascii_case(http::SUFFIX_REPLICATION_STATUS)
|
||||
|| suffix.eq_ignore_ascii_case(http::SUFFIX_REPLICATION_TIMESTAMP)
|
||||
|| suffix.eq_ignore_ascii_case(http::SUFFIX_PURGESTATUS)
|
||||
|| http::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_RESET_ARN_PREFIX)
|
||||
|| Self::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_RESET_ARN_PREFIX)
|
||||
// Raw compatibility keys are normalized and hashed separately below.
|
||||
|| http::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)
|
||||
|| Self::starts_with_ignore_ascii_case(suffix, http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX)
|
||||
}
|
||||
|
||||
fn update_hash_quorum_metadata_map(hasher: &mut Sha256, entries: &HashMap<String, String>) {
|
||||
@@ -1584,41 +1590,6 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// Guards the switch to `rustfs_utils::http::starts_with_ignore_ascii_case`:
|
||||
/// internal prefixes must keep matching case-insensitively, and keys shorter
|
||||
/// than the prefix must keep being rejected. Misclassifying either way leaks
|
||||
/// internal metadata into the quorum hash (or drops it out of it).
|
||||
#[test]
|
||||
fn internal_metadata_suffix_is_prefix_case_insensitive_and_rejects_short_keys() {
|
||||
assert_eq!(
|
||||
SetDisks::internal_metadata_suffix("X-RustFS-Internal-Replica-Status"),
|
||||
Some("Replica-Status"),
|
||||
"mixed-case RustFS prefix must match and preserve the suffix casing"
|
||||
);
|
||||
assert_eq!(
|
||||
SetDisks::internal_metadata_suffix("X-MINIO-INTERNAL-replica-status"),
|
||||
Some("replica-status"),
|
||||
"mixed-case MinIO prefix must match"
|
||||
);
|
||||
assert_eq!(SetDisks::internal_metadata_suffix(http::RUSTFS_INTERNAL_PREFIX), Some(""));
|
||||
|
||||
// Keys shorter than either prefix, and non-internal keys, stay unmatched.
|
||||
assert_eq!(SetDisks::internal_metadata_suffix(""), None);
|
||||
assert_eq!(SetDisks::internal_metadata_suffix("x-rustfs-interna"), None);
|
||||
assert_eq!(SetDisks::internal_metadata_suffix("x-minio-interna"), None);
|
||||
assert_eq!(SetDisks::internal_metadata_suffix("x-amz-meta-custom"), None);
|
||||
|
||||
// The suffix-prefix comparisons behind the classifier follow the same rules.
|
||||
assert!(SetDisks::is_replication_quorum_metadata_key(
|
||||
"X-RustFS-Internal-Replication-Reset-arn:rustfs:replication::target:bucket"
|
||||
));
|
||||
assert!(SetDisks::is_replication_quorum_metadata_key(
|
||||
"X-Minio-Internal-Replication-Delete-Marker-Version-arn:rustfs:replication::target:bucket"
|
||||
));
|
||||
assert!(!SetDisks::is_replication_quorum_metadata_key("x-rustfs-interna"));
|
||||
assert!(!SetDisks::is_replication_quorum_metadata_key("x-rustfs-internal-replication-res"));
|
||||
}
|
||||
|
||||
/// rustfs#5801: parity counts outside [0, total_shards] come from corrupt
|
||||
/// or foreign metadata and must be treated as invalid entries instead of
|
||||
/// clamped values that poison `common_parity`'s occurrence counting.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user