mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-29 15:29:35 +00:00
Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 唐小鸭
|
eda40d5ede | ||
|
Chris
|
a33dd896e2 | ||
|
AL
|
6796382cee | ||
|
overtrue
|
99a2af8687 |
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: rustfs-release-publish
|
||||
description: "Run the RustFS console gate, source version bump, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
|
||||
description: "Run the RustFS console gate, source version bump, release-branch CI, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
|
||||
---
|
||||
# RustFS Release Publish (preview-validated pipeline)
|
||||
|
||||
@@ -19,7 +19,9 @@ Pipeline shape:
|
||||
check console main against its latest Release
|
||||
-> if ahead: publish console -> wait for Release asset + latest API
|
||||
-> bump Cargo.toml and Cargo.lock to <target> -> merge
|
||||
-> tag <preview-tag> at that commit -> CI green
|
||||
-> cut release/<target> from a selected main commit containing the bump
|
||||
-> CI green for that exact release-branch commit
|
||||
-> tag <preview-tag> at that commit
|
||||
-> verify preview Release assets -> run binary locally + console checks
|
||||
-> validate with latest rc client
|
||||
-> report preview acceptance results -> STOP for explicit human confirmation
|
||||
@@ -29,7 +31,7 @@ check console main against its latest Release
|
||||
-> update installation references on main -> update rustfs.com announcement
|
||||
```
|
||||
|
||||
On preview validation failure: fix lands on main via normal PR (Cargo versions are already at `<target>`, no new source bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2. Installation references remain on the previous published deliverable throughout preview validation.
|
||||
On candidate or preview validation failure: fix lands on main via normal PR, then backport only the needed fix to `release/<target>` through a reviewed PR. Re-run release-branch CI at the new commit and, if a preview was already tagged, tag `<preview-tag N+1>` there and restart acceptance. Installation references remain on the previous published deliverable throughout preview validation.
|
||||
|
||||
## Required inputs
|
||||
|
||||
@@ -68,14 +70,14 @@ Rules:
|
||||
- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
|
||||
- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
|
||||
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
|
||||
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
|
||||
- The release branch is the candidate source during multi-day acceptance. Do not merge main wholesale into it, force-push it, or advance it with unrelated changes. The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD or an advanced release-branch HEAD, and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
|
||||
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
|
||||
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
|
||||
- Phases run in order; a failure blocks dependent steps. For a preview/source acceptance failure before the final tag exists, land the fix, verify Cargo still matches the confirmed target, and restart from Phase 2 with the next preview at the fixed commit. If main has advanced to another target, resolve that source/version mismatch before another preview.
|
||||
- Phases run in order; a failure blocks dependent steps. For a candidate or preview/source acceptance failure before the final tag exists, land the fix on main, backport it to the release branch, verify Cargo there still matches the confirmed target, and restart from release-branch CI in Phase 2. Use the next preview iteration if any preview was already tagged. Main may advance to another target without changing this release candidate.
|
||||
- Once the final tag exists, preserve its commit. Retry failed publication jobs for that exact tag; do not recreate the tag or restart preview on newer main. If a source change is required after final-tag publication, report the failure and obtain a new target version. Phase 7 failures resume only the failed follow-up after rechecking artifact availability and whether a newer release has superseded it.
|
||||
- Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
|
||||
- Confirmation is scoped to the reported `<target>`, `<preview-tag>`, and `PREVIEW_HASH`. A failed or repeated acceptance cycle, including any new preview iteration, invalidates prior confirmation and requires a new one.
|
||||
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
|
||||
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision. Do not delete the release branch or preview tags as part of abandonment without an explicit decision.
|
||||
- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
|
||||
|
||||
## Phase 0 — Preflight
|
||||
@@ -83,6 +85,7 @@ Rules:
|
||||
- `git status --short` clean; `git fetch origin main --tags`.
|
||||
- `gh auth status` works; confirm you can view `gh release list -L 3`.
|
||||
- Confirm the exact final target version with the user if not explicit.
|
||||
- Check whether `release/<target>` already exists before creating it; a resumed release reuses the existing branch and its history. For an in-flight release that already has a preview tag but no branch, anchor the new branch to that tag's commit only after verifying its target versions and prior acceptance evidence.
|
||||
|
||||
### Console release gate
|
||||
|
||||
@@ -90,18 +93,19 @@ Read and complete [the Console gate](references/console-gate.md) before Phase 1.
|
||||
|
||||
## Phase 1 — Source version bump to the final target (once)
|
||||
|
||||
- If Cargo.toml and all workspace members in Cargo.lock already read `<target>` (e.g. this is a restart after a failed preview), verify both and skip the source bump. Installation versions intentionally differ during preview; do not treat them as leftovers. If a previous preparation advanced installation references to an unavailable target, restore those references to the verified published baseline before selecting the next preview commit.
|
||||
- Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
|
||||
- Get the PR merged into main. Record the resulting main commit:
|
||||
- If `release/<target>` exists, verify its Cargo.toml and workspace members in Cargo.lock read `<target>` and skip the source bump, even if main has advanced. If the branch versions differ, stop and resolve the mismatch without downgrading main.
|
||||
- If no branch exists but an unfinished preview tag does, verify its commit has `<target>` in both version files and skip the bump; Phase 2 anchors the branch to that commit. Stop if those files disagree. Otherwise, if main already has `<target>` in both files, skip the bump. Installation references intentionally retain the previous published deliverable during preview; if they were advanced to an unavailable target, restore that baseline before choosing the next candidate.
|
||||
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
|
||||
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
|
||||
|
||||
```bash
|
||||
git fetch origin main
|
||||
PREVIEW_HASH=$(git rev-parse origin/main) # must contain the bump PR
|
||||
```
|
||||
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
|
||||
|
||||
`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
|
||||
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
|
||||
|
||||
## Phase 2 — Publish the preview tag
|
||||
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
|
||||
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
|
||||
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
|
||||
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
|
||||
|
||||
```bash
|
||||
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
|
||||
@@ -112,11 +116,12 @@ Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `d
|
||||
|
||||
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
|
||||
|
||||
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
|
||||
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
|
||||
|
||||
## Phase 3 — CI and preview Release verification
|
||||
|
||||
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
|
||||
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
|
||||
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
|
||||
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
|
||||
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
|
||||
@@ -128,13 +133,13 @@ Read and complete [preview acceptance](references/preview-acceptance.md): verify
|
||||
|
||||
### Manual confirmation gate
|
||||
|
||||
After every Phase 3–5 check passes, report the target, preview tag, `PREVIEW_HASH`, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.
|
||||
After every Phase 3–5 check passes, verify the release branch still points to `PREVIEW_HASH`. Report the target, release branch, preview tag, `PREVIEW_HASH`, release-branch CI run, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.
|
||||
|
||||
Continue to Phase 6 only after a new user reply explicitly confirms the reported target, preview tag, and commit. A clear affirmative reply to that exact report, such as `确认继续`, is sufficient; if the reply is ambiguous or any reported value changed, ask again.
|
||||
|
||||
## Phase 6 — Publish the final tag on the validated commit
|
||||
|
||||
No second source version bump. The final tag goes on the exact commit the preview validated:
|
||||
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH`. If it moved after confirmation, the approval no longer applies; validate and obtain confirmation for a new preview. The final tag goes on the exact commit the preview validated:
|
||||
|
||||
```bash
|
||||
git fetch origin --tags
|
||||
@@ -158,7 +163,7 @@ Only after Phase 6 succeeds, complete [post-release updates](references/post-rel
|
||||
Always report:
|
||||
|
||||
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
|
||||
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
|
||||
- Target version, release branch, release-branch CI run and validated SHA, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
|
||||
- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
|
||||
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
|
||||
- Post-release installation PR and verification results; website banner target, text, link, PR, and observed deployment state. Report any remaining merge authorization or failed deployment explicitly rather than claiming the banner is live.
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
sha256-darwin=3b6ff112ff16acbaa4148c4736095a1b6b71cc11af0dd3513ff28c6bdc334ca8
|
||||
sha256-linux=4e8bfc01870565a0a608a1634e6919fc593ce120a970aa04d974f7edf2675c21
|
||||
sha256-linux=80d0fae627007c31f250fb1fb21c8a952861195d53d7f60b6e71636d5ca65b46
|
||||
|
||||
@@ -542,10 +542,11 @@ fail-fast = false
|
||||
[profile.e2e-smoke.junit]
|
||||
path = "junit.xml"
|
||||
|
||||
# These storage-heavy fixtures reserve capacity. Pagination tests bound listing
|
||||
# requests separately, after their seed objects have been written.
|
||||
# Bound the whole pagination fixture, including startup, seeding, and cleanup;
|
||||
# its LIST deadline begins only after the seed objects have been written.
|
||||
[[profile.e2e-smoke.overrides]]
|
||||
filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
|
||||
slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
|
||||
threads-required = "num-test-threads"
|
||||
|
||||
[[profile.e2e-smoke.overrides]]
|
||||
|
||||
@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Security
|
||||
|
||||
- **Per-write keys for encrypted multipart parts**: parts of an SSE-C, SSE-S3 or SSE-KMS multipart upload share the upload's data key and base nonce, so uploading the same part number again within one upload reused that part's AES-GCM nonces. A new part-keyed segment layout opens each part write with a random salt (frame type `0x03`) and seals the part's v2 frames under a key derived from the salt, the part number and the data key. Its write switch `RUSTFS_ENCRYPTION_MULTIPART_PART_KEY` is **off by default** for rolling-upgrade safety: nodes without part-keyed read support cannot decrypt these parts, and ciphertext travels verbatim through transition, decommission and SSE-C replication passthrough, so enable it only after every node — and every RustFS warm/replication target that receives raw ciphertext — runs this release. Reading part-keyed parts needs no switch, existing parts stay readable, and one object may mix both layouts. ETags, S3 error codes and the API are unchanged.
|
||||
|
||||
- **Header-signed SigV4 requests honour only signed headers** (GHSA-xm99-m3gq-83g8): a request authenticated with a SigV4 `Authorization` header that carries an `x-amz-*` request header not listed in its `SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3 and the presigned rule from GHSA-g8w9-qw9q-fghr. Previously anyone holding one header-signed `PutObject` request could add an unsigned `x-amz-copy-source` and turn it into a `CopyObject` that ran with the signer's permissions, copying any object the signer could read into the target. An `Authorization` header whose algorithm token is not `AWS4-HMAC-SHA256` is now rejected instead of being verified as SigV4. The request-envelope headers `x-amz-content-sha256`, `x-amz-decoded-content-length`, `x-amz-trailer` and `x-amz-checksum-algorithm` (the same set the upstream `s3s` fix exempts) and `x-amz-cf-id` (CloudFront) remain tolerated unsigned; AWS SDKs and RustFS's own signers already sign every other `x-amz-*` header. SigV2, JWT and anonymous requests are unchanged.
|
||||
|
||||
### Replication
|
||||
|
||||
Generated
+1
@@ -10609,6 +10609,7 @@ dependencies = [
|
||||
"faster-hex",
|
||||
"futures",
|
||||
"hex-simd",
|
||||
"hmac 0.13.0",
|
||||
"hotpath",
|
||||
"http 1.5.0",
|
||||
"http-body-util",
|
||||
|
||||
@@ -346,6 +346,42 @@ pub(crate) fn encryption_frame_v2_enabled() -> bool {
|
||||
}
|
||||
}
|
||||
|
||||
/// Write-side switch for part-keyed multipart segments.
|
||||
///
|
||||
/// Every part of a Direct-mode multipart upload shares one data key and base
|
||||
/// nonce, so the legacy and v2 part layouts derive the same nonce sequence
|
||||
/// each time a part number is written. With this switch on, each part write
|
||||
/// opens with a random salt and seals its frames under a key derived from it,
|
||||
/// so rewriting a part number never reuses a (key, nonce) pair.
|
||||
///
|
||||
/// Off by default for rolling-upgrade safety, like
|
||||
/// [`ENV_RUSTFS_ENCRYPTION_FRAME_V2`]: nodes without part-keyed read support
|
||||
/// reject these segments, and encrypted ciphertext travels verbatim through
|
||||
/// transition, decommission and SSE-C replication passthrough. Turn it on only
|
||||
/// after every node (and every RustFS warm/replication target that receives
|
||||
/// raw ciphertext) runs a release that reads part-keyed segments. Reading them
|
||||
/// needs no switch — the decrypt reader dispatches on the frame type byte.
|
||||
// RUSTFS_COMPAT_TODO(multipart-part-key-default-off-window): staged rollout switch for part-keyed multipart encryption, flipping the default to enabled on retirement. Remove after the minimum supported direct-upgrade release reads part-keyed segments.
|
||||
#[cfg(not(feature = "rio-v2"))]
|
||||
pub(crate) const ENV_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY: &str = "RUSTFS_ENCRYPTION_MULTIPART_PART_KEY";
|
||||
#[cfg(not(feature = "rio-v2"))]
|
||||
pub(crate) const DEFAULT_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY: bool = false;
|
||||
|
||||
#[cfg(not(feature = "rio-v2"))]
|
||||
pub(crate) fn encryption_multipart_part_key_enabled() -> bool {
|
||||
#[cfg(test)]
|
||||
{
|
||||
rustfs_utils::get_env_bool(ENV_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY, DEFAULT_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY)
|
||||
}
|
||||
#[cfg(not(test))]
|
||||
{
|
||||
static CACHED: std::sync::OnceLock<bool> = std::sync::OnceLock::new();
|
||||
*CACHED.get_or_init(|| {
|
||||
rustfs_utils::get_env_bool(ENV_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY, DEFAULT_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
enum WriteEncryptionMode {
|
||||
SinglepartObjectKey,
|
||||
@@ -470,7 +506,9 @@ impl WritePlan {
|
||||
multipart_part_number,
|
||||
} => {
|
||||
#[cfg(not(feature = "rio-v2"))]
|
||||
let encrypt_reader = if encryption_frame_v2_enabled() {
|
||||
let encrypt_reader = if encryption_multipart_part_key_enabled() {
|
||||
EncryptReader::new_multipart_part_keyed(reader, encryption.key_bytes, base_nonce, multipart_part_number)
|
||||
} else if encryption_frame_v2_enabled() {
|
||||
EncryptReader::new_multipart_v2(reader, encryption.key_bytes, base_nonce, multipart_part_number)
|
||||
} else {
|
||||
EncryptReader::new_multipart(reader, encryption.key_bytes, base_nonce, multipart_part_number)
|
||||
@@ -646,6 +684,118 @@ mod tests {
|
||||
assert_eq!(actual, plaintext);
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "rio-v2"))]
|
||||
async fn write_multipart_part(
|
||||
plaintext: &[u8],
|
||||
key_bytes: [u8; 32],
|
||||
base_nonce: [u8; 12],
|
||||
part_number: usize,
|
||||
) -> (Vec<u8>, String) {
|
||||
let actual_size = plaintext.len() as i64;
|
||||
let reader = HashReader::from_stream(Cursor::new(plaintext.to_vec()), actual_size, actual_size, None, None, false)
|
||||
.expect("create hash reader");
|
||||
let mut transformed = WritePlan::new()
|
||||
.with_encryption(WriteEncryption::multipart(key_bytes, base_nonce, part_number))
|
||||
.apply(reader, actual_size)
|
||||
.expect("apply multipart encryption plan");
|
||||
let mut ciphertext = Vec::new();
|
||||
transformed
|
||||
.read_to_end(&mut ciphertext)
|
||||
.await
|
||||
.expect("read transformed ciphertext");
|
||||
let etag = transformed.try_resolve_etag().expect("part ETag resolves");
|
||||
(ciphertext, etag)
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "rio-v2"))]
|
||||
#[tokio::test]
|
||||
async fn write_plan_part_key_switch_gives_each_part_write_its_own_key() {
|
||||
let key_bytes = [0x5Au8; 32];
|
||||
let base_nonce = [0xA5u8; 12];
|
||||
let part_number = 4;
|
||||
let first = b"first-part-write-".repeat(1024);
|
||||
let second = b"other-part-write-".repeat(1024);
|
||||
|
||||
// Default: the rolling-upgrade-safe legacy layout, which repeats the
|
||||
// part's nonce sequence on every write of the same part number.
|
||||
let (legacy_one, legacy_etag) = temp_env::async_with_vars(
|
||||
[(ENV_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY, None::<&str>)],
|
||||
write_multipart_part(&first, key_bytes, base_nonce, part_number),
|
||||
)
|
||||
.await;
|
||||
let (legacy_retry, _) = temp_env::async_with_vars(
|
||||
[(ENV_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY, None::<&str>)],
|
||||
write_multipart_part(&first, key_bytes, base_nonce, part_number),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(legacy_one[0], 0x00, "the default writer keeps the legacy v1 frame layout");
|
||||
assert_eq!(legacy_one, legacy_retry);
|
||||
|
||||
let (keyed_one, keyed_etag) = temp_env::async_with_vars(
|
||||
[(ENV_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY, Some("true"))],
|
||||
write_multipart_part(&first, key_bytes, base_nonce, part_number),
|
||||
)
|
||||
.await;
|
||||
let (keyed_retry, keyed_retry_etag) = temp_env::async_with_vars(
|
||||
[(ENV_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY, Some("true"))],
|
||||
write_multipart_part(&first, key_bytes, base_nonce, part_number),
|
||||
)
|
||||
.await;
|
||||
let (keyed_other, _) = temp_env::async_with_vars(
|
||||
[(ENV_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY, Some("true"))],
|
||||
write_multipart_part(&second, key_bytes, base_nonce, part_number),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(keyed_one[0], 0x03, "a part-keyed segment opens with its key frame");
|
||||
assert_ne!(keyed_one[8..40], keyed_retry[8..40], "every part write draws a fresh salt");
|
||||
assert_ne!(keyed_one[8..40], keyed_other[8..40]);
|
||||
assert_ne!(keyed_one, keyed_retry);
|
||||
// The part ETag stays the plaintext MD5: unchanged by the layout.
|
||||
assert_eq!(keyed_etag, legacy_etag);
|
||||
assert_eq!(keyed_retry_etag, legacy_etag);
|
||||
|
||||
// Legacy and part-keyed parts read back through one multipart reader.
|
||||
let (legacy_two, _) = write_multipart_part(&second, key_bytes, base_nonce, part_number + 1).await;
|
||||
let stream = [keyed_one.as_slice(), legacy_two.as_slice()].concat();
|
||||
let mut actual = Vec::new();
|
||||
DecryptReader::new_multipart(Cursor::new(stream), key_bytes, base_nonce, vec![part_number, part_number + 1])
|
||||
.read_to_end(&mut actual)
|
||||
.await
|
||||
.expect("decrypt mixed part layouts");
|
||||
assert_eq!(actual, [first.as_slice(), second.as_slice()].concat());
|
||||
|
||||
// Compressed multipart parts are encrypted after compression; the
|
||||
// part-keyed layout wraps that stream the same way.
|
||||
let compressed_ciphertext =
|
||||
temp_env::async_with_vars([(ENV_RUSTFS_ENCRYPTION_MULTIPART_PART_KEY, Some("true"))], async {
|
||||
let actual_size = first.len() as i64;
|
||||
let reader = HashReader::from_stream(Cursor::new(first.clone()), actual_size, actual_size, None, None, false)
|
||||
.expect("create hash reader");
|
||||
let mut transformed = WritePlan::new()
|
||||
.with_compression(CompressionAlgorithm::default())
|
||||
.with_encryption(WriteEncryption::multipart(key_bytes, base_nonce, part_number))
|
||||
.apply(reader, actual_size)
|
||||
.expect("apply compression and part-keyed encryption plan");
|
||||
let mut ciphertext = Vec::new();
|
||||
transformed
|
||||
.read_to_end(&mut ciphertext)
|
||||
.await
|
||||
.expect("read transformed ciphertext");
|
||||
ciphertext
|
||||
})
|
||||
.await;
|
||||
assert_eq!(compressed_ciphertext[0], 0x03);
|
||||
let decrypt_reader =
|
||||
DecryptReader::new_multipart(Cursor::new(compressed_ciphertext), key_bytes, base_nonce, vec![part_number]);
|
||||
let mut decompressed = DecompressReader::new(Box::new(decrypt_reader), CompressionAlgorithm::default());
|
||||
let mut actual = Vec::new();
|
||||
decompressed
|
||||
.read_to_end(&mut actual)
|
||||
.await
|
||||
.expect("decrypt and decompress part-keyed stream");
|
||||
assert_eq!(actual, first);
|
||||
}
|
||||
|
||||
#[cfg(feature = "rio-v2")]
|
||||
#[tokio::test]
|
||||
async fn write_plan_supports_singlepart_object_key_encryption_roundtrip() {
|
||||
|
||||
@@ -3078,26 +3078,58 @@ mod tests {
|
||||
object: &str,
|
||||
key_bytes: [u8; 32],
|
||||
part_plain_sizes: &[usize],
|
||||
user_defined: HashMap<String, String>,
|
||||
) -> LegacyMultipartFixture {
|
||||
let layouts = vec![FixturePartLayout::Legacy; part_plain_sizes.len()];
|
||||
build_multipart_fixture_with_layouts(bucket, object, key_bytes, part_plain_sizes, &layouts, user_defined).await
|
||||
}
|
||||
|
||||
/// Stored layout of one fixture part: the legacy v1 segment, or the
|
||||
/// part-keyed v2 segment written under `RUSTFS_ENCRYPTION_MULTIPART_PART_KEY`.
|
||||
#[derive(Clone, Copy)]
|
||||
enum FixturePartLayout {
|
||||
Legacy,
|
||||
PartKeyed,
|
||||
}
|
||||
|
||||
async fn build_multipart_fixture_with_layouts(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
key_bytes: [u8; 32],
|
||||
part_plain_sizes: &[usize],
|
||||
layouts: &[FixturePartLayout],
|
||||
mut user_defined: HashMap<String, String>,
|
||||
) -> LegacyMultipartFixture {
|
||||
assert_eq!(part_plain_sizes.len(), layouts.len());
|
||||
let mut plaintext = Vec::new();
|
||||
let mut ciphertext = Vec::new();
|
||||
let mut parts = Vec::new();
|
||||
let mut part_physical_sizes = Vec::new();
|
||||
|
||||
for (part_index, &part_plain_size) in part_plain_sizes.iter().enumerate() {
|
||||
for (part_index, (&part_plain_size, layout)) in part_plain_sizes.iter().zip(layouts).enumerate() {
|
||||
let part_number = part_index + 1;
|
||||
let part_plain = legacy_fixture_part_plaintext(part_number, part_plain_size);
|
||||
let mut part_cipher = Vec::new();
|
||||
rustfs_rio::EncryptReader::new_multipart(
|
||||
Cursor::new(part_plain.clone()),
|
||||
key_bytes,
|
||||
LEGACY_FIXTURE_BASE_NONCE,
|
||||
part_number,
|
||||
)
|
||||
.read_to_end(&mut part_cipher)
|
||||
.await
|
||||
.expect("encrypt multipart fixture part");
|
||||
match layout {
|
||||
FixturePartLayout::Legacy => rustfs_rio::EncryptReader::new_multipart(
|
||||
Cursor::new(part_plain.clone()),
|
||||
key_bytes,
|
||||
LEGACY_FIXTURE_BASE_NONCE,
|
||||
part_number,
|
||||
)
|
||||
.read_to_end(&mut part_cipher)
|
||||
.await
|
||||
.expect("encrypt multipart fixture part"),
|
||||
FixturePartLayout::PartKeyed => rustfs_rio::EncryptReader::new_multipart_part_keyed(
|
||||
Cursor::new(part_plain.clone()),
|
||||
key_bytes,
|
||||
LEGACY_FIXTURE_BASE_NONCE,
|
||||
part_number,
|
||||
)
|
||||
.read_to_end(&mut part_cipher)
|
||||
.await
|
||||
.expect("encrypt part-keyed multipart fixture part"),
|
||||
};
|
||||
|
||||
parts.push(ObjectPartInfo {
|
||||
number: part_number,
|
||||
@@ -3486,6 +3518,71 @@ mod tests {
|
||||
.await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn part_keyed_and_mixed_ssec_multipart_reads_are_byte_exact() {
|
||||
use FixturePartLayout::{Legacy, PartKeyed};
|
||||
let key_bytes = [0x74; 32];
|
||||
let sizes = [20_000, 9_000, 5_000];
|
||||
let total_plaintext: usize = sizes.iter().sum();
|
||||
let headers = ssec_headers_from_key(key_bytes);
|
||||
for layouts in [
|
||||
[PartKeyed, PartKeyed, PartKeyed],
|
||||
[Legacy, PartKeyed, Legacy],
|
||||
[PartKeyed, Legacy, PartKeyed],
|
||||
] {
|
||||
let fixture = build_multipart_fixture_with_layouts(
|
||||
"bucket",
|
||||
"part-keyed-multipart",
|
||||
key_bytes,
|
||||
&sizes,
|
||||
&layouts,
|
||||
legacy_ssec_multipart_metadata(key_bytes, total_plaintext),
|
||||
)
|
||||
.await;
|
||||
let total = fixture.plaintext.len() as i64;
|
||||
let starts = [0, fixture.physical_part_start(1), fixture.physical_part_start(2)];
|
||||
let ranges: [(i64, i64, usize); 9] = [
|
||||
(0, 9, starts[0]),
|
||||
(8_191, 8_193, starts[0]),
|
||||
(19_999, 20_000, starts[0]),
|
||||
(20_000, 20_000, starts[1]),
|
||||
(20_100, 20_199, starts[1]),
|
||||
(20_000, 29_100, starts[1]),
|
||||
(28_999, 29_000, starts[1]),
|
||||
(33_900, 33_999, starts[2]),
|
||||
(0, total - 1, starts[0]),
|
||||
];
|
||||
|
||||
for seek in ["true", "false"] {
|
||||
async_with_vars([(ENV_RUSTFS_ENCRYPTED_RANGE_SEEK, Some(seek))], async {
|
||||
let opts = ObjectOptions::default();
|
||||
for (start, end, part_start) in ranges {
|
||||
let label = format!("seek={seek} range {start}-{end}");
|
||||
let (body, offset, _, reported) =
|
||||
read_via_seek_window(&fixture, Some(range(start, end)), &opts, &headers).await;
|
||||
let expected = &fixture.plaintext[start as usize..=end as usize];
|
||||
assert_eq!(body, expected, "{label}: body bytes");
|
||||
assert_eq!(reported, end - start + 1, "{label}: reported size");
|
||||
let expected_offset = if seek == "true" { part_start } else { 0 };
|
||||
assert_eq!(offset, expected_offset, "{label}: physical offset");
|
||||
}
|
||||
|
||||
let (body, offset, length, _) = read_via_seek_window(&fixture, None, &opts, &headers).await;
|
||||
assert_eq!((offset, length), (0, fixture.ciphertext.len() as i64), "seek={seek}: full read");
|
||||
assert_eq!(body, fixture.plaintext, "seek={seek}: full body");
|
||||
|
||||
let part_two = ObjectOptions {
|
||||
part_number: Some(2),
|
||||
..Default::default()
|
||||
};
|
||||
let (body, _, _, _) = read_via_seek_window(&fixture, None, &part_two, &headers).await;
|
||||
assert_eq!(body, &fixture.plaintext[20_000..29_000], "seek={seek}: partNumber=2 body");
|
||||
})
|
||||
.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The amplification-inversion guard: a small Range inside part 2 must schedule
|
||||
/// only part 2 and must not pull more than part 2's ciphertext. The stream is
|
||||
/// served from the planned offset all the way to the object end WITHOUT an end
|
||||
|
||||
@@ -65,6 +65,7 @@ rand = { workspace = true, features = ["serde"] }
|
||||
http.workspace = true
|
||||
hyper.workspace = true
|
||||
aes-gcm = { workspace = true, features = ["rand_core"] }
|
||||
hmac.workspace = true
|
||||
crc-fast = { workspace = true }
|
||||
pin-project-lite.workspace = true
|
||||
serde = { workspace = true, features = ["derive"] }
|
||||
|
||||
@@ -15,8 +15,11 @@
|
||||
use crate::compress_index::{Index, TryGetIndex};
|
||||
use aes_gcm::aead::{Aead, Payload};
|
||||
use aes_gcm::{Aes256Gcm, KeyInit, Nonce};
|
||||
use hmac::{Hmac, Mac};
|
||||
use pin_project_lite::pin_project;
|
||||
use rand::Rng;
|
||||
use rustfs_utils::{put_uvarint, put_uvarint_len};
|
||||
use sha2::Sha256;
|
||||
use std::io::Error;
|
||||
use std::pin::Pin;
|
||||
use std::task::{Context, Poll};
|
||||
@@ -46,6 +49,54 @@ const FRAME_TYPE_V2: u8 = 0x01;
|
||||
const FRAME_TYPE_V2_FINAL: u8 = 0x02;
|
||||
const FRAME_TYPE_END: u8 = 0xFF;
|
||||
|
||||
/// Leading frame of a part-keyed multipart segment.
|
||||
///
|
||||
/// Every part segment of a multipart object shares the upload's data key and
|
||||
/// base nonce, and the per-block nonces depend only on the part number and
|
||||
/// the block index. Writing the same part number twice within one upload
|
||||
/// therefore repeats the exact (key, nonce) sequence. A part-keyed segment
|
||||
/// opens with this frame, which carries a fresh random salt; the segment's
|
||||
/// v2 frames are then sealed under a key derived from the data key, the salt
|
||||
/// and the part number, so every part write uses its own AES-GCM key.
|
||||
///
|
||||
/// Layout: the usual 8-byte header (type, `len = salt + 4` as u24 LE, CRC32
|
||||
/// of the salt) followed by the raw salt. The salt is not secret; a modified
|
||||
/// salt derives a different key and the segment's frames fail authentication.
|
||||
const FRAME_TYPE_PART_KEY: u8 = 0x03;
|
||||
const PART_KEY_SALT_LEN: usize = 32;
|
||||
const PART_KEY_DERIVATION_LABEL: &[u8] = b"rustfs-sse-multipart-part-key-v1";
|
||||
|
||||
type HmacSha256 = Hmac<Sha256>;
|
||||
|
||||
/// Key that seals one part-keyed segment: HMAC-SHA256 over a fixed label,
|
||||
/// the part number and the segment's salt, keyed by the object data key.
|
||||
fn derive_part_segment_key(key: &[u8; 32], salt: &[u8; PART_KEY_SALT_LEN], part_number: usize) -> [u8; 32] {
|
||||
let mut mac = HmacSha256::new_from_slice(key).expect("HMAC-SHA256 accepts 32-byte keys");
|
||||
mac.update(PART_KEY_DERIVATION_LABEL);
|
||||
mac.update(&(part_number as u64).to_be_bytes());
|
||||
mac.update(salt);
|
||||
let mut segment_key = [0u8; 32];
|
||||
segment_key.copy_from_slice(mac.finalize().into_bytes().as_slice());
|
||||
segment_key
|
||||
}
|
||||
|
||||
fn crc32(bytes: &[u8]) -> u32 {
|
||||
let mut hasher = crc_fast::Digest::new(crc_fast::CrcAlgorithm::Crc32IsoHdlc);
|
||||
hasher.update(bytes);
|
||||
hasher.finalize() as u32
|
||||
}
|
||||
|
||||
fn part_key_frame(salt: &[u8; PART_KEY_SALT_LEN]) -> Vec<u8> {
|
||||
let len = PART_KEY_SALT_LEN + 4;
|
||||
let crc = crc32(salt);
|
||||
let mut out = Vec::with_capacity(8 + PART_KEY_SALT_LEN);
|
||||
out.push(FRAME_TYPE_PART_KEY);
|
||||
out.extend_from_slice(&(len as u32).to_le_bytes()[..3]);
|
||||
out.extend_from_slice(&crc.to_le_bytes());
|
||||
out.extend_from_slice(salt);
|
||||
out
|
||||
}
|
||||
|
||||
/// AEAD associated data of a v2 frame: the 8-byte header followed by the
|
||||
/// frame index within its segment, little-endian.
|
||||
fn v2_frame_aad(header: &[u8; 8], block_index: usize) -> [u8; 16] {
|
||||
@@ -72,6 +123,8 @@ pin_project! {
|
||||
frame_v2: bool,
|
||||
pending: usize,
|
||||
input_done: bool,
|
||||
// Salt of a part-keyed segment, emitted as the segment's first frame.
|
||||
pending_part_key_frame: Option<[u8; PART_KEY_SALT_LEN]>,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -92,6 +145,7 @@ where
|
||||
frame_v2: false,
|
||||
pending: 0,
|
||||
input_done: false,
|
||||
pending_part_key_frame: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -116,6 +170,28 @@ where
|
||||
reader.frame_v2 = true;
|
||||
reader
|
||||
}
|
||||
|
||||
/// Multipart writer for a part-keyed v2 segment (see the
|
||||
/// `FRAME_TYPE_PART_KEY` frame): a fresh random salt per call, so writing the
|
||||
/// same part number again never reuses a (key, nonce) pair.
|
||||
pub fn new_multipart_part_keyed(inner: R, key: [u8; 32], base_nonce: [u8; 12], part_number: usize) -> Self {
|
||||
let mut salt = [0u8; PART_KEY_SALT_LEN];
|
||||
rand::rng().fill_bytes(&mut salt);
|
||||
Self::new_multipart_part_keyed_with_salt(inner, key, base_nonce, part_number, salt)
|
||||
}
|
||||
|
||||
fn new_multipart_part_keyed_with_salt(
|
||||
inner: R,
|
||||
key: [u8; 32],
|
||||
base_nonce: [u8; 12],
|
||||
part_number: usize,
|
||||
salt: [u8; PART_KEY_SALT_LEN],
|
||||
) -> Self {
|
||||
let segment_key = derive_part_segment_key(&key, &salt, part_number);
|
||||
let mut reader = Self::new_multipart_v2(inner, segment_key, base_nonce, part_number);
|
||||
reader.pending_part_key_frame = Some(salt);
|
||||
reader
|
||||
}
|
||||
}
|
||||
|
||||
/// Build one frame: header, plaintext-length uvarint, ciphertext. For v2
|
||||
@@ -193,6 +269,14 @@ where
|
||||
return Poll::Ready(Ok(()));
|
||||
}
|
||||
|
||||
if let Some(salt) = this.pending_part_key_frame.take() {
|
||||
*this.buffer = part_key_frame(&salt);
|
||||
let to_copy = std::cmp::min(buf.remaining(), this.buffer.len());
|
||||
buf.put_slice(&this.buffer[..to_copy]);
|
||||
*this.buffer_pos = to_copy;
|
||||
return Poll::Ready(Ok(()));
|
||||
}
|
||||
|
||||
if *this.frame_v2 {
|
||||
// Accumulate a full block so every non-final frame carries exactly
|
||||
// ENCRYPTION_BLOCK_SIZE plaintext bytes (fixed-length frames give
|
||||
@@ -409,6 +493,11 @@ pin_project! {
|
||||
segments_completed: usize,
|
||||
v1_nonce_layout: Option<V1NonceLayout>,
|
||||
legacy_nonce_fallback: bool,
|
||||
// Part-keyed segments (see FRAME_TYPE_PART_KEY): the object data key
|
||||
// the segment keys derive from, and the current segment's cipher.
|
||||
key: [u8; 32],
|
||||
segment_keyed: bool,
|
||||
segment_cipher: Option<Aes256Gcm>,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -444,6 +533,9 @@ where
|
||||
segments_completed: 0,
|
||||
v1_nonce_layout: None,
|
||||
legacy_nonce_fallback: legacy_nonce_fallback_enabled(),
|
||||
key,
|
||||
segment_keyed: false,
|
||||
segment_cipher: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -496,6 +588,9 @@ where
|
||||
segments_completed: 0,
|
||||
v1_nonce_layout: None,
|
||||
legacy_nonce_fallback: legacy_nonce_fallback_enabled(),
|
||||
key,
|
||||
segment_keyed: false,
|
||||
segment_cipher: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -540,7 +635,7 @@ where
|
||||
// frames were dropped.
|
||||
if *this.segment_frame_version == Some(2)
|
||||
&& !*this.saw_final_frame
|
||||
&& *this.segment_frames > 0
|
||||
&& (*this.segment_frames > 0 || *this.segment_keyed)
|
||||
{
|
||||
return Poll::Ready(Err(Error::new(
|
||||
std::io::ErrorKind::UnexpectedEof,
|
||||
@@ -603,6 +698,8 @@ where
|
||||
*this.saw_final_frame = false;
|
||||
*this.segment_frames = 0;
|
||||
*this.v1_nonce_layout = None;
|
||||
*this.segment_keyed = false;
|
||||
*this.segment_cipher = None;
|
||||
|
||||
if *this.multipart_mode {
|
||||
let next_part = if *this.current_part_index + 1 < this.multipart_parts.len() {
|
||||
@@ -630,6 +727,33 @@ where
|
||||
continue;
|
||||
}
|
||||
|
||||
if typ == FRAME_TYPE_PART_KEY {
|
||||
// Only the first frame of a multipart part segment may
|
||||
// name the segment key; anywhere else it would let a
|
||||
// spliced frame switch keys mid-segment.
|
||||
if !*this.multipart_mode || this.segment_frame_version.is_some() || *this.segment_keyed {
|
||||
return Poll::Ready(Err(Error::new(
|
||||
std::io::ErrorKind::InvalidData,
|
||||
"part key frame outside the start of a multipart part segment",
|
||||
)));
|
||||
}
|
||||
if len != PART_KEY_SALT_LEN + 4 {
|
||||
return Poll::Ready(Err(Error::new(std::io::ErrorKind::InvalidData, "invalid part key frame length")));
|
||||
}
|
||||
// A part-keyed segment carries only authenticated v2 frames
|
||||
// and must end with its final frame.
|
||||
*this.segment_keyed = true;
|
||||
*this.segment_frame_version = Some(2);
|
||||
*this.stream_saw_v2 = true;
|
||||
*this.current_frame_type = typ;
|
||||
if this.ciphertext_buf.len() < PART_KEY_SALT_LEN {
|
||||
this.ciphertext_buf.resize(PART_KEY_SALT_LEN, 0);
|
||||
}
|
||||
*this.ciphertext_len = PART_KEY_SALT_LEN;
|
||||
*this.ciphertext_read = 0;
|
||||
continue;
|
||||
}
|
||||
|
||||
let frame_version = match typ {
|
||||
FRAME_TYPE_V1 => 1,
|
||||
FRAME_TYPE_V2 | FRAME_TYPE_V2_FINAL => 2,
|
||||
@@ -714,6 +838,22 @@ where
|
||||
return Poll::Pending;
|
||||
}
|
||||
|
||||
if *this.current_frame_type == FRAME_TYPE_PART_KEY {
|
||||
let mut salt = [0u8; PART_KEY_SALT_LEN];
|
||||
salt.copy_from_slice(&this.ciphertext_buf[..PART_KEY_SALT_LEN]);
|
||||
*this.ciphertext_read = 0;
|
||||
*this.ciphertext_len = 0;
|
||||
let expected_crc =
|
||||
u32::from_le_bytes([this.header_buf[4], this.header_buf[5], this.header_buf[6], this.header_buf[7]]);
|
||||
if crc32(&salt) != expected_crc {
|
||||
return Poll::Ready(Err(Error::new(std::io::ErrorKind::InvalidData, "part key frame CRC32 mismatch")));
|
||||
}
|
||||
let segment_key = derive_part_segment_key(this.key, &salt, *this.current_part);
|
||||
*this.segment_cipher =
|
||||
Some(Aes256Gcm::new_from_slice(&segment_key).map_err(|_| Error::other("invalid part segment key length"))?);
|
||||
continue;
|
||||
}
|
||||
|
||||
let ciphertext_buf = &this.ciphertext_buf[..*this.ciphertext_len];
|
||||
// `ciphertext_buf`'s length derives from the untrusted 24-bit header length field, so
|
||||
// it can be shorter than 16 bytes. `uvarint` is safe on any slice length, so pass the
|
||||
@@ -736,7 +876,11 @@ where
|
||||
// derivation is exactly the modern scheme, and there are no
|
||||
// legacy fallbacks — any mismatch is tampering, not history.
|
||||
let aad = v2_frame_aad(this.header_buf, *this.block_index);
|
||||
this.cipher
|
||||
let cipher: &Aes256Gcm = match this.segment_cipher.as_ref() {
|
||||
Some(segment_cipher) => segment_cipher,
|
||||
None => this.cipher,
|
||||
};
|
||||
cipher
|
||||
.decrypt(
|
||||
&nonce,
|
||||
Payload {
|
||||
@@ -1819,4 +1963,276 @@ mod tests {
|
||||
.await
|
||||
.expect_err("a wrong absolute frame index must fail authentication");
|
||||
}
|
||||
|
||||
// Frozen ciphertext written by the pre-part-key multipart writers
|
||||
// (`new_multipart` v1 for part 1, `new_multipart_v2` for part 2) under
|
||||
// FIXTURE_KEY / FIXTURE_BASE_NONCE. Existing objects keep these bytes on
|
||||
// disk, so they must decrypt unchanged, alone and next to part-keyed parts.
|
||||
const FIXTURE_KEY: [u8; 32] = [0x11; 32];
|
||||
const FIXTURE_BASE_NONCE: [u8; 12] = [0x22; 12];
|
||||
const FROZEN_V1_PART_ONE: &str = "003d00003c2ea60d28cc95ba74f479d9ba69fba8f3ae0dc5ff36bd9017ba93f2bf3538623cb9b1ea23527d7122e15f68054fab6f3a97001384264c8dbed0a6923aff00000000000000";
|
||||
const FROZEN_V2_PART_TWO: &str =
|
||||
"022d0000be2e9f09184b9d704c7411f32671790af637f706520d50a5da1459ee0c5e122ce4e5e9ccd0bf68abb412bb0439ff00000000000000";
|
||||
|
||||
fn frozen_part(hex: &str) -> Vec<u8> {
|
||||
let mut out = vec![0u8; hex.len() / 2];
|
||||
faster_hex::hex_decode(hex.as_bytes(), &mut out).expect("valid fixture hex");
|
||||
out
|
||||
}
|
||||
|
||||
fn frozen_part_one_plaintext() -> Vec<u8> {
|
||||
(0..40u8).collect()
|
||||
}
|
||||
|
||||
fn frozen_part_two_plaintext() -> Vec<u8> {
|
||||
(100..124u8).collect()
|
||||
}
|
||||
|
||||
async fn encrypt_part_keyed(data: &[u8], key: [u8; 32], base_nonce: [u8; 12], part_number: usize) -> Vec<u8> {
|
||||
let mut out = Vec::new();
|
||||
EncryptReader::new_multipart_part_keyed(Cursor::new(data.to_vec()), key, base_nonce, part_number)
|
||||
.read_to_end(&mut out)
|
||||
.await
|
||||
.expect("part-keyed encryption succeeds");
|
||||
out
|
||||
}
|
||||
|
||||
async fn decrypt_parts(stream: Vec<u8>, key: [u8; 32], base_nonce: [u8; 12], parts: Vec<usize>) -> std::io::Result<Vec<u8>> {
|
||||
let mut out = Vec::new();
|
||||
DecryptReader::new_multipart(BufReader::new(Cursor::new(stream)), key, base_nonce, parts)
|
||||
.read_to_end(&mut out)
|
||||
.await?;
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn part_key_salt(segment: &[u8]) -> [u8; super::PART_KEY_SALT_LEN] {
|
||||
assert_eq!(segment[0], super::FRAME_TYPE_PART_KEY, "a part-keyed segment opens with its key frame");
|
||||
segment[8..8 + super::PART_KEY_SALT_LEN].try_into().expect("salt slice")
|
||||
}
|
||||
|
||||
/// Ciphertext bytes (tag excluded) of the frame whose header starts at
|
||||
/// `frame_start`.
|
||||
fn frame_ciphertext(segment: &[u8], frame_start: usize, plaintext_len: usize) -> Vec<u8> {
|
||||
assert_ne!(segment[frame_start], super::FRAME_TYPE_END);
|
||||
let ct_start = frame_start + 8 + put_uvarint_len(plaintext_len as u64);
|
||||
segment[ct_start..ct_start + plaintext_len].to_vec()
|
||||
}
|
||||
|
||||
fn xor(a: &[u8], b: &[u8]) -> Vec<u8> {
|
||||
a.iter().zip(b).map(|(x, y)| x ^ y).collect()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn legacy_part_writes_of_one_part_number_share_a_keystream() {
|
||||
// The defect the part-keyed layout closes: the legacy writer derives
|
||||
// the same (key, nonce) sequence each time a part number is written,
|
||||
// so the XOR of two ciphertexts equals the XOR of their plaintexts.
|
||||
let first = vec![0x41u8; 64];
|
||||
let second = vec![0x7Au8; 64];
|
||||
let mut c1 = Vec::new();
|
||||
EncryptReader::new_multipart(Cursor::new(first.clone()), FIXTURE_KEY, FIXTURE_BASE_NONCE, 3)
|
||||
.read_to_end(&mut c1)
|
||||
.await
|
||||
.expect("legacy encryption succeeds");
|
||||
let mut c2 = Vec::new();
|
||||
EncryptReader::new_multipart(Cursor::new(second.clone()), FIXTURE_KEY, FIXTURE_BASE_NONCE, 3)
|
||||
.read_to_end(&mut c2)
|
||||
.await
|
||||
.expect("legacy encryption succeeds");
|
||||
assert_eq!(xor(&frame_ciphertext(&c1, 0, 64), &frame_ciphertext(&c2, 0, 64)), xor(&first, &second));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn part_keyed_rewrites_of_one_part_number_never_reuse_a_key_nonce_pair() {
|
||||
let block = ENCRYPTION_BLOCK_SIZE;
|
||||
let first = vec![0x41u8; block + 64];
|
||||
let second = vec![0x7Au8; block + 64];
|
||||
let c1 = encrypt_part_keyed(&first, FIXTURE_KEY, FIXTURE_BASE_NONCE, 3).await;
|
||||
let c2 = encrypt_part_keyed(&second, FIXTURE_KEY, FIXTURE_BASE_NONCE, 3).await;
|
||||
|
||||
let (s1, s2) = (part_key_salt(&c1), part_key_salt(&c2));
|
||||
assert_ne!(s1, s2, "each part write draws a fresh salt");
|
||||
let k1 = super::derive_part_segment_key(&FIXTURE_KEY, &s1, 3);
|
||||
let k2 = super::derive_part_segment_key(&FIXTURE_KEY, &s2, 3);
|
||||
assert_ne!(k1, k2, "rewrites of one part number seal under different keys");
|
||||
assert_ne!(k1, FIXTURE_KEY, "the object data key never seals a part-keyed frame");
|
||||
|
||||
// Same block position in both writes: the nonce is identical by
|
||||
// construction, so the (key, nonce) pair differs only through the key,
|
||||
// and the keystream no longer cancels out.
|
||||
let frame_start = 8 + super::PART_KEY_SALT_LEN;
|
||||
assert_ne!(
|
||||
xor(&frame_ciphertext(&c1, frame_start, block), &frame_ciphertext(&c2, frame_start, block)),
|
||||
xor(&first[..block], &second[..block])
|
||||
);
|
||||
|
||||
// An identical-content retry also yields distinct ciphertext.
|
||||
let c3 = encrypt_part_keyed(&first, FIXTURE_KEY, FIXTURE_BASE_NONCE, 3).await;
|
||||
assert_ne!(c1, c3);
|
||||
for (stream, expected) in [(c1, &first), (c2, &second), (c3, &first)] {
|
||||
let decrypted = decrypt_parts(stream, FIXTURE_KEY, FIXTURE_BASE_NONCE, vec![3])
|
||||
.await
|
||||
.expect("part-keyed segment decrypts");
|
||||
assert_eq!(&decrypted, expected);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn part_keyed_segments_round_trip_every_boundary_length() {
|
||||
let block = ENCRYPTION_BLOCK_SIZE;
|
||||
for len in [0, 1, block - 1, block, block + 1, 3 * block, 3 * block + 5] {
|
||||
let data: Vec<u8> = (0..len).map(|i| (i % 251) as u8).collect();
|
||||
let tail: Vec<u8> = vec![0xC3; 17];
|
||||
let mut stream = encrypt_part_keyed(&data, FIXTURE_KEY, FIXTURE_BASE_NONCE, 1).await;
|
||||
stream.extend_from_slice(&encrypt_part_keyed(&tail, FIXTURE_KEY, FIXTURE_BASE_NONCE, 2).await);
|
||||
let mut expected = data;
|
||||
expected.extend_from_slice(&tail);
|
||||
|
||||
// Small upstream reads split the key frame across polls.
|
||||
let mut chunked = Vec::new();
|
||||
DecryptReader::new_multipart(ChunkedCursor::new(stream.clone(), 5), FIXTURE_KEY, FIXTURE_BASE_NONCE, vec![1, 2])
|
||||
.read_to_end(&mut chunked)
|
||||
.await
|
||||
.unwrap_or_else(|e| panic!("chunked len {len}: {e}"));
|
||||
assert_eq!(chunked, expected, "chunked len {len}");
|
||||
|
||||
let decrypted = decrypt_parts(stream, FIXTURE_KEY, FIXTURE_BASE_NONCE, vec![1, 2])
|
||||
.await
|
||||
.unwrap_or_else(|e| panic!("len {len}: {e}"));
|
||||
assert_eq!(decrypted, expected, "len {len}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn frozen_legacy_parts_still_decrypt_alone_and_mixed_with_part_keyed_parts() {
|
||||
let part_one = frozen_part(FROZEN_V1_PART_ONE);
|
||||
let part_two = frozen_part(FROZEN_V2_PART_TWO);
|
||||
let mut expected = frozen_part_one_plaintext();
|
||||
expected.extend_from_slice(&frozen_part_two_plaintext());
|
||||
let legacy = [part_one.as_slice(), part_two.as_slice()].concat();
|
||||
assert_eq!(
|
||||
decrypt_parts(legacy, FIXTURE_KEY, FIXTURE_BASE_NONCE, vec![1, 2])
|
||||
.await
|
||||
.expect("frozen legacy parts decrypt"),
|
||||
expected
|
||||
);
|
||||
|
||||
// Part 3 was written after the upgrade; parts 1 and 2 stay legacy.
|
||||
let part_three_plain = vec![0x5Cu8; ENCRYPTION_BLOCK_SIZE + 9];
|
||||
let part_three = encrypt_part_keyed(&part_three_plain, FIXTURE_KEY, FIXTURE_BASE_NONCE, 3).await;
|
||||
let mut expected_all = expected.clone();
|
||||
expected_all.extend_from_slice(&part_three_plain);
|
||||
let mixed = [part_one.as_slice(), part_two.as_slice(), part_three.as_slice()].concat();
|
||||
assert_eq!(
|
||||
decrypt_parts(mixed, FIXTURE_KEY, FIXTURE_BASE_NONCE, vec![1, 2, 3])
|
||||
.await
|
||||
.expect("legacy and part-keyed parts decrypt together"),
|
||||
expected_all
|
||||
);
|
||||
|
||||
// A legacy part between part-keyed parts.
|
||||
let keyed_one = encrypt_part_keyed(&frozen_part_one_plaintext(), FIXTURE_KEY, FIXTURE_BASE_NONCE, 1).await;
|
||||
let interleaved = [keyed_one.as_slice(), part_two.as_slice(), part_three.as_slice()].concat();
|
||||
assert_eq!(
|
||||
decrypt_parts(interleaved, FIXTURE_KEY, FIXTURE_BASE_NONCE, vec![1, 2, 3])
|
||||
.await
|
||||
.expect("part-keyed, legacy v2 and part-keyed parts decrypt together"),
|
||||
expected_all
|
||||
);
|
||||
|
||||
// A read that starts at a later part boundary (multipart range seek).
|
||||
let from_two = [part_two.as_slice(), part_three.as_slice()].concat();
|
||||
let mut expected_from_two = frozen_part_two_plaintext();
|
||||
expected_from_two.extend_from_slice(&part_three_plain);
|
||||
assert_eq!(
|
||||
decrypt_parts(from_two, FIXTURE_KEY, FIXTURE_BASE_NONCE, vec![2, 3])
|
||||
.await
|
||||
.expect("seek to a part boundary"),
|
||||
expected_from_two
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn part_keyed_segments_reject_tampering_and_misplacement() {
|
||||
let data = vec![0x6Du8; ENCRYPTION_BLOCK_SIZE + 3];
|
||||
let segment = encrypt_part_keyed(&data, FIXTURE_KEY, FIXTURE_BASE_NONCE, 2).await;
|
||||
let key_frame_len = 8 + super::PART_KEY_SALT_LEN;
|
||||
let decrypt = |stream: Vec<u8>, parts: Vec<usize>| decrypt_parts(stream, FIXTURE_KEY, FIXTURE_BASE_NONCE, parts);
|
||||
|
||||
// A modified salt with a consistent CRC derives another key.
|
||||
let mut salt_flip = segment.clone();
|
||||
salt_flip[8] ^= 0x01;
|
||||
let crc = super::crc32(&salt_flip[8..key_frame_len]);
|
||||
salt_flip[4..8].copy_from_slice(&crc.to_le_bytes());
|
||||
let err = decrypt(salt_flip, vec![2]).await.expect_err("salt tampering must fail");
|
||||
assert_eq!(err.to_string(), "v2 encrypted frame failed authentication");
|
||||
|
||||
// A salt that disagrees with its header CRC is corruption.
|
||||
let mut crc_flip = segment.clone();
|
||||
crc_flip[9] ^= 0x01;
|
||||
let err = decrypt(crc_flip, vec![2]).await.expect_err("salt CRC mismatch must fail");
|
||||
assert_eq!(err.to_string(), "part key frame CRC32 mismatch");
|
||||
|
||||
// The segment key binds the part number: the same bytes listed as
|
||||
// another part fail authentication.
|
||||
decrypt(segment.clone(), vec![5])
|
||||
.await
|
||||
.expect_err("a segment moved to another part must fail");
|
||||
|
||||
// Without its key frame the segment is decrypted under the data key.
|
||||
decrypt(segment[key_frame_len..].to_vec(), vec![2])
|
||||
.await
|
||||
.expect_err("a stripped key frame must fail");
|
||||
|
||||
// A key frame alone, or followed only by the end marker, is truncation.
|
||||
decrypt(segment[..key_frame_len].to_vec(), vec![2])
|
||||
.await
|
||||
.expect_err("a lone key frame must fail");
|
||||
let mut key_then_end = segment[..key_frame_len].to_vec();
|
||||
key_then_end.extend_from_slice(&[super::FRAME_TYPE_END, 0, 0, 0, 0, 0, 0, 0]);
|
||||
decrypt(key_then_end, vec![2])
|
||||
.await
|
||||
.expect_err("an empty keyed segment must fail");
|
||||
|
||||
// A second key frame cannot switch keys mid-segment.
|
||||
let first_frame_end = key_frame_len + V2_FULL_FRAME_LEN;
|
||||
let rekeyed = [
|
||||
&segment[..first_frame_end],
|
||||
&segment[..key_frame_len],
|
||||
&segment[first_frame_end..],
|
||||
]
|
||||
.concat();
|
||||
let err = decrypt(rekeyed, vec![2])
|
||||
.await
|
||||
.expect_err("a mid-segment key frame must fail");
|
||||
assert_eq!(err.to_string(), "part key frame outside the start of a multipart part segment");
|
||||
|
||||
// A keyed segment carries only v2 frames.
|
||||
let mut v1 = Vec::new();
|
||||
EncryptReader::new_multipart(Cursor::new(vec![0x01u8; 16]), FIXTURE_KEY, FIXTURE_BASE_NONCE, 2)
|
||||
.read_to_end(&mut v1)
|
||||
.await
|
||||
.expect("v1 encryption succeeds");
|
||||
let v1_after_key = [&segment[..key_frame_len], v1.as_slice()].concat();
|
||||
decrypt(v1_after_key, vec![2])
|
||||
.await
|
||||
.expect_err("v1 frames in a keyed segment must fail");
|
||||
|
||||
// Malformed key-frame length.
|
||||
let mut bad_len = segment.clone();
|
||||
bad_len[1] = bad_len[1].wrapping_add(1);
|
||||
let err = decrypt(bad_len, vec![2]).await.expect_err("a malformed key frame must fail");
|
||||
assert_eq!(err.to_string(), "invalid part key frame length");
|
||||
|
||||
// Single-part streams never carry part keys.
|
||||
let mut single = Vec::new();
|
||||
let err = DecryptReader::new(Cursor::new(segment.clone()), FIXTURE_KEY, FIXTURE_BASE_NONCE)
|
||||
.read_to_end(&mut single)
|
||||
.await
|
||||
.expect_err("a single-part reader must reject a key frame");
|
||||
assert_eq!(err.to_string(), "part key frame outside the start of a multipart part segment");
|
||||
|
||||
// Control: the untampered segment decrypts.
|
||||
assert_eq!(decrypt(segment, vec![2]).await.expect("control decrypt"), data);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -840,6 +840,7 @@ async fn scan_and_persist_local_bucket(
|
||||
}
|
||||
match persist_scanner_checkpoint(
|
||||
set.clone(),
|
||||
store.clone(),
|
||||
ScannerCheckpointPersistContext {
|
||||
ctx: &scan_ctx,
|
||||
expected_publication_epoch,
|
||||
|
||||
@@ -69,10 +69,13 @@ async fn wait_for_checkpoint_foreground_quiet(ctx: &CancellationToken) -> bool {
|
||||
/// Persist one bounded checkpoint and refresh its CAS revisions.
|
||||
///
|
||||
/// Local and remote workers share the same publication/leader fencing and
|
||||
/// revision-refresh contract; only their lock/cancellation handling remains
|
||||
/// revision-refresh contract. Cycle/leader state belongs to the global store,
|
||||
/// while cache revisions and publication admission belong to the set store.
|
||||
/// Only their lock/cancellation handling remains
|
||||
/// at the caller because those guards have different concrete types.
|
||||
pub(crate) async fn persist_scanner_checkpoint<S>(
|
||||
pub(crate) async fn persist_scanner_checkpoint<S, F>(
|
||||
store: Arc<S>,
|
||||
fence_store: Arc<F>,
|
||||
context: ScannerCheckpointPersistContext<'_>,
|
||||
cache_name: &str,
|
||||
checkpoint: &DataUsageCache,
|
||||
@@ -80,6 +83,7 @@ pub(crate) async fn persist_scanner_checkpoint<S>(
|
||||
) -> ScannerCheckpointPersistResult
|
||||
where
|
||||
S: ScannerObjectIO + ScannerConfigObjectDelete,
|
||||
F: ScannerObjectIO,
|
||||
{
|
||||
let foreground_quiet = wait_for_checkpoint_foreground_quiet(context.ctx).await;
|
||||
if !foreground_quiet && context.ctx.is_cancelled() {
|
||||
@@ -97,9 +101,13 @@ where
|
||||
);
|
||||
}
|
||||
|
||||
if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(context.cycle, context.leader_epoch, store.clone())
|
||||
.await
|
||||
.is_err()
|
||||
if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(
|
||||
context.cycle,
|
||||
context.leader_epoch,
|
||||
fence_store.clone(),
|
||||
)
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
return ScannerCheckpointPersistResult::FenceChanged;
|
||||
}
|
||||
@@ -117,7 +125,7 @@ where
|
||||
return ScannerCheckpointPersistResult::Failed(error);
|
||||
}
|
||||
|
||||
if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(context.cycle, context.leader_epoch, store.clone())
|
||||
if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(context.cycle, context.leader_epoch, fence_store)
|
||||
.await
|
||||
.is_err()
|
||||
|| scanner_publication_admission_for_epoch(store.clone(), context.expected_publication_epoch)
|
||||
|
||||
@@ -1189,8 +1189,14 @@ impl ScannerIOCache for SetDisks {
|
||||
await_scanner_disk_shutdown(scan.as_mut()).await;
|
||||
break Err(Error::other("scanner leader fence changed before checkpoint save"));
|
||||
}
|
||||
let Some(fence_store) = crate::resolve_scanner_object_store_handle() else {
|
||||
scan_ctx.cancel();
|
||||
await_scanner_disk_shutdown(scan.as_mut()).await;
|
||||
break Err(Error::other("scanner global cycle fence store is unavailable before checkpoint save"));
|
||||
};
|
||||
match persist_scanner_checkpoint(
|
||||
store_clone_clone.clone(),
|
||||
fence_store,
|
||||
ScannerCheckpointPersistContext {
|
||||
ctx: &ctx_clone,
|
||||
expected_publication_epoch: expected_publication_epoch_clone,
|
||||
|
||||
@@ -27,9 +27,10 @@ use crate::storage_api::owner::{
|
||||
};
|
||||
use crate::storage_api::scan::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions, ObjectIO as _};
|
||||
use crate::{
|
||||
DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints, ScannerObjectOptions,
|
||||
ScannerPutObjReader, UNKNOWN_TIER, init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests,
|
||||
init_local_disks_with_instance_ctx, new_disk, path2_bucket_object_with_base_path,
|
||||
DATA_USAGE_BLOOM_NAME_PATH, DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints,
|
||||
ScannerObjectOptions, ScannerPutObjReader, UNKNOWN_TIER, init_bucket_metadata_sys_for_scanner_tests,
|
||||
init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx, new_disk, path2_bucket_object_with_base_path,
|
||||
save_config,
|
||||
};
|
||||
use rustfs_concurrency::{
|
||||
AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadAdmissionSnapshotProvider,
|
||||
@@ -566,6 +567,303 @@ async fn setup_two_pool_scanner_store() -> (tempfile::TempDir, Arc<ECStore>) {
|
||||
(temp_dir, store)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn checkpoint_uses_global_cycle_fence_with_set_scoped_cache_revisions() {
|
||||
let (_temp_dir, store) = setup_two_pool_scanner_store().await;
|
||||
let cycle = 41341;
|
||||
let leader_epoch = 17;
|
||||
save_config(
|
||||
store.clone(),
|
||||
DATA_USAGE_BLOOM_NAME_PATH.as_str(),
|
||||
crate::scanner::encode_scanner_cycle_fence_for_test(cycle, leader_epoch),
|
||||
)
|
||||
.await
|
||||
.expect("persist cycle fence through global scanner store");
|
||||
crate::remote_scanner::validate_remote_scanner_request_fence_with_store(cycle, leader_epoch, store.clone())
|
||||
.await
|
||||
.expect("global scanner store must read the fence");
|
||||
let ctx = CancellationToken::new();
|
||||
let cache_name = "checkpoint-bucket/.usage-cache.bin";
|
||||
let mut checkpoint = DataUsageCache::default();
|
||||
checkpoint.info.name = "checkpoint-bucket".to_string();
|
||||
checkpoint.info.next_cycle = cycle;
|
||||
checkpoint.info.leader_epoch = leader_epoch;
|
||||
let mut mismatched_sets = 0;
|
||||
for (pool_index, pool) in store.pools.iter().enumerate() {
|
||||
for (set_index, set) in pool.disk_set.iter().enumerate() {
|
||||
let publication_epoch = scanner_publication_epoch(set.clone())
|
||||
.await
|
||||
.expect("set publication admission");
|
||||
DataUsageCache::default()
|
||||
.save(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("seed set-scoped cache");
|
||||
let mut revisions = DataUsageCache::read_revisions(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("initial set-scoped revisions");
|
||||
let initial_revisions = revisions.clone();
|
||||
let make_context = || ScannerCheckpointPersistContext {
|
||||
ctx: &ctx,
|
||||
expected_publication_epoch: publication_epoch,
|
||||
cycle,
|
||||
leader_epoch,
|
||||
};
|
||||
if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(cycle, leader_epoch, set.clone())
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
mismatched_sets += 1;
|
||||
assert!(
|
||||
matches!(
|
||||
persist_scanner_checkpoint(
|
||||
set.clone(),
|
||||
set.clone(),
|
||||
make_context(),
|
||||
cache_name,
|
||||
&checkpoint,
|
||||
&mut revisions,
|
||||
)
|
||||
.await,
|
||||
ScannerCheckpointPersistResult::FenceChanged
|
||||
),
|
||||
"set-scoped cycle lookup must reproduce the original failure for pool={pool_index}, set={set_index}"
|
||||
);
|
||||
}
|
||||
assert_eq!(revisions, initial_revisions, "rejected checkpoint must not update revisions");
|
||||
for skip_healing in [false, true] {
|
||||
checkpoint.info.skip_healing = skip_healing;
|
||||
let before_save = revisions.clone();
|
||||
assert!(
|
||||
matches!(
|
||||
persist_scanner_checkpoint(
|
||||
set.clone(),
|
||||
store.clone(),
|
||||
make_context(),
|
||||
cache_name,
|
||||
&checkpoint,
|
||||
&mut revisions,
|
||||
)
|
||||
.await,
|
||||
ScannerCheckpointPersistResult::Saved
|
||||
),
|
||||
"checkpoint helper must use the global cycle fence"
|
||||
);
|
||||
assert_ne!(revisions, before_save, "new checkpoint content must advance cache revisions");
|
||||
}
|
||||
let saved_revisions = revisions.clone();
|
||||
for rejected_context in [
|
||||
ScannerCheckpointPersistContext {
|
||||
cycle: cycle - 1,
|
||||
..make_context()
|
||||
},
|
||||
ScannerCheckpointPersistContext {
|
||||
leader_epoch: leader_epoch - 1,
|
||||
..make_context()
|
||||
},
|
||||
ScannerCheckpointPersistContext {
|
||||
expected_publication_epoch: publication_epoch + 1,
|
||||
..make_context()
|
||||
},
|
||||
] {
|
||||
assert!(
|
||||
matches!(
|
||||
persist_scanner_checkpoint(
|
||||
set.clone(),
|
||||
store.clone(),
|
||||
rejected_context,
|
||||
cache_name,
|
||||
&checkpoint,
|
||||
&mut revisions,
|
||||
)
|
||||
.await,
|
||||
ScannerCheckpointPersistResult::FenceChanged
|
||||
),
|
||||
"stale cycle, leader or publication epoch must reject checkpoint persistence"
|
||||
);
|
||||
assert_eq!(revisions, saved_revisions, "rejected writer must not refresh revisions");
|
||||
assert_eq!(
|
||||
DataUsageCache::read_revisions(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("read rejected checkpoint"),
|
||||
saved_revisions,
|
||||
"rejected writer must not mutate persisted caches"
|
||||
);
|
||||
}
|
||||
|
||||
let mut newer = checkpoint.clone();
|
||||
newer.info.next_cycle = cycle + 1;
|
||||
newer
|
||||
.save(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("simulate a newer cache writer");
|
||||
let newer_revisions = DataUsageCache::read_revisions(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("capture newer cache revisions");
|
||||
assert_ne!(newer_revisions, saved_revisions);
|
||||
assert!(
|
||||
matches!(
|
||||
persist_scanner_checkpoint(
|
||||
set.clone(),
|
||||
store.clone(),
|
||||
make_context(),
|
||||
cache_name,
|
||||
&checkpoint,
|
||||
&mut revisions,
|
||||
)
|
||||
.await,
|
||||
ScannerCheckpointPersistResult::Failed(StorageError::PreconditionFailed)
|
||||
),
|
||||
"matching global fence must not authorize stale cache CAS revisions"
|
||||
);
|
||||
assert_eq!(revisions, saved_revisions, "CAS failure must not refresh the stale writer");
|
||||
assert_eq!(
|
||||
DataUsageCache::read_revisions(set.clone(), cache_name)
|
||||
.await
|
||||
.expect("read cache after CAS failure"),
|
||||
newer_revisions,
|
||||
"CAS failure must preserve the newer persisted caches"
|
||||
);
|
||||
}
|
||||
}
|
||||
assert!(mismatched_sets > 0, "multi-pool fixture must expose a set-scoped cycle fence mismatch");
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct CheckpointFenceStore {
|
||||
before: Vec<u8>,
|
||||
after: Vec<u8>,
|
||||
fail_on_read: Option<usize>,
|
||||
reads: std::sync::atomic::AtomicUsize,
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl crate::storage_api::owner::ObjectIO for CheckpointFenceStore {
|
||||
type Error = crate::EcstoreError;
|
||||
type RangeSpec = crate::storage_api::owner::HTTPRangeSpec;
|
||||
type HeaderMap = http::HeaderMap;
|
||||
type ObjectOptions = crate::ScannerObjectOptions;
|
||||
type ObjectInfo = crate::ScannerObjectInfo;
|
||||
type GetObjectReader = crate::ScannerGetObjectReader;
|
||||
type PutObjectReader = crate::ScannerPutObjReader;
|
||||
|
||||
async fn get_object_reader(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
_range: Option<Self::RangeSpec>,
|
||||
_headers: Self::HeaderMap,
|
||||
_opts: &Self::ObjectOptions,
|
||||
) -> crate::EcstoreResult<Self::GetObjectReader> {
|
||||
assert_eq!(bucket, RUSTFS_META_BUCKET);
|
||||
assert_eq!(object, DATA_USAGE_BLOOM_NAME_PATH.as_str(), "global store must only read the cycle fence");
|
||||
let read = self.reads.fetch_add(1, Ordering::SeqCst);
|
||||
if self.fail_on_read == Some(read) {
|
||||
return Err(crate::EcstoreError::other("injected global fence read failure"));
|
||||
}
|
||||
let state = if read == 0 { &self.before } else { &self.after };
|
||||
Ok(crate::ScannerGetObjectReader {
|
||||
stream: Box::new(std::io::Cursor::new(state.clone())),
|
||||
object_info: crate::ScannerObjectInfo::default(),
|
||||
buffered_body: None,
|
||||
body_source: Default::default(),
|
||||
})
|
||||
}
|
||||
|
||||
async fn put_object(
|
||||
&self,
|
||||
_bucket: &str,
|
||||
_object: &str,
|
||||
_data: &mut Self::PutObjectReader,
|
||||
_opts: &Self::ObjectOptions,
|
||||
) -> crate::EcstoreResult<Self::ObjectInfo> {
|
||||
panic!("checkpoint persistence must never write through the global fence store");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn checkpoint_rejects_unreadable_or_changed_global_fence() {
|
||||
let (_temp_dir, store) = setup_two_pool_scanner_store().await;
|
||||
let set = &store.pools[1].disk_set[0];
|
||||
let publication_epoch = scanner_publication_epoch(set.clone()).await.expect("idle set admission");
|
||||
let cycle = 41341;
|
||||
let leader_epoch = 17;
|
||||
let valid = crate::scanner::encode_scanner_cycle_fence_for_test(cycle, leader_epoch);
|
||||
let ctx = CancellationToken::new();
|
||||
let mut checkpoint = DataUsageCache::default();
|
||||
checkpoint.info.name = "checkpoint-bucket".to_string();
|
||||
checkpoint.info.next_cycle = cycle;
|
||||
checkpoint.info.leader_epoch = leader_epoch;
|
||||
|
||||
for (case, before, after, fail_on_read, expected_reads) in [
|
||||
(
|
||||
"cycle_changed",
|
||||
valid.clone(),
|
||||
crate::scanner::encode_scanner_cycle_fence_for_test(cycle + 1, leader_epoch),
|
||||
None,
|
||||
2,
|
||||
),
|
||||
(
|
||||
"leader_changed",
|
||||
valid.clone(),
|
||||
crate::scanner::encode_scanner_cycle_fence_for_test(cycle, leader_epoch + 1),
|
||||
None,
|
||||
2,
|
||||
),
|
||||
("post_save_corrupt", valid.clone(), vec![0; 7], None, 2),
|
||||
("post_save_read_failure", valid.clone(), valid.clone(), Some(1), 2),
|
||||
("pre_save_corrupt", vec![0; 7], valid.clone(), None, 1),
|
||||
("pre_save_read_failure", valid.clone(), valid.clone(), Some(0), 1),
|
||||
] {
|
||||
let cache_name = format!("checkpoint-bucket/{case}/.usage-cache.bin");
|
||||
DataUsageCache::default()
|
||||
.save(set.clone(), &cache_name)
|
||||
.await
|
||||
.expect("seed checkpoint cache");
|
||||
let mut revisions = DataUsageCache::read_revisions(set.clone(), &cache_name)
|
||||
.await
|
||||
.expect("read seeded revisions");
|
||||
let initial = revisions.clone();
|
||||
let fence_store = Arc::new(CheckpointFenceStore {
|
||||
before,
|
||||
after,
|
||||
fail_on_read,
|
||||
reads: std::sync::atomic::AtomicUsize::new(0),
|
||||
});
|
||||
assert!(
|
||||
matches!(
|
||||
persist_scanner_checkpoint(
|
||||
set.clone(),
|
||||
fence_store.clone(),
|
||||
ScannerCheckpointPersistContext {
|
||||
ctx: &ctx,
|
||||
expected_publication_epoch: publication_epoch,
|
||||
cycle,
|
||||
leader_epoch
|
||||
},
|
||||
&cache_name,
|
||||
&checkpoint,
|
||||
&mut revisions,
|
||||
)
|
||||
.await,
|
||||
ScannerCheckpointPersistResult::FenceChanged
|
||||
),
|
||||
"unreadable or changed global fence must reject checkpoint success: {case}"
|
||||
);
|
||||
assert_eq!(fence_store.reads.load(Ordering::SeqCst), expected_reads, "global fence checks: {case}");
|
||||
assert_eq!(revisions, initial, "rejected checkpoint must not refresh caller revisions: {case}");
|
||||
let persisted = DataUsageCache::read_revisions(set.clone(), &cache_name)
|
||||
.await
|
||||
.expect("read persisted checkpoint");
|
||||
if expected_reads == 1 {
|
||||
assert_eq!(persisted, initial, "pre-save rejection must not mutate persisted caches: {case}");
|
||||
} else {
|
||||
assert_ne!(persisted, initial, "post-save fence check must run after a real cache write: {case}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_for_namespace_commit_tails(store: &ECStore) {
|
||||
tokio::time::timeout(Duration::from_secs(30), async {
|
||||
while store.scanner_data_usage_publication_blocked().await {
|
||||
|
||||
@@ -53,6 +53,7 @@
|
||||
- `backlog-2097-tier-mutation-v4-error-text` tier-mutation Prepare rejection classification: a v3 server rejects a v4 request before store/runtime dispatch with the FailedPrecondition status and an authenticated, byte-exact unsupported-version message. A v4 coordinator recognizes only that exact code/message/requested-version tuple as definitely not persisted and fails the mutation without sending that peer an incompatible Abort; Unimplemented, near-text, missing/unknown failure classes, timeouts, and every other transport outcome remain ambiguous and stay in identity-bound Abort fanout. There is no automatic v3 retry. New servers retain v3 request/proof decoding for older coordinators, while operators must pause tier edit/remove/clear during a mixed v3/v4 rollout. Remove the text classifier after the minimum supported RustFS peer version returns the signed v4 PreDispatchRejected failure class.
|
||||
- `backlog-2097-tier-delete-journal-v6` tier-delete sole-owner recovery: v6 distinguishes transactions that may replace the xl.meta free-version owner and therefore require the all-pool live-source proof. v5-and-older readers reject and retain v6 records during rolling upgrades instead of performing an unsafe remote delete. Keep v1-v5 readers for upgrade recovery and keep the downgrade prohibition while any v6 record exists; retire the fence only after every supported rollback release understands and enforces v6 proof semantics.
|
||||
- `multipart-compression-default-off-window` staged multipart disk-compression rollout: releases before the resumable legacy decompressor fail transient reads of compressed objects under mid-payload suspension, so multipart uploads advertise the compression marker only when RUSTFS_COMPRESSION_MULTIPART_ENABLED is set in addition to RUSTFS_COMPRESSION_ENABLED, keeping rolling upgrades from creating new compressed multipart objects while pre-fix nodes may still serve reads. Flip the default to enabled (and retire the extra switch) after the minimum supported direct-upgrade release ships the resumable decompressor.
|
||||
- `multipart-part-key-default-off-window` staged part-keyed multipart encryption rollout: Direct-mode multipart parts share one data key and base nonce per upload, so rewriting a part number repeats its nonce sequence. Part-keyed segments open with a random salt frame (type 0x03) and seal their v2 frames under a per-write key, but releases without that frame type reject them, and ciphertext travels verbatim through transition, decommission, and SSE-C replication passthrough. Multipart writes therefore use the layout only when RUSTFS_ENCRYPTION_MULTIPART_PART_KEY is set; reads need no switch. Flip the default to enabled (and retire the switch) after the minimum supported direct-upgrade release, and every RustFS warm or replication target that receives raw ciphertext, reads part-keyed segments.
|
||||
- `s3gate-trailer-adapter` aws-chunked trailer handle bridge: s3s decodes aws-chunked request bodies and publishes their x-amz-checksum-* trailers through its own handle, while rio consumes trailers only through its framework-neutral TrailerSource trait. The RustFS application crate adapts the s3s handle at the object write path so rio carries no s3s dependency. Remove the adapter after the gateway stack replaces s3s as the request body decoder and publishes trailers through its own TrailerSource implementation.
|
||||
|
||||
## Review Checklist
|
||||
|
||||
@@ -156,6 +156,8 @@ Two historical shapes additionally reuse a GCM nonce and cannot be repaired by a
|
||||
| Multipart objects written before `1.0.0-alpha.91` | The pre-alpha.91 writer reused a segment's part nonce for every block in it | The whole segment shares one nonce; a frame from index zero authenticates anywhere in that segment | Rewrite in place with CopyObject; then set `RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK=false` |
|
||||
| SSE-C objects written before `1.0.0-beta.9` that carry no stored IV | The nonce was derived deterministically from bucket and key | Historical versions of the same key share a nonce, which affects confidentiality as well as forgeability | Rewrite in place with CopyObject |
|
||||
|
||||
A third shape is still written by default: every part of a multipart upload shares the upload's data key and base nonce, and a part's nonces depend only on its part number, so a part number uploaded again within the same upload with different content reuses that part's nonces. Identical retries produce identical ciphertext. For SSE-C the data key is the customer key itself, so the exposure is not confined to one upload. `RUSTFS_ENCRYPTION_MULTIPART_PART_KEY=true` (default `false`) makes every part write open with a random salt and seal its frames under a key derived from that salt, the part number and the data key; such parts always use the v2 frame layout. Reading them needs no switch and existing parts stay readable, but nodes without part-keyed read support cannot decrypt them. Enable it only after every node — and every RustFS warm or replication target that receives raw ciphertext — runs a release that reads part-keyed segments, the same rollout rule as `RUSTFS_ENCRYPTION_FRAME_V2`. Parts written before the switch keep their stored bytes; rewrite an affected object with CopyObject to re-encrypt it.
|
||||
|
||||
The decrypt reader locks each segment to whichever nonce layout decoded its first non-zero-index frame, so a replayed frame is rejected as soon as any later frame disagrees. A stream that is nothing but repeats of frame zero has no later frame to disagree, so a deployment that holds no pre-alpha.91 objects should set `RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK=false` (default `true`) to drop that layout entirely. Turning it off refuses to decrypt pre-alpha.91 objects, so migrate first.
|
||||
|
||||
## Mixed-version clusters during a rolling upgrade
|
||||
|
||||
@@ -2599,6 +2599,136 @@ mod tests {
|
||||
.await;
|
||||
}
|
||||
|
||||
/// Both Direct-mode session kinds — SSE-C (customer key) and managed SSE
|
||||
/// (session DEK) — resolve one key and base nonce for every part of an
|
||||
/// upload. A part-keyed rewrite of a part number must still seal under a
|
||||
/// fresh key, and the GET-side material must read the rewritten part next
|
||||
/// to a legacy part.
|
||||
#[cfg(not(feature = "rio-v2"))]
|
||||
#[tokio::test]
|
||||
async fn part_keyed_rewrites_round_trip_for_ssec_and_managed_sessions() {
|
||||
use base64_simd::STANDARD as BASE64;
|
||||
use md5::{Digest, Md5};
|
||||
|
||||
let customer_key_bytes = [0x5Eu8; 32];
|
||||
let customer_key = BASE64.encode_to_string(customer_key_bytes);
|
||||
let customer_key_md5 = BASE64.encode_to_string(Md5::digest(customer_key_bytes));
|
||||
let local_sse_master_key = BASE64.encode_to_string([0x24u8; 32]);
|
||||
|
||||
async_with_vars(
|
||||
[
|
||||
("__RUSTFS_SSE_SIMPLE_CMK", None::<String>),
|
||||
("RUSTFS_SSE_S3_MASTER_KEY", Some(local_sse_master_key)),
|
||||
],
|
||||
async {
|
||||
for ssec in [true, false] {
|
||||
let (sse, algorithm, sse_key, sse_key_md5) = if ssec {
|
||||
(
|
||||
None,
|
||||
Some("AES256".to_string()),
|
||||
Some(customer_key.clone()),
|
||||
Some(customer_key_md5.clone()),
|
||||
)
|
||||
} else {
|
||||
(Some(ServerSideEncryption::from_static(ServerSideEncryption::AES256)), None, None, None)
|
||||
};
|
||||
let session_material = sse_prepare_encryption(PrepareEncryptionRequest {
|
||||
bucket: "bucket",
|
||||
key: "object",
|
||||
server_side_encryption: sse,
|
||||
ssekms_key_id: None,
|
||||
ssekms_context: None,
|
||||
sse_customer_algorithm: algorithm,
|
||||
sse_customer_key: sse_key,
|
||||
sse_customer_key_md5: sse_key_md5,
|
||||
principal: None,
|
||||
})
|
||||
.await
|
||||
.expect("prepare multipart encryption")
|
||||
.expect("multipart session material");
|
||||
assert_eq!(session_material.key_kind, EncryptionKeyKind::Direct);
|
||||
let mut session_metadata =
|
||||
encryption_material_to_metadata(&session_material).expect("multipart session metadata");
|
||||
mark_encrypted_multipart_metadata(&mut session_metadata);
|
||||
|
||||
let resolve = || {
|
||||
sse_decryption(DecryptionRequest {
|
||||
bucket: "bucket",
|
||||
key: "object",
|
||||
metadata: &session_metadata,
|
||||
sse_customer_key: ssec.then_some(&customer_key),
|
||||
sse_customer_key_md5: ssec.then_some(&customer_key_md5),
|
||||
principal: None,
|
||||
})
|
||||
};
|
||||
|
||||
// UploadPart resolves the session material per part.
|
||||
let part_material = resolve().await.expect("resolve part material").expect("part material");
|
||||
let encrypt = |plaintext: Vec<u8>, part_number: usize, part_keyed: bool| {
|
||||
let (key, nonce) = (part_material.key_bytes, part_material.base_nonce);
|
||||
async move {
|
||||
let mut out = Vec::new();
|
||||
if part_keyed {
|
||||
EncryptReader::new_multipart_part_keyed(Cursor::new(plaintext), key, nonce, part_number)
|
||||
.read_to_end(&mut out)
|
||||
.await
|
||||
} else {
|
||||
EncryptReader::new_multipart(Cursor::new(plaintext), key, nonce, part_number)
|
||||
.read_to_end(&mut out)
|
||||
.await
|
||||
}
|
||||
.expect("encrypt part");
|
||||
out
|
||||
}
|
||||
};
|
||||
|
||||
let part_one = vec![0x31; rustfs_rio::DEFAULT_ENCRYPTION_BLOCK_SIZE + 23];
|
||||
let first_write = encrypt(vec![0x99; part_one.len()], 1, true).await;
|
||||
let rewrite = encrypt(part_one.clone(), 1, true).await;
|
||||
assert_eq!((first_write[0], rewrite[0]), (0x03, 0x03), "ssec={ssec}: part-keyed segments");
|
||||
assert_ne!(first_write[8..40], rewrite[8..40], "ssec={ssec}: a rewrite draws a fresh part key");
|
||||
|
||||
let part_two = vec![0x32; 777];
|
||||
let legacy_two = encrypt(part_two.clone(), 2, false).await;
|
||||
let parts = vec![
|
||||
ObjectPartInfo {
|
||||
number: 1,
|
||||
size: rewrite.len(),
|
||||
actual_size: part_one.len() as i64,
|
||||
..Default::default()
|
||||
},
|
||||
ObjectPartInfo {
|
||||
number: 2,
|
||||
size: legacy_two.len(),
|
||||
actual_size: part_two.len() as i64,
|
||||
..Default::default()
|
||||
},
|
||||
];
|
||||
|
||||
// GET resolves the stored object's material independently.
|
||||
let read_material = resolve().await.expect("resolve read material").expect("read material");
|
||||
let plaintext_size = multipart_plaintext_size(&parts, -1);
|
||||
let mut decrypted_reader = HardLimitReader::new(
|
||||
boxed_reader(DecryptReader::new_multipart(
|
||||
wrap_reader(Cursor::new([rewrite, legacy_two].concat())),
|
||||
read_material.key_bytes,
|
||||
read_material.base_nonce,
|
||||
multipart_part_numbers(&parts),
|
||||
)),
|
||||
plaintext_size,
|
||||
);
|
||||
let mut decrypted = Vec::new();
|
||||
decrypted_reader
|
||||
.read_to_end(&mut decrypted)
|
||||
.await
|
||||
.expect("read part-keyed and legacy parts");
|
||||
assert_eq!(decrypted, [part_one, part_two].concat(), "ssec={ssec}");
|
||||
}
|
||||
},
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn execute_abort_multipart_upload_returns_internal_error_when_store_uninitialized() {
|
||||
let input = AbortMultipartUploadInput::builder()
|
||||
|
||||
Reference in New Issue
Block a user