mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-24 13:16:28 +00:00
docs(operations): document rebalance impact assessment
This commit is contained in:
Zhengchao An
@@ -0,0 +1,263 @@
|
||||
# Rebalance Stored-Representation Impact Guide
|
||||
|
||||
This guide covers the historical data-movement read defect tracked by
|
||||
[`rustfs/backlog#1850`](https://github.com/rustfs/backlog/issues/1850). It is an
|
||||
impact-assessment and read-only triage guide. It does not repair, rewrite,
|
||||
migrate, delete, or quarantine any object.
|
||||
|
||||
The defect affected data movement when the source reader returned logical
|
||||
plaintext but the target writer preserved the source's stored-representation
|
||||
metadata and sizes. Compressed objects could therefore be copied as plaintext
|
||||
under compression metadata. Server-managed encrypted objects could be copied as
|
||||
plaintext under encryption metadata. The forward rebalance fix reached `main`
|
||||
in commit
|
||||
[`e11fcfbd`](https://github.com/rustfs/rustfs/commit/e11fcfbd087f8a8dae2c0f2c62bc0f6e40e3f10a)
|
||||
through [PR #6057](https://github.com/rustfs/rustfs/pull/6057).
|
||||
|
||||
Upgrading prevents this defect in later rebalance runs. It does not validate or
|
||||
repair copies produced by an earlier run.
|
||||
|
||||
## Immediate Operator Decision
|
||||
|
||||
Treat a deployment as exposed when both conditions are true:
|
||||
|
||||
1. it ran rebalance in an affected build, or decommission in the narrower
|
||||
historical window described below; and
|
||||
2. the operation could have selected compressed, SSE-S3, or SSE-KMS objects.
|
||||
|
||||
For an exposed deployment:
|
||||
|
||||
- preserve old pool media, snapshots, replicas, and backups before any pool is
|
||||
removed, reformatted, reused, or returned;
|
||||
- stop destructive cleanup and do not use another rebalance or decommission run
|
||||
as a repair mechanism;
|
||||
- inventory and validate candidates with read-only operations;
|
||||
- handle SSE-S3 and SSE-KMS candidates as a confidentiality incident as well as
|
||||
a data-integrity incident;
|
||||
- restore only from a separately verified source under an incident-specific
|
||||
recovery plan.
|
||||
|
||||
## Affected Versions
|
||||
|
||||
The release boundaries below were verified by tag ancestry. Commit
|
||||
[`a236b0d0`](https://github.com/rustfs/rustfs/commit/a236b0d01d40a152309446a553756ea991c9f901)
|
||||
introduced the merged rebalance and decommission implementation. Commit
|
||||
[`2f25cf60`](https://github.com/rustfs/rustfs/commit/2f25cf606e5ca814fe992be6327a91e31fe066b3)
|
||||
introduced the raw stored-representation read mode and wired it into
|
||||
decommission. Commit `e11fcfbd` wired the same mode into rebalance.
|
||||
|
||||
| Release or commit range | Rebalance | Decommission | Operator classification |
|
||||
| --- | --- | --- | --- |
|
||||
| Through `1.0.0-alpha.90`, before `a236b0d0` | Path not present | Path not present | Not affected by this data-movement path |
|
||||
| `1.0.0-alpha.91` through `1.0.0-beta.8`, from `a236b0d0` up to but excluding `2f25cf60` | Decoded read | Decoded read | Both operations require assessment |
|
||||
| `1.0.0-beta.9` through `1.0.0-rc.1`, from `2f25cf60` up to but excluding `e11fcfbd` | Decoded read | Raw stored-representation read | Rebalance requires assessment; decommission is not affected by this defect |
|
||||
| `1.0.0-rc.2` and later, at or after `e11fcfbd` | Raw stored-representation read | Raw stored-representation read | Forward-fixed; earlier copies still require assessment |
|
||||
|
||||
Preview tags follow the commit they reference. In particular, the `rc.1`
|
||||
preview is affected and the `rc.2` preview contains the forward fix. For custom
|
||||
or untagged builds, compare the deployed commit with the three commit boundaries
|
||||
rather than inferring behavior from a version string.
|
||||
|
||||
The historical decommission result is narrower than the rebalance result but is
|
||||
not empty. Before `2f25cf60`, decommission used the same ordinary decoded reader.
|
||||
From `1.0.0-beta.9` onward it explicitly used `raw_data_movement_read: true`.
|
||||
Any code change or automated remediation for the earlier decommission window is
|
||||
outside this report and requires a separate issue.
|
||||
|
||||
## Why The Copy Could Be Accepted
|
||||
|
||||
The migration pipeline is a stored-representation copier. It preserves the
|
||||
source ETag and internal metadata, uses stored `part.size` values to divide the
|
||||
stream, and carries the decoded compression index. The affected rebalance read
|
||||
options supplied only the version ID and lock setting, so the normal GET read
|
||||
plan decompressed or decrypted the stream first. A target write could therefore
|
||||
complete while its bytes no longer matched the metadata that described them.
|
||||
|
||||
Historical rebalance cleanup ran only after every version in an entry was
|
||||
reported moved. It then deleted the source entry. A target write accepted as a
|
||||
successful move could therefore be followed by source deletion even though a
|
||||
later GET of the target would fail. Conversely, a source-read failure prevented
|
||||
the version from being counted as moved and prevented normal source cleanup.
|
||||
|
||||
## Object Classification
|
||||
|
||||
| Stored object class | Affected read result | Risk | Triage priority |
|
||||
| --- | --- | --- | --- |
|
||||
| Plain, uncompressed, unencrypted | Stored bytes and logical bytes are the same | No corruption expected from this defect alone | Low; sample to validate the scope assumption |
|
||||
| Compressed | Decompressed bytes were divided using compressed part sizes while compression metadata and indexes were retained | Silent truncation or malformed compressed representation; GET can fail or return truncated data | High |
|
||||
| SSE-S3 | Decrypted plaintext could be written while encryption metadata and ciphertext sizes were retained | Plaintext at rest on the target plus later decrypt failure | Critical |
|
||||
| SSE-KMS | Decrypted plaintext could be written while KMS/encryption metadata and ciphertext sizes were retained | Plaintext at rest on the target plus later decrypt failure | Critical |
|
||||
| SSE-C | The migration request did not have the customer key, so the normal read failed closed | Migration failure and possible incomplete progress; no successful corrupting copy is expected from this path | Medium; confirm the source was retained |
|
||||
| Any compressed and encrypted combination | Multiple stored-representation assumptions were violated | Confidentiality exposure and data corruption | Critical |
|
||||
|
||||
The classification is specific to this defect. A low-risk classification does
|
||||
not certify an object against unrelated corruption.
|
||||
|
||||
## Read-Only Assessment Workflow
|
||||
|
||||
### 1. Establish The Operation Window
|
||||
|
||||
Record the exact RustFS version and commit for every node that participated.
|
||||
Collect the authenticated rebalance status response, decommission status when
|
||||
applicable, service logs, deployment change records, and release history.
|
||||
|
||||
Persisted rebalance metadata records the run ID, participating pools, start and
|
||||
end state, bucket lists, counters, and the last bucket/object progress value. It
|
||||
does not persist a complete per-object movement ledger. Status metadata can
|
||||
prove that a run occurred and narrow time, pool, and bucket scope, but it cannot
|
||||
by itself enumerate every moved object.
|
||||
|
||||
If no reliable operation record remains, assume that every object version in a
|
||||
bucket present during the affected deployment interval is a candidate until
|
||||
other evidence narrows the set.
|
||||
|
||||
### 2. Build A Candidate Inventory
|
||||
|
||||
Use read-only S3 list and list-object-versions operations for the buckets in
|
||||
scope. Preserve bucket, key, version ID, last-modified time, size, ETag, storage
|
||||
class, and any client-side content digest. Join that list with:
|
||||
|
||||
- upload records that identify compression settings or SSE mode;
|
||||
- KMS audit history and application catalogs;
|
||||
- replication inventory and external backup manifests;
|
||||
- rebalance/decommission timestamps and source/target pool records;
|
||||
- server access logs showing successful or failed reads after movement.
|
||||
|
||||
Do not use ETag equality as proof of content integrity. The migration writer
|
||||
preserved the source ETag, including for a malformed target copy, and multipart
|
||||
or encrypted ETags are not general-purpose content hashes.
|
||||
|
||||
### 3. Classify Stored Metadata On Evidence Copies
|
||||
|
||||
When API and application records cannot classify a candidate, copy `xl.meta`
|
||||
from each relevant shard disk to a restricted evidence location and inspect the
|
||||
copy on an offline host. Do not edit or decode metadata in place on a live data
|
||||
path. Keep the evidence copies under the same access controls as the object.
|
||||
|
||||
The existing `rustfs-filemeta` example can decode an evidence copy. It prints
|
||||
metadata values, some of which are sensitive encryption material, so redact
|
||||
metadata values before they reach a terminal or report:
|
||||
|
||||
```bash
|
||||
cargo run --quiet -p rustfs-filemeta --example dump_fileinfo -- /evidence/object/xl.meta |
|
||||
sed -E 's/^(meta\[[^]]+\])=.*/\1=<redacted>/'
|
||||
```
|
||||
|
||||
Use the output only as a screen:
|
||||
|
||||
- either the `x-rustfs-internal-compression` or
|
||||
`x-minio-internal-compression` key marks a compressed representation;
|
||||
- `actual-size`, per-part `size`/`actual_size`, and compression-index totals
|
||||
should be arithmetically consistent;
|
||||
- SSE-C customer-algorithm/MD5 markers identify SSE-C;
|
||||
- KMS key-ID/context markers identify SSE-KMS;
|
||||
- a managed encryption envelope without SSE-C or KMS markers identifies an
|
||||
SSE-S3 candidate.
|
||||
|
||||
Never include encryption metadata values in tickets, logs, chat, or assessment
|
||||
reports. Metadata consistency is necessary but not sufficient: the defect
|
||||
preserved metadata, so plausible sizes and a decodable index do not prove that
|
||||
the stored bytes match it.
|
||||
|
||||
### 4. Validate Logical Content Without Mutation
|
||||
|
||||
For each high- or critical-risk candidate, perform a complete authenticated GET
|
||||
of the exact version into a restricted validation sink. Supply the customer key
|
||||
only for an authorized SSE-C check. Record the status, byte count, and a
|
||||
cryptographic digest calculated by the validation client. Compare it with a
|
||||
digest from an independently trusted source, backup, replica, or application
|
||||
record.
|
||||
|
||||
Interpret the result conservatively:
|
||||
|
||||
- a GET decode/decrypt error, unexpected EOF, or short byte count is a strong
|
||||
affected-copy signal, but may also have another corruption cause;
|
||||
- a matching independent cryptographic digest validates that logical version;
|
||||
- a successful GET without an independent digest proves readability, not
|
||||
identity;
|
||||
- a matching ETag alone is inconclusive;
|
||||
- an SSE-S3/KMS candidate moved in the affected window remains a confidentiality
|
||||
incident until storage-level review excludes plaintext target copies and
|
||||
derivative snapshots or backups.
|
||||
|
||||
Storage-level confirmation for managed-SSE candidates may expose plaintext and
|
||||
sealed-key material. It must be performed only by the incident/security owner on
|
||||
offline evidence copies. Do not print, upload, or serve raw shard bytes, and do
|
||||
not bypass RustFS to return them to an application.
|
||||
|
||||
### 5. Record Confidence And Outcome
|
||||
|
||||
Record one result for every candidate version:
|
||||
|
||||
- `confirmed-good`: full logical bytes match an independent digest;
|
||||
- `confirmed-affected`: target decode/decrypt/length evidence and a trusted
|
||||
source establish the mismatch, or authorized storage review confirms
|
||||
plaintext under managed-SSE metadata;
|
||||
- `suspected`: the version and operation window match, but proof is incomplete;
|
||||
- `not-applicable`: evidence proves the object was plain and uncompressed or was
|
||||
never selected by an affected operation;
|
||||
- `unrecoverable-pending-source`: affected or suspected, with no verified source
|
||||
yet found.
|
||||
|
||||
Retain the evidence used for each decision. Do not collapse object versions with
|
||||
the same key into one result.
|
||||
|
||||
## Source Retention And Recovery Limits
|
||||
|
||||
Successful historical migration could be followed by source-entry deletion.
|
||||
Therefore, neither successful rebalance status nor absence from the old source
|
||||
pool proves that the target bytes are sound. Recovery is possible only from a
|
||||
separately verified source, such as:
|
||||
|
||||
- retained source-pool media or a snapshot taken before cleanup;
|
||||
- an independently validated replica;
|
||||
- an external backup;
|
||||
- the original application or upstream source with a trusted digest.
|
||||
|
||||
SSE-C normally failed before the target copy was accepted because the migration
|
||||
read had no customer key. That failure prevented normal source cleanup, but
|
||||
operators must verify the exact version on retained source media rather than
|
||||
assuming it is present.
|
||||
|
||||
If no verified source exists, mark the version unrecoverable for this incident.
|
||||
Do not edit `xl.meta`, rewrite shard files, clear encryption/compression markers,
|
||||
or overwrite the object in place. Those actions can destroy evidence, violate
|
||||
retention/versioning policy, or turn a visible read failure into silent data
|
||||
substitution. Any restoration or replacement procedure needs its own reviewed,
|
||||
rollback-aware plan.
|
||||
|
||||
## Release Guidance
|
||||
|
||||
Release notes for `1.0.0-rc.2` and later should state:
|
||||
|
||||
> Rebalance now copies the stored object representation for compressed and
|
||||
> encrypted objects. Deployments that ran rebalance on versions from
|
||||
> `1.0.0-alpha.91` through `1.0.0-rc.1` should preserve old pool media and run
|
||||
> the read-only assessment in this guide. Upgrading prevents new copies from
|
||||
> this defect but does not repair historical copies. Deployments that ran
|
||||
> decommission from `1.0.0-alpha.91` through `1.0.0-beta.8` require the same
|
||||
> assessment. SSE-S3 and SSE-KMS candidates require security incident handling.
|
||||
|
||||
Do not recommend rerunning rebalance as remediation. Do not remove or repurpose
|
||||
old pool media until high- and critical-risk candidates have a recorded outcome
|
||||
and the incident owner has accepted the recovery limits.
|
||||
|
||||
## Evidence Audit
|
||||
|
||||
The conclusions above are grounded in these repository facts:
|
||||
|
||||
- `crates/ecstore/src/services/rebalance/migration.rs` now sets both
|
||||
`data_movement` and `raw_data_movement_read` for rebalance source reads;
|
||||
- `crates/ecstore/src/core/pools.rs` sets the same flags for decommission source
|
||||
reads;
|
||||
- `crates/ecstore/src/object_api/readers.rs` returns the stored byte range before
|
||||
compression or encryption transforms when `raw_data_movement_read` is set;
|
||||
- `crates/ecstore/src/data_movement/mod.rs` preserves stored part sizes, ETags,
|
||||
indexes, and internal metadata during migration;
|
||||
- the historical `a236b0d0` rebalance and decommission readers both used normal
|
||||
read options, while `2f25cf60` changed only decommission to the raw mode;
|
||||
- the historical rebalance entry deleted its source prefix only after all
|
||||
versions were counted as moved;
|
||||
- the tag ancestry boundaries are `1.0.0-alpha.91`, `1.0.0-beta.9`, and
|
||||
`1.0.0-rc.2` for the implementation, decommission raw-read fix, and rebalance
|
||||
raw-read fix respectively.
|
||||
Reference in New Issue
Block a user