mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-05 21:07:43 +00:00
cxymds
113 lines
3.5 KiB
Markdown
113 lines
3.5 KiB
Markdown
# Decommission Tier Lifecycle Recovery
|
|
|
|
This manual E2E flow covers the Tier + ILM + Decommission interaction path
|
|
against a disposable local deployment.
|
|
|
|
## Goal
|
|
|
|
Validate that transitioned objects can be expired and recovered safely while
|
|
decommission also handles tiered objects without removing remote tier data
|
|
incorrectly.
|
|
|
|
## What It Covers
|
|
|
|
- RustFS tier configuration backed by a disposable MinIO service.
|
|
- Immediate transition of two object prefixes to the configured tier.
|
|
- Lifecycle expiry of transitioned objects with RustFS restart recovery.
|
|
- Concurrent GET traffic while expiry and cleanup are running.
|
|
- Decommission of a pool containing transitioned objects.
|
|
- Remote tier count checks before and after decommission.
|
|
|
|
## What It Does Not Cover Automatically
|
|
|
|
- True distributed node failure injection.
|
|
- Long-running high-cardinality stress with 10,000+ objects.
|
|
- Replication target side-effects.
|
|
- All lifecycle worker pause/failure modes.
|
|
|
|
Those scenarios should be added as nightly or cluster-specific E2E coverage
|
|
after this manual flow is stable.
|
|
|
|
## Prerequisites
|
|
|
|
- Docker with Compose support.
|
|
- `mc` configured or available on `PATH`.
|
|
- `awscurl` available on `PATH`.
|
|
- The local RustFS image used by `docker-compose.decommission.yml`.
|
|
|
|
If the RustFS image is not present, build it first:
|
|
|
|
```bash
|
|
./scripts/test/decommission_docker.sh build
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
Run the full flow:
|
|
|
|
```bash
|
|
./scripts/test/decommission_tier_lifecycle_recovery.sh run
|
|
```
|
|
|
|
Clean up all generated containers, data, logs, and state:
|
|
|
|
```bash
|
|
./scripts/test/decommission_tier_lifecycle_recovery.sh reset
|
|
```
|
|
|
|
## Manual Flow
|
|
|
|
```bash
|
|
./scripts/test/decommission_tier_lifecycle_recovery.sh up
|
|
./scripts/test/decommission_tier_lifecycle_recovery.sh prepare
|
|
./scripts/test/decommission_tier_lifecycle_recovery.sh wait-transition
|
|
./scripts/test/decommission_tier_lifecycle_recovery.sh expire-recovery
|
|
./scripts/test/decommission_tier_lifecycle_recovery.sh decommission
|
|
./scripts/test/decommission_tier_lifecycle_recovery.sh verify
|
|
```
|
|
|
|
## Useful Overrides
|
|
|
|
- `OBJECT_COUNT=100` increases objects per prefix.
|
|
- `TRANSITION_DAYS=0` keeps transition immediate.
|
|
- `EXPIRY_DAYS=0` keeps expiry immediate for local validation.
|
|
- `WAIT_TIMEOUT_SECONDS=1800` gives slow machines more time.
|
|
- `GET_WORKERS=32` increases concurrent GET pressure.
|
|
- `GET_LOAD_SECONDS=300` keeps GET traffic running longer.
|
|
|
|
Example:
|
|
|
|
```bash
|
|
OBJECT_COUNT=100 GET_WORKERS=32 WAIT_TIMEOUT_SECONDS=1800 \
|
|
./scripts/test/decommission_tier_lifecycle_recovery.sh run
|
|
```
|
|
|
|
## Expected Output
|
|
|
|
The flow should end with:
|
|
|
|
- transitioned `expire/` objects no longer readable as current objects
|
|
- remote tier object count drained by at least the number of expired objects
|
|
- concurrent GET log without `NoSuchVersion` or invalid version-state errors
|
|
- transitioned `decom/` objects still readable after decommission
|
|
- remote tier object count not lower after decommission
|
|
|
|
The script stores evidence under `.tmp/decommission-tier-lifecycle/work`,
|
|
including scanner status samples, decommission status output, and concurrent
|
|
GET stderr.
|
|
|
|
Missed lifecycle or journal-style counters may increase during restart or queue
|
|
pressure, but recovery should leave no residual cleanup work for the expired
|
|
objects.
|
|
|
|
## Nightly Follow-Up
|
|
|
|
Before wiring this into nightly CI, run it repeatedly with larger `OBJECT_COUNT`
|
|
values and collect:
|
|
|
|
- scanner status samples
|
|
- tier backend object counts before and after expiry
|
|
- GET error logs
|
|
- decommission status JSON
|
|
- total runtime and flake rate
|