rustfs/CLAUDE.md

# RustFS — CLAUDE.md

S3-compatible object store in Rust, derived from MinIO. Erasure-coded, multi-pool, supports ILM tiering/lifecycle.

## Commands

```bash
cargo build --release --bin rustfs        # production binary
cargo build                               # dev build
cargo check -p <crate>                    # fast type-check one crate
cargo test -p <crate>                     # test one crate
cargo fmt --all                           # format (required before PR)
make pre-commit                           # full pre-PR gate (fmt + clippy + test)
make build-docker BUILD_OS=ubuntu22.04    # Docker cross-build
```

> **Docker build note**: `buildx build` without `--load` keeps the image in the buildx cache only — `docker run` will use a stale local image. The Makefile already includes `--load`; if you suspect a stale binary, add `--no-cache` to the `buildx build` invocation inside `.config/make/build-docker.mak`.

> Agent/PR rules: see `.github/copilot-instructions.md`.
> Crate membership: `Cargo.toml` `[workspace].members`.
> CI gates: `.github/workflows/ci.yml`.

## Workspace layout

```
rustfs/src/main.rs                          # binary entry point
crates/ecstore/src/
  set_disk.rs                               # ErasureSet: transition_object, restore_transitioned_object
  store.rs / store_api/                     # ECStore trait + ObjectInfo / TransitionedObject types
  bucket/lifecycle/
    bucket_lifecycle_ops.rs                 # ILM actions: transition_object, expire_transitioned_object,
                                            #   get_transitioned_object_reader, gen_transition_objname
    tier_sweeper.rs                         # background sweep: delete_object_from_remote_tier
  tier/
    warm_backend.rs                         # WarmBackend trait (put/get/remove/in_use)
    warm_backend_s3.rs                      # HTTP-client based (TransitionClient) — used for S3/MinIO
    warm_backend_s3sdk.rs                   # aws-sdk-s3 based — alternative S3 backend
    warm_backend_minio.rs / _rustfs.rs / …  # per-provider wrappers (all delegate to _s3 or _s3sdk)
    tier.rs                                 # TierConfigMgr, new_warm_backend dispatch
  client/transition_api.rs                  # TransitionClient HTTP plumbing; UploadInfo, to_object_info
  client/api_put_object_streaming.rs        # put_object_do → UploadInfo (version_id from x-amz-version-id)
crates/filemeta/src/
  filemeta.rs                               # FileMeta (xl.meta top-level), is_skip_meta_key
  filemeta/version.rs                       # FileMetaVersion, MetaObject, MetaDeleteMarker
                                            #   → to_fileinfo() reads transition_version_id
                                            #   → set_transition() writes raw UUID bytes
                                            #   → From<FileInfo> for MetaObject writes all meta
  fileinfo.rs                               # FileInfo struct (transition_version_id: Option<Uuid>)
  examples/
    dump_fileinfo.rs                        # CLI: parse xl.meta, print transition fields + metadata
    dump_versions.rs                        # CLI: list all versions in xl.meta
crates/utils/src/http/metadata_compat.rs    # SUFFIX_* constants, insert_bytes/get_bytes (dual RustFS+MinIO keys)
```

## Metadata key conventions

Internal metadata is stored under **both** `x-rustfs-internal-<suffix>` and `x-minio-internal-<suffix>` for MinIO interoperability. `get_bytes` prefers the RustFS key with MinIO fallback.

Key suffixes (from `metadata_compat.rs`):
| Suffix | Meaning |
|--------|---------|
| `transition-status` | `"complete"` when tiered |
| `transitioned-object` | tier key path (without prefix) |
| `transitioned-versionID` | S3 version_id returned by tier PUT (16 raw UUID bytes, or absent) |
| `transition-tier` | tier name |
| `tier-free-versionID` | delete-marker version for free-version sweep |

## Tier / ILM transition architecture

### Transition flow (hot → cold)
1. `transition_object` (lifecycle_ops) → `ECStore::transition_object` → `set_disk.rs`
2. `gen_transition_objname(bucket)` → `{sha256_hash[0..16]}/{uuid[0..2]}/{uuid[2..4]}/{uuid}` (unique per object version)
3. `tgt_client.put_with_meta(dest_obj, …)` → returns `rv: String` (remote S3 version_id, or `""`)
4. `fi.transition_version_id = if rv.is_empty() { None } else { Some(Uuid::parse_str(&rv)?) }`
5. `fi.transitioned_objname = dest_obj` (without tier prefix)
6. Written to xl.meta via `MetaObject::from(FileInfo)` → `insert_bytes(SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes())` (16 raw bytes)

### Tier GET flow (restore/read)
`get_transitioned_object_reader` (lifecycle_ops):
- reads `oi.transitioned_object.name` (= `fi.transitioned_objname`)
- reads `oi.transitioned_object.version_id` (= `fi.transition_version_id.to_string()` or `""`)
- calls `warm_backend.get(name, version_id, opts)`
- `warm_backend_s3.rs::get`: adds `?versionId=…` only when `rv != ""`

### Tier prefix handling
`WarmBackendS3::get_dest(object)` prepends `self.prefix` to the object name.
`transitioned_objname` is stored **without** the prefix — `get_dest` adds it on every call.

### xl.meta on disk
Path: `{disk}/{bucket}/{object}/xl.meta` — one per erasure shard disk.
All shards should be identical for a healthy object.

## Known bugs & fixes

### Bug 1: `NoSuchVersion` on tier GET — nil UUID sent as versionId
**Root cause**: `transitioned-versionID` metadata key exists with empty string value (0 bytes). Old reading code:
```rust
// OLD — unwrap_or_default() converts 0-byte or wrong-length slice to Uuid::nil()
get_bytes(…).map(|v| Uuid::from_slice(v.as_slice()).unwrap_or_default())
// → Some(Uuid::nil()) → sends ?versionId=00000000-… → NoSuchVersion
```
**Fix** (version.rs, `MetaObject::to_fileinfo` + `MetaDeleteMarker::to_fileinfo`):
```rust
get_bytes(…)
    .and_then(|v| Uuid::from_slice(v.as_slice()).ok())  // None for wrong-length bytes
    .filter(|u| !u.is_nil())                            // None for nil UUID (old write-back)
```
**Regression tests** (`crates/filemeta/src/filemeta/version.rs` `mod tests`): 6 tests cover absent key, empty bytes, nil UUID, and valid UUID round-trip for both `MetaObject` and `MetaDeleteMarker` paths.

### Bug 2: `warm_backend_s3sdk.rs` ignored rv and range opts
**Fix**: added `req.version_id(rv)` and `req.range(…)` to GET; `req.version_id(rv)` to DELETE.

### Bug 4: `set_disk::copy_object` returns 501 for tiered objects (storage class restore)
**Root cause**: `set_disk::copy_object` immediately returns `StorageError::NotImplemented` when `src_info.metadata_only = false`. For tiered objects, `metadata_only` is never set to `true` (guarded by `transitioned_object.tier.is_empty()`). So `mc cp --storage-class STANDARD obj obj` on a tiered object always returns 501.
**Fix** (`crates/ecstore/src/set_disk.rs`, `copy_object`):
```rust
if !src_info.metadata_only {
    if path_join_buf(&[src_bucket, src_object]) == path_join_buf(&[dst_bucket, dst_object]) {
        if let Some(mut put_reader) = src_info.put_object_reader.take() {
            return self.put_object(dst_bucket, dst_object, &mut put_reader, dst_opts).await;
        }
    }
    return Err(StorageError::NotImplemented);
}
```
When a self-copy has a `put_object_reader` (data already fetched from tier in `execute_copy_object`), writes it back locally via `put_object`, effectively de-tiering the object.
**How `mc cp --storage-class STANDARD` flows**:
1. mc sends `PUT /bucket/key` with `x-amz-copy-source`, `x-amz-metadata-directive: REPLACE`, `x-amz-storage-class: STANDARD`
2. `execute_copy_object` → `get_object_reader` fetches data from tier backend → stores in `src_info.put_object_reader`
3. `store.copy_object(...)` → now calls `put_object` with tier data and STANDARD storage class in `dst_opts`
4. New xl.meta written locally with STANDARD class, no tier metadata → object de-tiered

### Bug 3 (open): race in `expire_transitioned_object`
Order is: delete remote tier version → delete local object.
A concurrent GET between those two steps fetches a valid stored version_id but the tier version is already gone → `NoSuchVersion`.
The proper fix is to delete local metadata first (making the object unreachable) before deleting the remote tier version.

## Debugging tier issues

### Inspect xl.meta directly
```bash
cargo build -p rustfs-filemeta --example dump_fileinfo
./target/debug/examples/dump_fileinfo /srv/rustfs/data/disk0/{bucket}/{object}/xl.meta
# Shows: transition_status, transition_tier, transitioned_obj, transition_ver_id
```
`transition_ver_id: <none>` → no versionId will be sent to tier (correct for non-versioned tier bucket).
`transition_ver_id: <uuid>` → that UUID will be sent as `?versionId=<uuid>`.

### Check what versionId is being sent at runtime
Enable debug logging:
```bash
RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug rustfs …
```
Log line: `fetching transitioned object from tier` (DEBUG before request).
Log line: `tier GET failed` (ERROR on failure, includes `tier_version_id`).

### Metadata key to watch
```
x-minio-internal-transitioned-versionID=   ← empty string = will cause NoSuchVersion with old code
x-rustfs-internal-transitioned-versionID=  ← same
```
If both are empty string, the object was transitioned to a non-versioned tier bucket. The versionId should NOT be sent — fixed by Bug 1 above.

## Common patterns

### Writing internal metadata (binary values)
```rust
insert_bytes(&mut meta_sys, SUFFIX_TRANSITIONED_VERSION_ID, uuid.as_bytes().to_vec());
// stores under both x-rustfs-internal-* and x-minio-internal-* keys
```

### Reading internal metadata (binary values)
```rust
get_bytes(&self.meta_sys, SUFFIX_TRANSITIONED_VERSION_ID)
    .and_then(|v| Uuid::from_slice(v.as_slice()).ok())
    .filter(|u| !u.is_nil())
// Returns None for: absent, wrong-length bytes, nil UUID
```

### WarmBackend trait
```rust
put_with_meta(object, reader, length, meta) -> Result<String>  // returns S3 version_id or ""
put(object, reader, length) -> Result<String>
get(object, rv, opts) -> Result<ReadCloser>  // rv="" means no versionId
remove(object, rv) -> Result<()>
in_use() -> Result<bool>
```
`rv` = remote version, always pass as empty string when `transition_version_id` is None.