Commit Graph

3389 Commits

Author SHA1 Message Date
Henry Guo 307d788da1 feat(table-catalog): add production iceberg surfaces (#3473) 2026-06-15 16:53:14 +08:00
Henry Guo d28aedfbb3 feat(scanner): expose replication repair kind metrics (#3476) 2026-06-15 16:52:39 +08:00
安正超 5063524b36 ci: guard migration loss-prevention coverage (#3475) 2026-06-15 16:32:41 +08:00
Henry Guo dd6b4c35ad fix(scanner): harden lifecycle and tiering backlog (#3469)
* fix(scanner): expose lifecycle transition backlog

* fix(scanner): expose lifecycle expiry backlog

* fix(scanner): preserve lifecycle backlog states

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-15 15:01:21 +08:00
安正超 57260c8314 refactor: centralize startup command bootstrap (#3471) 2026-06-15 14:55:33 +08:00
Henry Guo 941b8afee8 feat(table-catalog): add snapshot maintenance foundation (#3470)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-15 14:08:28 +08:00
houseme 73a48c06e5 refactor(logging): align API response boundary events (#3466)
* refactor(logging): align api response boundary events

* refactor(logging): align heal admin boundary events

* refactor(logging): restore admin callsite line numbers

* refactor(logging): simplify audit target error logging

* refactor(logging): restore rpc callsite line numbers
2026-06-15 14:07:54 +08:00
安正超 218f473fb5 refactor: centralize startup lifecycle runtime (#3468) 2026-06-15 14:07:24 +08:00
安正超 e26eea7f36 refactor: centralize startup runtime services (#3467) 2026-06-15 13:24:17 +08:00
安正超 3e723a2476 refactor: centralize startup storage runtime (#3465) 2026-06-15 12:12:27 +08:00
Henry Guo 46fb4bdc2f feat(scanner): add source-aware maintenance controls (#3461)
* feat(scanner): add source-aware maintenance controls

* fix(scanner): preserve source control between cycles

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-15 12:12:05 +08:00
Henry Guo f68967de7a feat(table-catalog): add maintenance reachability reports (#3462)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-15 12:11:53 +08:00
dependabot[bot] fbf1f9e5b8 build(deps): bump brotli from 8.0.3 to 8.0.4 in the dependencies group (#3458)
* build(deps): bump brotli from 8.0.3 to 8.0.4 in the dependencies group

Bumps the dependencies group with 1 update: [brotli](https://github.com/dropbox/rust-brotli).


Updates `brotli` from 8.0.3 to 8.0.4
- [Release notes](https://github.com/dropbox/rust-brotli/releases)
- [Commits](https://github.com/dropbox/rust-brotli/compare/8.0.3...8.0.4)

---
updated-dependencies:
- dependency-name: brotli
  dependency-version: 8.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump s3s from cf4c3346 to c59b62f7ba in the dependencies group

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-15 12:06:01 +08:00
abdullahnah92 a19560da41 fix(bucket-repl): persist MRF retry queue to disk and reload on startup (#3456)
* fix(bucket-repl): persist MRF retry queue to disk and reload on startup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(bucket-repl): address three blocking MRF issues from review

1. MRF replay loses delete operations — add `MrfOpKind` discriminator to
   `MrfReplicateEntry` (Object | Delete, default=Object for backward
   compat).  `DeletedObjectReplicationInfo::to_mrf_entry` now persists
   `op=Delete`, `version_id`, `delete_marker_version_id`, and
   `delete_marker`.  `start_mrf_processor` branches on `op`: delete
   entries skip `get_object_info` and replay via
   `schedule_replication_delete` with `ReplicationType::Heal`; object
   entries follow the existing heal path.

2. `flush_mrf_to_disk` cleared the in-memory batch even on encode/write
   failure — changed return type to `bool` and callers now only
   `pending.clear()` on `true`, so a transient storage error retries
   on the next tick instead of silently dropping the batch.

3. Add focused tests: encode/decode roundtrips for object, delete-marker,
   versioned-delete, and mixed-batch entries; a routing test confirming
   op-kind propagates correctly and that the default is Object for
   legacy files; a legacy-compat test verifying old entries round-trip
   cleanly through the new format.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style: fix clippy redundant-clone in MRF tests

Replace &[entry.clone()] with std::slice::from_ref(&entry) in two
encode_mrf_file call sites flagged by clippy's redundant_clone lint
under --all-targets --features rio-v2.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(bucket-repl): strengthen legacy MRF compat test with hand-built msgpack

The previous mrf_legacy_file_without_op_field_decoded_as_object test
round-tripped through encode_mrf_file, so it exercised the new format
and never touched a truly-legacy payload.

Replace it with a hand-built msgpack payload that genuinely omits the
"op", "deleteMarker", and "deleteMarkerVersionID" keys — exactly what
the old binary would have written before MrfOpKind existed.  The test
now fails if #[serde(default)] is removed from the op field, which
proves real backward compatibility rather than round-trip stability.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-15 11:56:50 +08:00
安正超 c99d6086cd refactor: centralize startup storage bootstrap (#3464) 2026-06-15 11:23:16 +08:00
安正超 6508f88d3a refactor: centralize startup listen bootstrap (#3460) 2026-06-15 09:41:11 +08:00
abdullahnah92 c2e0792f6f fix(scanner): skip startup delay when replication is active so failed… (#3455)
fix(scanner): skip startup delay when replication is active so failed objects heal promptly after restart

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-15 09:02:21 +08:00
安正超 513f06f268 refactor: centralize startup server preflight (#3459) 2026-06-15 08:32:44 +08:00
安正超 2288b1bb63 refactor: centralize startup runtime bootstrap (#3457) 2026-06-15 07:57:36 +08:00
Henry Guo 73e534682e feat(table-catalog): add catalog recovery diagnostics (#3444)
* feat(table-catalog): add commit recovery observability

* feat(table-catalog): harden catalog recovery diagnostics

* test(table-catalog): cover recovery admin route policy

* test(table-catalog): update recovery route count

* fix(table-catalog): satisfy duration clippy lint

* fix(obs): promote span request id field

* Update local.rs

Signed-off-by: houseme <housemecn@gmail.com>

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-15 07:06:29 +08:00
Henry Guo 2c5615d2ea feat(scanner): expose distributed metrics (#3452)
* feat(scanner): expose distributed metrics

* docs(scanner): clarify distributed metrics collection

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-15 07:05:43 +08:00
Ramakrishna Chilaka e4c1ee9941 fix(server): actionable 501 for virtual-hosted-style S3 requests (#3442) 2026-06-15 06:49:31 +08:00
安正超 f8e300bf3b refactor: centralize startup protocol bootstrap (#3450) 2026-06-15 06:48:55 +08:00
houseme 036741cb1c refactor(request-id): align contracts and lock field names (#3454)
* refactor(request-id): align header log and trace contracts

* test(contract): lock external request-id field names

* chore(deps): drop unused rustfs-ecstore links

Co-Authored-By: heihutu <heihutu@gmail.com>
2026-06-15 01:59:11 +08:00
houseme efaf07d323 feat: preserve request ids across async recovery logs (#3451)
* feat(obs): promote request ids in structured logs

* refactor(tracing): propagate spans into request tasks

* test(ecstore): baseline recovery monitor log chains

* fix(replication): reduce startup resync log noise

* chore(docs): stop tracking local recovery baseline

* chore(obs): polish request id logging cleanup
2026-06-14 23:16:04 +08:00
安正超 9499391370 refactor: centralize startup service bootstrap (#3448) 2026-06-14 21:59:17 +08:00
安正超 b2e6cc520b docs: update security advisory lessons (#3447) 2026-06-14 21:51:54 +08:00
安正超 b49057c8e3 refactor: centralize startup readiness bootstrap (#3446) 2026-06-14 21:24:07 +08:00
安正超 8d23ce06c6 refactor: consolidate app object store fallback (#3445) 2026-06-14 20:43:42 +08:00
houseme fc894c9b50 fix(obs): harden startup and shutdown logging (#3443) 2026-06-14 20:16:00 +08:00
安正超 323302255c refactor: route ecstore internals through object resolver (#3441) 2026-06-14 19:55:21 +08:00
cxymds bf3a3a6189 fix(rebalance): ignore source cleanup failures (#3440) 2026-06-14 19:50:26 +08:00
cxymds 8f6b1d47b5 chore(obs): standardize runtime logging batch one (#3438) 2026-06-14 18:55:14 +08:00
安正超 bed4c7288b refactor: route server storage lookups through resolver (#3439) 2026-06-14 18:42:21 +08:00
Henry Guo 9372ee7032 feat(table-catalog): bridge table data-plane policy (#3436)
* feat(table-catalog): bridge table data-plane policy

* test(table-catalog): harden vended credential smoke

* test(table-catalog): cover data-plane policy denials

* fix(table-catalog): protect relocated warehouse scope

* fix(table-catalog): skip invalid warehouse entries

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-14 18:13:02 +08:00
GatewayJ e29c136ad5 fix(table-catalog): stabilize standard commit metadata names (#3435) 2026-06-14 18:03:42 +08:00
安正超 84f7027e90 refactor: add object store resolver for standalone crates (#3437) 2026-06-14 18:02:39 +08:00
安正超 59f99a30e2 refactor: prefer app context object store in admin zip (#3434) 2026-06-14 16:37:08 +08:00
GatewayJ 3928117c8f fix(policy): preserve IAM policy readback shape (#3431) 2026-06-14 16:36:28 +08:00
Henry Guo fc17e75fb2 test(table-catalog): verify vended credential data-plane scope (#3429) 2026-06-14 16:35:54 +08:00
安正超 5e68cf8a29 refactor: prefer app context object store in ecfs (#3433) 2026-06-14 15:52:26 +08:00
cxymds 046d5386ba feat: stream object zip downloads (#3380)
* feat: stream object zip downloads

* fix: stream zip downloads page by page

* fix: prepare zip downloads before streaming

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-14 14:26:44 +08:00
安正超 bdeee173c8 refactor: prefer app context object store in admin (#3432)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-14 13:49:03 +08:00
houseme e8012bd1ba refactor(logging): normalize admin telemetry and error messages (#3430) 2026-06-14 13:27:10 +08:00
Henry Guo dc82efbab4 test(scanner): add validation harness (#3428)
* test(scanner): add validation harness

* fix(scanner): harden validation harness

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-14 12:40:05 +08:00
houseme 61a5500232 chore(deps): update flake.lock (#3423)
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/cbb5cf3' (2026-06-06)
  → 'github:NixOS/nixpkgs/49a4bd0' (2026-06-12)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/27b7e78' (2026-06-06)
  → 'github:oxalica/rust-overlay/5b929d8' (2026-06-13)
2026-06-14 11:46:45 +08:00
安正超 6fcd62ba56 refactor: prefer app context object store in usecases (#3425) 2026-06-14 09:38:12 +08:00
安正超 bed875d3e0 fix(logging): remove guardrail script trailing whitespace (#3424) 2026-06-14 08:35:59 +08:00
houseme efa89a98ed refactor(logging): standardize protocol and observability events (#3419)
* refactor(logging): standardize object capacity events

* refactor(logging): standardize protocol server events

* refactor(logging): standardize swift protocol events

* refactor(logging): standardize observability events

* refactor(logging): move masking helper and extend guardrails
2026-06-14 07:14:45 +08:00
abdullahnah92 22460243bf fix(bucket-repl): honor op_type in replicate_object so ExistingObject… (#3420)
fix(bucket-repl): honor op_type in replicate_object so ExistingObject resync respects DISABLED targets

replicate_object was calling filter_target_arns with hard-coded
op_type: Object and existing_object: false regardless of what was
stored in roi.op_type. This meant that a resync worker setting
roi.op_type = ExistingObject (resync_bucket, line 889) had no effect
on target filtering: all configured targets were included, even ones
whose rule had ExistingObjectReplicationStatus::DISABLED.

Fix: pass op_type: roi.op_type and derive existing_object from it
(true only for ExistingObject, not Heal — Heal intentionally bypasses
the existing-object opt-out to repair past failures).

Also add warn! logs at all four MRF channel-overflow sites that were
previously silently returning Missed with no observability.

Verified with a live two-instance test: after resync, objects reached
the ENABLED target and were correctly blocked from the DISABLED target.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-14 07:14:11 +08:00