gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 08:18:18 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,225 Commits 23 Branches 117 Tags
f7724d223bec3dd27c1fc615ef0874cabbc0905e
Commit Graph

98 Commits

Author SHA1 Message Date
唐小鸭 f7724d223b feat(rio): rio_v2 is compatible with minio for storing data. (#3115) 
* Set up a compatibility layer for replacing old Rio components with new ones.

* fix(rio). compress range

* feat(rio). Add the experimental feature rio_v2 to support minio data at the binary level.

* feat(rio_v2): add sse-c test

* test compression component

* simple fix

* fix minlz encode

* fix metadata

* fix kms key cache error

* Update launch.json

* ci: set nix crate download user agent

* fix: gate obs pyroscope backend

* ignore minio test

* fix encrypt check

* fix

* fix

* fix

* Update object_usecase.rs

* Update ci.yml

* fix

* ci add rio-v2 test

* fix

* ci fix

* fix

* Reconstructed into a more reasonable compatibility mode

* fix

* fix

---------

Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2026-06-08 11:59:14 +00:00
Marcelo Bartsch f00898d070 fix(tier): recover (#3182) 
* fix(tier): stop sending nil/garbage versionId to warm backend S3

Three bugs caused NoSuchVersion errors when reading tiered objects:

1. warm_backend_s3sdk: GET and DELETE ignored rv/range opts entirely —
   fixed to forward version_id and byte-range to the SDK request.

2. version.rs (MetaObject + MetaDeleteMarker): transition_version_id was
   parsed with unwrap_or_default(), turning invalid/wrong-length bytes
   into Uuid::nil(). The nil UUID was then serialized and sent as
   ?versionId=00000000-... to the tier backend -> NoSuchVersion.
   Fixed: .and_then(.ok()).filter(!is_nil()) so only valid non-nil UUIDs
   are forwarded as versionId.

3. bucket_lifecycle_ops: add debug/error logs in
   get_transitioned_object_reader to record tier, tier_object, and
   tier_version_id before and on failure of the tier GET.

Also adds tier transition fields to dump_fileinfo example for offline
xl.meta inspection, and fixes Docker build (cargo path + entrypoint).
Adds CLAUDE.md with tier architecture and debugging notes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* more fixes for versionId

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Marcelo Bartsch <marcelo@bartsch.cl>

* remove branch

* Add tests and fix cargo path, add load to build-docker

* update documentation (CLAUDE.md)

* more fixes for recover

* More fixes to ILM recover

* final fix

* chore: add missing-shard first-scene diagnostics (#3213)

chore(ecstore): add missing-shard first-scene diagnostics

Log rename_data quorum context behind RUSTFS_ISSUE3031_DIAG_ENABLE so partial-disk success can be correlated with later missing shard reads.

Also log put_object commit success and tmp cleanup boundaries to capture when successful quorum writes are followed by tmp_dir cleanup.

* fix test anmd fmt

* fix cargo path
fix test

* fix(tier): format copy_object self-copy guard

---------

Signed-off-by: Marcelo Bartsch <marcelo@bartsch.cl>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
 2026-06-07 09:38:28 +00:00
houseme 25c6bdf490 perf(filemeta): phase-1~3 rename_data metadata optimization (#3011) 
* chore(perf): harden amd64 profiling benchmark flow

* fix(profiling): isolate bench buckets and map protobuf conflict

* perf: avoid blocking owned local writes

* style: format profile admin handler

* docs: clarify observability trace validation

* perf: reduce mkdir overhead on local writes

* perf: add rename_data meta microbenchmark

* perf(filemeta): fast-path data_dir decode in version meta

* perf(filemeta): collapse data-dir lookup into one scan

* perf(filemeta): reduce scan allocs and refresh meta bench

* perf(ecstore): skip mkdir path on read-only open

* perf(filemeta): single-pass unshared data-dir scan

* perf(filemeta): add two-key inline remove fast path

* perf(filemeta): compare remove-two keys by bytes first

* bench(ecstore): add remove_two-only micro benchmark

* bench(ecstore): stabilize rename_data meta benchmark timing

* bench(ecstore): align rename_data path with remove_two

* perf(filemeta): avoid uuid string alloc in remove_two

* perf(filemeta): add fast-path for empty inline data

* perf(filemeta): streamline add_version match branch

* perf(filemeta): fast-return remove_key on miss

* perf(filemeta): speed up add_version insertion lookup

* style(ecstore): normalize formatting in perf-tuning files

* refactor(filemeta): unify inline data removal paths
 2026-05-19 10:20:24 +00:00
安正超 50ddec3ffc fix: preserve data on self metadata copy (#2819) 
Co-authored-by: cxymds <Cxymds@qq.com>
 2026-05-06 06:28:36 +00:00
安正超 f07fed0c49 test(filemeta): cover no-wait refresh coalescing (#2755) 2026-05-01 02:48:30 +00:00
安正超 87e1c7aeb6 test(filemeta): cover future cache timestamp refresh (#2762) 2026-05-01 02:48:08 +00:00
安正超 ef5ccc232a test(filemeta): cover shared cache reuse (#2754) 2026-04-30 12:14:15 +00:00
GatewayJ c29c8a5a1e fix(filemeta): harden and optimize metacache path (#2724) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-04-30 07:47:31 +00:00
唐小鸭 fb0d096d5d fix(sse). Resolving Nonce Overwriting Issues in Multi-Package Scenarios (#2582) 
Signed-off-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 2026-04-18 14:00:14 +00:00
houseme 478720d2ee Centralize lifecycle state updates and fix systemd running status (#2567) 2026-04-17 03:20:11 +00:00
houseme 6ce24f3b63 feat(lifecycle): improve ILM compatibility and scanner runtime config (#2534) 
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
 2026-04-16 10:28:03 +00:00
Tunglies 579b124726 lint: clippy rules or_fun_call (#2561) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-04-16 02:48:39 +00:00
weisd c0d3f53f7a test(filemeta): flush xl meta test fixture before reopen (#2557) 2026-04-16 00:40:46 +00:00
Tunglies f57dd5a3c7 chore(lint): clippy rules needless_collect (#2522) 2026-04-15 10:00:03 +00:00
安正超 1979fc7fb1 fix: restore bitrot stream and checksum metadata (#2542) 2026-04-15 08:36:55 +08:00
安正超 68d3dba9fc fix: revert standalone #2351 artifacts (phase 5) (#2535) 2026-04-14 22:35:05 +08:00
安正超 80cb6b3939 test(filemeta): cover legacy nil UUID metadata (#2464) 2026-04-10 10:26:23 +08:00
weisd 30c8bead63 fix(filemeta): accept nil legacy pool metadata (#2459) 2026-04-10 08:27:41 +08:00
houseme 32bf8f5bf3 feat(storage): add direct chunk GET fast path (#2351) 
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
 2026-04-07 08:33:46 +08:00
houseme d2901fd78c feat(admin): add audit target APIs and harden target source handling (#2350) 
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2026-04-04 09:07:22 +08:00
安正超 bd36cf3588 test(filemeta): cover legacy delete marker decoding (#2333) 
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-03-31 22:08:45 +08:00
安正超 7fb405526b test(filemeta): cover legacy delete marker fallback (#2322) 2026-03-29 20:05:48 +08:00
weisd 5e21c398f5 fix(filemeta): support legacy xl.meta compatibility (#2304) 2026-03-27 13:42:06 +08:00
安正超 a236b0d01d feat(ecstore): implement decommission and rebalance (#2281) 
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-03-26 11:44:02 +08:00
heihutu 3c28f0a0ba feat(metrics): migrate system monitoring from rustfs-obs to rustfs-metrics (#2242) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-03-20 18:52:33 +08:00
weisd 35e1f28f23 fix: support legacy bucket metadata decoding (#2227) 2026-03-19 12:43:53 +08:00
weisd b9b7d86ae4 feat: improve legacy metadata and admin compatibility (#2202) 2026-03-18 21:05:09 +08:00
安正超 b48f273c7d refactor(filemeta): split filemeta into focused submodules (#1946) 2026-02-25 08:38:32 +08:00
LeonWang0735 c7211c9df7 fix:correctly handle replicate delete (#1850) 
Co-authored-by: loverustfs <hello@rustfs.com>
 2026-02-19 13:23:29 +08:00
evan slack 229f0f89c8 perf(read): Remove unecessary allocation in read_xl_meta_no_data (#1846) 
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
 2026-02-16 19:08:12 +08:00
weisd 4d19b069c3 fix: replication delete (#1714) 2026-02-04 13:39:35 +08:00
likewu 087f58b7c8 fix(lifecycle): lifecycle fixes (#1625) 
Signed-off-by: likewu <likewu@126.com>
Co-authored-by: loverustfs <hello@rustfs.com>
 2026-01-31 19:14:47 +08:00
weisd dce117840c refactor: NamespaceLock (nslock), AHM→Heal Crate, and Lock/Clippy Fixes (#1664) 
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: weisd <2057561+weisd@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-01-30 13:13:41 +08:00
houseme e5284a85ed fix: Fixed detection warnings in rust v1.93.0 (#1591) 2026-01-23 13:09:41 +08:00
weisd 6631407416 feat: Add RustFS Scanner Module and Multiple Bug Fixes (#1579) 2026-01-22 13:39:38 +08:00
weisd 359c9d2d26 Enhance Object Version Management and Replication Status Handling (#1413) 2026-01-07 10:44:35 +08:00
houseme 8d7cd4cb1b chore: upgrade dependencies and migrate to aws-lc-rs (#1333) 2026-01-02 00:02:34 +08:00
0xdx2 b19e8070a2 fix(tagging): fix e2e test_object_tagging failure (#1327) 2026-01-01 17:38:37 +08:00
houseme eb33e82b56 fix: Prevent panic in GetMetrics gRPC handler on invalid input (#1291) 
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2025-12-29 03:10:23 +08:00
houseme 7e75c9b1f5 remove unlinked file (#1258) 2025-12-24 23:37:43 +08:00
sunfkny e8fe9731fd Fix memory leak in Cache update method (#1143) 2025-12-15 10:04:14 +08:00
yihong 834025d9e3 docs: fix some dead link (#1053) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-12-08 11:23:24 +08:00
weisd 030d3c9426 fix filemeta nil versionid (#1002) 2025-12-05 20:30:08 +08:00
weisd fce4e64da4 Fix/objectdelete (#917) 
* fix getobject content length resp

* fix delete object

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2025-11-24 16:35:51 +08:00
shiro.lee 9b9bbb662b fix: removing the Limit on the Number of Object Versions (#819) (#892) 
removing the Limit on the Number of Object Versions (#819)
 2025-11-19 22:34:26 +08:00
weisd 85bc0ce2d5 fix: filemeta version handling and delete operations (#879) 
* fix filemeta version

* fix clippy

* fix delete version

* fix clippy/test
 2025-11-18 09:24:22 +08:00
Copilot 601f3456bc Fix large file upload freeze with adaptive buffer sizing (#869) 
* Initial plan

* Fix large file upload freeze by increasing StreamReader buffer size

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add comprehensive documentation for large file upload freeze fix

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade s3s version

* Fix compilation error: use BufReader instead of non-existent StreamReader::with_capacity

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Update documentation with correct BufReader implementation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add tokio feature `io-util`

* Implement adaptive buffer sizing based on file size

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Constants are managed uniformly and fmt code

* fix

* Fix: Trigger self-heal on read when shards missing from rejoined nodes (#871)

* Initial plan

* Fix: Trigger self-heal when missing shards detected during read

- Added proactive heal detection in get_object_with_fileinfo
- When reading an object, now checks if any shards are missing even if read succeeds
- Sends low-priority heal request to reconstruct missing shards on rejoined nodes
- This fixes the issue where data written during node outage is not healed when node rejoins

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* Unify CRC implementations to crc-fast (#873)

* Initial plan

* Replace CRC libraries with unified crc-fast implementation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fix: replace low to Normal

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-11-17 23:15:20 +08:00
houseme d934e3905b Refactor telemetry initialization for non-production environments (#789) 
* add dep `scopeguard`

* improve for tracing

* fix

* fix

* improve code for import

* add logger trace id

* fix

* fix

* fix

* fix

* fix
 2025-11-05 00:55:08 +08:00
安正超 dd47fcf2a8 fix: restore localized samples in tests (#749) 
* fix: restore required localized examples

* style: fix formatting issues
 2025-10-29 13:16:31 +08:00
likewu 2aca1f77af Fix/ilm (#721) 
* fix tip remote tier error
* fix transitioned_object
* fix filemeta
* add GCS R2
* add aliyun tencent huaweicloud azure gcs r2 backend tier
* fix signer
* change azure to s3
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-10-27 20:23:50 +08:00