gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-05 12:57:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,051 Commits 28 Branches 119 Tags
fe4fabb195c0fe90f4fc0ca04ca564bb5ffb837f
Commit Graph

138 Commits

Author SHA1 Message Date
houseme 9e2fa148ee Fix type errors in ecfs.rs and apply clippy fixes for Rust 1.92.0 (#1121) 2025-12-12 00:49:21 +08:00
houseme 1a4e95e940 chore: remove unused dependencies to optimize build (#1117) 2025-12-11 18:13:26 +08:00
dependabot[bot] 0da943a6a4 build(deps): bump s3s from 0.12.0-rc.4 to 0.12.0-rc.5 in the s3s group (#1046) 
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2025-12-11 15:20:36 +08:00
guojidan fba201df3d fix: harden data usage aggregation and cache handling (#1102) 
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: loverustfs <hello@rustfs.com>
 2025-12-11 09:55:25 +08:00
yxrxy ccbab3232b fix: ListObjectsV2 correctly handles repeated folder names in prefixes (#1104) 
Co-authored-by: loverustfs <hello@rustfs.com>
 2025-12-11 09:38:52 +08:00
tennisleng 978845b555 fix(lifecycle): Fix ObjectInfo fields and mod_time error handling (#1088) 
Co-authored-by: loverustfs <hello@rustfs.com>
 2025-12-11 07:17:35 +08:00
Jacob 53c126d678 fix: decode percent-encoded paths in get_file_path() (#1072) 
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
 2025-12-10 22:30:02 +08:00
Jitter 76d25d9a20 Fix/issue #1001 dead node detection (#1054) 
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: Jitterx69 <mohit@example.com>
 2025-12-08 12:29:46 +08:00
yihong 834025d9e3 docs: fix some dead link (#1053) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-12-08 11:23:24 +08:00
0xdx2 7c6cbaf837 feat: enhance error handling and add precondition checks for object o… (#1008) 2025-12-06 20:39:03 +08:00
weisd 030d3c9426 fix filemeta nil versionid (#1002) 2025-12-05 20:30:08 +08:00
weisd b3c80ae362 fix: listdir rpc (#979) 
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
 2025-12-04 16:12:10 +08:00
houseme 5b0a3a0764 upgrade crate version and improve heal config (#963) 2025-12-03 18:49:11 +08:00
weisd a8b7b28fd0 Fix Admin Heal API and Add Pagination Support for Large Buckets (#933) 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-12-03 18:10:46 +08:00
weisd 4d7bf98c82 add logs (#962) 2025-12-03 13:17:47 +08:00
唐小鸭 701960dd81 fix out of range for slice (#931) 2025-11-27 15:57:38 +08:00
Shyim ee04cc77a0 remove debug (#912) 
* remove debug

* Refactor get_global_encryption_service function

* Refactor get_global_encryption_service function

---------

Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-11-26 11:56:01 +08:00
Copilot 277d80de13 Fix: Implement priority-based heal queue with comprehensive diagnostic logging (#884) 
* Initial plan

* Implement priority-based heal queue with deduplication

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Apply cargo fmt formatting fixes

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add queue monitoring, better error handling, and adaptive processing

- Add priority-based queue statistics tracking
- Implement queue capacity warnings (>80% full)
- Process multiple tasks per cycle when capacity allows
- Add proper error logging for failed heal request submissions
- Add Hash trait to HealPriority for HashMap support
- Improve observability with detailed queue status logs

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add comprehensive diagnostic logging to heal flow

- Add detailed logging for shard availability checks during reads
- Log successful heal request submissions
- Add detailed logging in heal_object for disk checking
- Log each disk's heal decision with reason and endpoint
- Add logging for heal writing and renaming phases
- Improve variable naming (nil_count -> available_shards)
- Add success confirmation messages for renamed healed data

These logs will help diagnose where the heal flow is failing.

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* feat(heal): enhance robustness, safety, and observability of heal process

- **Logging & Observability**:
  - Upgrade critical heal logs from [info](/crates/e2e_test/src/reliant/node_interact_test.rs:196:0-213:1) to `warn` for better visibility.
  - Implement structured logging with `tracing` fields for machine readability.
  - Add `#[tracing::instrument]` to [HealTask](c/crates/ahm/src/heal/task.rs:182:0-205:1) and [SetDisks](/crates/ecstore/src/set_disk.rs:120:0-131:1) methods for automatic context propagation.

- **Robustness**:
  - Add exponential backoff retry (3 attempts) for acquiring write locks in [heal_object](/crates/ahm/src/heal/storage.rs:438:4-460:5) to handle contention.
  - Handle [rename_data](/crates/ecstore/src/set_disk.rs:392:4-516:5) failures gracefully by preserving temporary files instead of forcing deletion, preventing potential data loss.

- **Data Safety**:
  - Fix [object_exists](/crates/ahm/src/heal/storage.rs:395:4-412:5) to propagate IO errors instead of treating them as "object not found".
  - Update [ErasureSetHealer](/crates/ahm/src/heal/erasure_healer.rs:28:0-33:1) to mark objects as failed rather than skipped when existence checks error, ensuring they are tracked for retry.

* fix

* fmt

* improve code for heal_object

* fix

* fix

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-11-20 00:36:25 +08:00
weisd 55d44622ed list object include deleted support (#882) 
Co-authored-by: houseme <housemecn@gmail.com>
 2025-11-18 21:51:10 +08:00
weisd 85bc0ce2d5 fix: filemeta version handling and delete operations (#879) 
* fix filemeta version

* fix clippy

* fix delete version

* fix clippy/test
 2025-11-18 09:24:22 +08:00
Copilot 601f3456bc Fix large file upload freeze with adaptive buffer sizing (#869) 
* Initial plan

* Fix large file upload freeze by increasing StreamReader buffer size

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add comprehensive documentation for large file upload freeze fix

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade s3s version

* Fix compilation error: use BufReader instead of non-existent StreamReader::with_capacity

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Update documentation with correct BufReader implementation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add tokio feature `io-util`

* Implement adaptive buffer sizing based on file size

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Constants are managed uniformly and fmt code

* fix

* Fix: Trigger self-heal on read when shards missing from rejoined nodes (#871)

* Initial plan

* Fix: Trigger self-heal when missing shards detected during read

- Added proactive heal detection in get_object_with_fileinfo
- When reading an object, now checks if any shards are missing even if read succeeds
- Sends low-priority heal request to reconstruct missing shards on rejoined nodes
- This fixes the issue where data written during node outage is not healed when node rejoins

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* Unify CRC implementations to crc-fast (#873)

* Initial plan

* Replace CRC libraries with unified crc-fast implementation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fix: replace low to Normal

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-11-17 23:15:20 +08:00
weisd 1279baa72b fix replication (#875) 2025-11-17 17:37:41 +08:00
weisd acdefb6703 fix read lock (#866) 2025-11-16 11:44:13 +08:00
houseme d4817a4bea fix: modify logger from warn to info (#842) 
* fix: modify logger from warn to info

* upgrade version
 2025-11-12 11:13:38 +08:00
houseme 7e1a9e2ede 🔒 Upgrade Cryptography Libraries to Latest RC Versions (#837) 
* fix

* chore: upgrade cryptography libraries to RC versions

- Upgrade aes-gcm to 0.11.0-rc.2 with rand_core support
- Upgrade chacha20poly1305 to 0.11.0-rc.2
- Upgrade argon2 to 0.6.0-rc.2 with std features
- Upgrade hmac to 0.13.0-rc.3
- Upgrade pbkdf2 to 0.13.0-rc.2
- Upgrade rsa to 0.10.0-rc.10
- Upgrade sha1 and sha2 to 0.11.0-rc.3
- Upgrade md-5 to 0.11.0-rc.3

These upgrades provide enhanced security features and performance
improvements while maintaining backward compatibility with existing
encryption workflows.

* add

* improve code

* fix
 2025-11-11 21:10:03 +08:00
安正超 8a020ec4d9 wip (#830) 2025-11-11 09:34:58 +08:00
weisd 77a3489ed2 fix list object err (#831) 
fix list object err (#831)

#827
#815
#635
#752
 2025-11-10 23:42:15 +08:00
weisd 5941062909 fix (#828) 2025-11-10 19:22:58 +08:00
houseme d934e3905b Refactor telemetry initialization for non-production environments (#789) 
* add dep `scopeguard`

* improve for tracing

* fix

* fix

* improve code for import

* add logger trace id

* fix

* fix

* fix

* fix

* fix
 2025-11-05 00:55:08 +08:00
Niklas Mollenhauer 79012be2c8 Add default storage class to ListObjectsV2 (#765) 
* Add InvalidRangeSpec error

* Add EntityTooSmall to from_u32

* Add InvalidRangeSpec to from_u32

* Map InvalidRangeSpec to correct S3ErrorCode

* Return Error::InvalidRangeSpec

* Use auto implementation

* Add default storage class to ListObjectsV2

Resolves #764

* Add storage_class to response

* Make storage class optional so default won't be an empty string

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2025-10-31 19:32:25 +08:00
loverustfs 325ff62684 Issue 762 (#763) 
* Add InvalidRangeSpec error

* Add EntityTooSmall to from_u32

* Add InvalidRangeSpec to from_u32

* Map InvalidRangeSpec to correct S3ErrorCode

* Return Error::InvalidRangeSpec

* Use auto implementation

---------

Co-authored-by: Niklas Mollenhauer <nikeee@outlook.com>
 2025-10-31 17:20:18 +08:00
houseme 0714c7a9ca modify logger level from info to error (#744) 
* modify logger level from `info` to `error`

* fix test

* improve tokio runtime config

* add rustfs helm chart files (#747)

* add rustfs helm chart files

* update readme file with helm chart

* delete helm chart license file

* fix typo in readme file

* fix: restore localized samples in tests (#749)

* fix: restore required localized examples

* style: fix formatting issues

* improve code for Observability

* upgrade crates version

* fix

* up

* fix

---------

Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2025-10-29 19:20:53 +08:00
安正超 dd47fcf2a8 fix: restore localized samples in tests (#749) 
* fix: restore required localized examples

* style: fix formatting issues
 2025-10-29 13:16:31 +08:00
weisd 40660e7b80 fix: scandir object (#733) 
* fix: scandir object count

* fix: base64 list continuation_token
 2025-10-28 15:02:43 +08:00
likewu 2aca1f77af Fix/ilm (#721) 
* fix tip remote tier error
* fix transitioned_object
* fix filemeta
* add GCS R2
* add aliyun tencent huaweicloud azure gcs r2 backend tier
* fix signer
* change azure to s3
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-10-27 20:23:50 +08:00
shiro.lee 6ab7619023 fix: The issue of multi-level objects created in Windows not being displayed has been fixed (#661) (#723) 2025-10-26 12:00:13 +08:00
weisd 6a59c0a474 fix: multipart upload checksum validation (#712) 
* fix multipart upload checksum
 2025-10-24 18:23:32 +08:00
houseme e22b24684f chore: bump dependencies, add metrics support, remove DNS resolver (#699) 
* upgrade version

* add metrics

* remove dns resolver

* add metrics counter for create bucket

* fix

* fix

* fix
 2025-10-24 00:16:17 +08:00
weisd 1d069fd351 Improve the peer client (#693) 2025-10-23 17:21:55 +08:00
houseme 416d3ad5b7 Refactor: Add observability enable flag, improve comments, remove unused config params, and enhance run function error logging. (#689) 
* improve code for dns log

* fix

* Improve comments, remove unused parameters in config.rs (opt), add observability enable flag, and enhance error logging in run function execution.
 2025-10-23 13:59:57 +08:00
houseme 7dcf01f127 feat: adjust metrics push interval to 3 seconds (#686) 
- Reduce metrics push frequency from default to 3s for better performance
- Optimize resource utilization during metrics collection
- Improve real-time monitoring responsiveness

Related to admin metrics optimization on fix/admin-metrics branch
 2025-10-22 23:47:11 +08:00
weisd e524a106c5 add make bucket error logs (#683) 
* add make bucket error logs
 2025-10-22 16:23:08 +08:00
weisd cd1e244c68 Refactor: Introduce content checksums and improve multipart/object metadata handling (#671) 
* feat:  adapt to s3s typed etag support

* refactor: move replication struct to rustfs_filemeta, fix filemeta transition bug

* add head_object checksum, filter object metadata output

* fix multipart checksum

* fix multipart checksum

* add content md5,sha256 check

* fix test

* fix cargo

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
 2025-10-20 23:46:13 +08:00
安正超 30e7f00b02 fix: update ahm integration test fixture (#659) 2025-10-17 09:13:56 +08:00
overtrue 58f8a8f46b fix: correct HTTP range suffix handling 2025-10-16 21:39:21 +08:00
安正超 d447b3e426 feat: adapt to s3s typed etag support (#653) 
* feat:  adapt to s3s typed etag support

* refactor: streamline etag handling
 2025-10-15 21:27:20 +08:00
安正超 639bf0c233 Revert "feat(append): implement object append operations with state tracking (#599)" (#646) 
This reverts commit 4f73760a45.
 2025-10-12 23:47:51 +08:00
Copilot ad99019749 Add complete MNMD Docker deployment example with startup coordination and VolumeNotFound fix (#642) 
* Initial plan

* Add MNMD Docker deployment example with 4 nodes x 4 drives

- Create docs/examples/mnmd/ directory structure
- Add docker-compose.yml with proper disk indexing (1..4)
- Add wait-and-start.sh for startup coordination
- Add README.md with usage instructions and alternatives
- Add CHECKLIST.md with step-by-step verification
- Fixes VolumeNotFound issue by using correct volume paths
- Implements health checks and startup ordering
- Uses service names for stable inter-node addressing

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add docs/examples README as index for deployment examples

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add automated test script for MNMD deployment

- Add test-deployment.sh with comprehensive validation
- Test container status, health, endpoints, connectivity
- Update README to reference test script
- Make script executable

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* improve code

* improve code

* improve dep crates `cargo shear --fix`

* upgrade aws-sdk-s3

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-10-12 13:15:14 +08:00
houseme aac9b1edb7 chore: improve event and docker-compose ,Improve the permissions of the endpoint health interface, upgrade otel from 0.30.0 to 0.31.0 (#620) 
* feat: improve code for notify

* upgrade starshard version

* upgrade version

* Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)

* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>

* Improve lock (#596)

* improve lock

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <1948535941@qq.com>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <1948535941@qq.com>

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(append): implement object append operations with state tracking (#599)

* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: rebase

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>

* build(deps): upgrade s3s (#595)

Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* fix: validate mqtt broker

* improve code for `import`

* upgrade otel relation crates version

* fix：dep("jsonwebtoken") feature = 'rust_crypto'

* fix

* fix

* fix

* upgrade version

* improve code for ecfs

* chore: improve event and docker-compose ,Improve the permissions of the `endpoint` health interface

* fix

* fix

* fix

* fix

* improve code

* fix

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: Nugine <nugine@foxmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-10-11 09:08:25 +08:00
安正超 007d9c0b21 fix: normalize ETag comparison in multipart upload and replication (#627) 
- Normalize ETags by removing quotes before comparison in complete_multipart_upload
- Fix ETag comparison in replication logic to handle quoted ETags from API responses
- Fix ETag comparison in transition object logic
- Add unit tests for trim_etag function

This fixes the ETag mismatch error when uploading large files (5GB+) via multipart upload,
which was caused by PR #592 adding quotes to ETag responses while internal storage remains unquoted.

Fixes #625
 2025-10-08 21:19:57 +08:00