2aca1f77af
Fix/ilm ( #721 )
...
* fix tip remote tier error
* fix transitioned_object
* fix filemeta
* add GCS R2
* add aliyun tencent huaweicloud azure gcs r2 backend tier
* fix signer
* change azure to s3
Co-authored-by: houseme <housemecn@gmail.com >
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com >
2025-10-27 20:23:50 +08:00
cd1e244c68
Refactor: Introduce content checksums and improve multipart/object metadata handling ( #671 )
...
* feat: adapt to s3s typed etag support
* refactor: move replication struct to rustfs_filemeta, fix filemeta transition bug
* add head_object checksum, filter object metadata output
* fix multipart checksum
* fix multipart checksum
* add content md5,sha256 check
* fix test
* fix cargo
---------
Co-authored-by: overtrue <anzhengchao@gmail.com >
2025-10-20 23:46:13 +08:00
30e7f00b02
fix: update ahm integration test fixture ( #659 )
2025-10-17 09:13:56 +08:00
0e680eae31
fix typos and bump the dependencies group with 9 updates ( #614 )
...
* fix typos
* build(deps): bump the dependencies group with 9 updates (#613 )
Bumps the dependencies group with 9 updates:
| Package | From | To |
| --- | --- | --- |
| [axum](https://github.com/tokio-rs/axum ) | `0.8.4` | `0.8.6` |
| [axum-extra](https://github.com/tokio-rs/axum ) | `0.10.1` | `0.10.3` |
| [regex](https://github.com/rust-lang/regex ) | `1.11.2` | `1.11.3` |
| [serde](https://github.com/serde-rs/serde ) | `1.0.226` | `1.0.228` |
| [shadow-rs](https://github.com/baoyachi/shadow-rs ) | `1.3.0` | `1.4.0` |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo ) | `0.37.0` | `0.37.1` |
| [thiserror](https://github.com/dtolnay/thiserror ) | `2.0.16` | `2.0.17` |
| [tokio-rustls](https://github.com/rustls/tokio-rustls ) | `0.26.3` | `0.26.4` |
| [zeroize](https://github.com/RustCrypto/utils ) | `1.8.1` | `1.8.2` |
Updates `axum` from 0.8.4 to 0.8.6
- [Release notes](https://github.com/tokio-rs/axum/releases )
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/axum/compare/axum-v0.8.4...axum-v0.8.6 )
Updates `axum-extra` from 0.10.1 to 0.10.3
- [Release notes](https://github.com/tokio-rs/axum/releases )
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md )
- [Commits](https://github.com/tokio-rs/axum/compare/axum-extra-v0.10.1...axum-extra-v0.10.3 )
Updates `regex` from 1.11.2 to 1.11.3
- [Release notes](https://github.com/rust-lang/regex/releases )
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-lang/regex/compare/1.11.2...1.11.3 )
Updates `serde` from 1.0.226 to 1.0.228
- [Release notes](https://github.com/serde-rs/serde/releases )
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.226...v1.0.228 )
Updates `shadow-rs` from 1.3.0 to 1.4.0
- [Release notes](https://github.com/baoyachi/shadow-rs/releases )
- [Commits](https://github.com/baoyachi/shadow-rs/compare/1.3.0...v1.4.0 )
Updates `sysinfo` from 0.37.0 to 0.37.1
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md )
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.37.0...v0.37.1 )
Updates `thiserror` from 2.0.16 to 2.0.17
- [Release notes](https://github.com/dtolnay/thiserror/releases )
- [Commits](https://github.com/dtolnay/thiserror/compare/2.0.16...2.0.17 )
Updates `tokio-rustls` from 0.26.3 to 0.26.4
- [Release notes](https://github.com/rustls/tokio-rustls/releases )
- [Commits](https://github.com/rustls/tokio-rustls/compare/v/0.26.3...v/0.26.4 )
Updates `zeroize` from 1.8.1 to 1.8.2
- [Commits](https://github.com/RustCrypto/utils/compare/zeroize-v1.8.1...zeroize-v1.8.2 )
---
updated-dependencies:
- dependency-name: axum
dependency-version: 0.8.6
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: axum-extra
dependency-version: 0.10.3
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: regex
dependency-version: 1.11.3
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: serde
dependency-version: 1.0.228
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: shadow-rs
dependency-version: 1.4.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: dependencies
- dependency-name: sysinfo
dependency-version: 0.37.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: thiserror
dependency-version: 2.0.17
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: tokio-rustls
dependency-version: 0.26.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
- dependency-name: zeroize
dependency-version: 1.8.2
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---------
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-02 23:29:18 +08:00
be66cf8bd3
Improve lock ( #596 )
...
* improve lock
Signed-off-by: Mu junxiang <1948535941@qq.com >
* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability
Signed-off-by: Mu junxiang <1948535941@qq.com >
* chore: remove dirty docs
Signed-off-by: Mu junxiang <1948535941@qq.com >
---------
Signed-off-by: Mu junxiang <1948535941@qq.com >
2025-09-27 17:57:56 -07:00
90f21a9102
refactor: Reimplement bucket replication system with enhanced architecture ( #590 )
...
* feat:refactor replication
* use aws sdk for replication client
* refactor/replication
* merge main
* fix lifecycle test
2025-09-26 14:27:53 +08:00
12ecb36c6d
Fix collect ( #586 )
...
* fix: fix datausageinfo
Signed-off-by: junxiang Mu <1948535941@qq.com >
* feat(data-usage): implement local disk snapshot aggregation for data usage statistics
Signed-off-by: junxiang Mu <1948535941@qq.com >
* feat(scanner): improve data usage collection with local scan aggregation
Signed-off-by: junxiang Mu <1948535941@qq.com >
* refactor: improve object existence check and code style
Signed-off-by: junxiang Mu <1948535941@qq.com >
---------
Signed-off-by: junxiang Mu <1948535941@qq.com >
2025-09-24 02:48:23 -07:00
9c97524c3b
feat: consolidate AI rules into unified AGENTS.md ( #501 )
...
- Merge all AI rules from .rules.md, .cursorrules, and CLAUDE.md into AGENTS.md
- Add competitor keyword prohibition rules (minio, ceph, swift, etc.)
- Simplify rules by removing overly detailed code examples
- Integrate new development principles as highest priority
- Remove old tool-specific rule files
- Fix clippy warnings for format string improvements
2025-09-09 21:36:34 +08:00
14a8802ce7
Fix: fix collect usage data ( #500 )
...
Signed-off-by: junxiang Mu <1948535941@qq.com >
2025-09-09 18:39:51 +08:00
9d5ed1acac
Feature/scanner performance optimization ( #498 )
...
* Refactor: reimplement scanner
Signed-off-by: RustFS Developer <dandan@rustfs.com >
* comment lock
Signed-off-by: junxiang Mu <1948535941@qq.com >
* remove dirty file
Signed-off-by: junxiang Mu <1948535941@qq.com >
* Fix: fix rebase
* fix(scanner): Improve error handling and logging
Signed-off-by: junxiang Mu <1948535941@qq.com >
---------
Signed-off-by: RustFS Developer <dandan@rustfs.com >
Signed-off-by: junxiang Mu <1948535941@qq.com >
Co-authored-by: RustFS Developer <dandan@rustfs.com >
2025-09-08 18:35:45 +08:00
c9d26c6e88
Fix/delete version ( #484 )
...
* fix:delete_version
* fix:test_lifecycle_expiry_basic
---------
Co-authored-by: likewu <likewu@126.com >
2025-09-03 15:12:58 +08:00
087df484a3
Fix/ilm ( #478 )
2025-09-02 18:18:26 +08:00
7462be983a
Feature up/ilm ( #470 )
...
* fix delete-marker expiration. add api_restore.
* time retry object upload
* lock file
* make fmt
* restore object
* serde-rs-xml -> quick-xml
* scanner_item prefix object_name
* object_path
* object_name
* fi version_purge_status
* old_dir None
Co-authored-by: houseme <housemecn@gmail.com >
2025-09-01 16:11:28 +08:00
fd2aab2bd9
fix：revet #443 #446 ( #452 )
...
* fix： revet #443 #446
* fix
2025-08-23 17:30:06 +08:00
46bd75c0f8
ahm(scanner): throttle scanning, skip recently-modified objects, and … ( #443 )
...
* ahm(scanner): throttle scanning, skip recently-modified objects, and gate missing-object heals to deep mode; adjust conservative defaults
Signed-off-by: loverustfs <hello@rustfs.com >
* ecstore: enable virtual-host AUTO heuristics and URL building; signer: fix SigV2 canonical resource for vhost; add unit tests
* ecstore: AUTO virtual-host style URL selection; signer: SigV2 canonical resource fixes for vhost; tests added.\nahm: fix clippy drop_non_drop; integration tests robust to existing bucket; ignore flaky lifecycle test.\nMakefile: test target falls back to cargo test when nextest missing.\npre-commit: all checks green.
---------
Signed-off-by: loverustfs <hello@rustfs.com >
2025-08-22 16:03:29 +08:00
d987686c14
feat(lifecycle): Implement object lifecycle management functionality ( #358 )
...
* feat(lifecycle): Implement object lifecycle management functionality
Add a lifecycle module to automatically handle object expiration and transition during scanning
Modify the file metadata cache module to be publicly visible to support lifecycle operations
Adjust the scanning interval to a shorter time for testing lifecycle rules
Implement the parsing and execution logic for S3 lifecycle configurations
Add integration tests to verify the lifecycle expiration functionality
Update dependencies to support the new lifecycle features
Signed-off-by: junxiang Mu <1948535941@qq.com >
* fix cargo dependencies
Signed-off-by: junxiang Mu <1948535941@qq.com >
* fix fmt
Signed-off-by: junxiang Mu <1948535941@qq.com >
---------
Signed-off-by: junxiang Mu <1948535941@qq.com >
Co-authored-by: houseme <housemecn@gmail.com >
2025-08-08 10:51:02 +08:00
ea210d52dc
refactor(heal): unify heal request interface, add disk field, update ahm/ecstore/common for erasure set healing
...
Signed-off-by: junxiang Mu <1948535941@qq.com >
2025-07-24 12:14:03 +08:00
e7d0a8d4b9
feat: integrate global metrics system into AHM scanner
...
- Add global metrics system to common crate for cross-module usage
- Integrate global metrics collection into AHM scanner operations
- Update ECStore to use common metrics system instead of local implementation
- Add chrono dependency to AHM crate for timestamp handling
- Re-export IlmAction from common metrics in ECStore lifecycle module
- Update scanner methods to use global metrics for cycle, disk, and volume scans
- Maintain backward compatibility with local metrics collector
- Fix clippy warnings and ensure proper code formatting
This change enables unified metrics collection across the entire RustFS system,
allowing better monitoring and observability of scanner operations.
Signed-off-by: junxiang Mu <1948535941@qq.com >
2025-07-24 12:12:49 +08:00
7d3b2b774c
fix heal disk
...
Signed-off-by: junxiang Mu <1948535941@qq.com >
2025-07-24 12:12:49 +08:00
c49414f6ac
fix: resolve test conflicts and improve data scanner functionality
...
- Fix multi-threaded test conflicts in AHM heal integration tests
- Remove global environment sharing to prevent test state pollution
- Fix test_all_disk_method by clearing global disk map before test
- Improve data scanner and cache value implementations
- Update dependencies and resolve clippy warnings
Signed-off-by: junxiang Mu <1948535941@qq.com >
2025-07-24 12:12:49 +08:00
8e766b90cd
feat: implement heal channel mechanism for admin-ahm communication
...
- Add global unbounded channel in common crate for heal requests
- Implement channel processor in ahm to handle heal commands
- Add Start/Query/Cancel commands support via channel
- Integrate heal manager initialization in main.rs
- Replace direct MRF calls with channel-based heal requests in ecstore
- Support advanced heal options including pool_index and set_index
- Enable admin handlers to send heal requests via channel
2025-07-24 12:12:49 +08:00
3409cd8dff
feat(ahm): add HealingTracker support & complete fresh-disk healing
...
• Introduce ecstore HealingTracker into ahm crate; load/init/save tracker
• Re-implement heal_fresh_disk to use heal_erasure_set with tracker
• Enhance auto-disk scanner: detect unformatted disks via get_disk_id()
• Remove DataUsageCache handling for now
• Refactor imports & types, clean up duplicate constants
2025-07-24 12:12:49 +08:00
f4973a681c
feat: implement complete ahm heal system with ecstore integration
...
- Add comprehensive heal storage API with ECStore integration
- Implement heal object, bucket, disk, metadata, and EC decode operations
- Add heal task management with progress tracking and statistics
- Optimize heal manager by removing unnecessary workers
- Add integration tests for core heal functionality (heal_object, heal_bucket, heal_format)
- Integrate with ecstore's native heal commands for actual repair operations
Signed-off-by: junxiang Mu <1948535941@qq.com >
2025-07-24 12:12:49 +08:00
likewu