Commit Graph

1749 Commits

Author SHA1 Message Date
weisd 56fd8132e9 fix:#303 returns empty when querying an empty or not dir (#304) 1.0.0-alpha.38 2025-07-28 16:17:40 +08:00
guojidan 35daa74430 Merge pull request #302 from guojidan/lock
Lock: add transactional
2025-07-28 12:00:44 +08:00
junxiang Mu dc156fb4cd Fix: clippy
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-28 11:38:42 +08:00
junxiang Mu de905a878c Cargo: use workspace dependence
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-28 11:02:40 +08:00
junxiang Mu f3252f989b Test: Add e2e test case for lock transactional
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-28 11:00:10 +08:00
junxiang Mu 01a2afca9a lock: Add transactional
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-28 10:59:43 +08:00
guojidan a4fe68ad21 Merge pull request #301 from guojidan/improve-sql
s3Select: add unit test case
2025-07-28 09:56:10 +08:00
junxiang Mu c03f86b23c s3Select: add unit test case
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-28 09:19:47 +08:00
guojidan 5667f324ae Merge pull request #297 from guojidan/improve-sql
Test: Add e2e_test case for sql && add script for e2e_test
2025-07-25 17:16:41 +08:00
junxiang Mu bcd806796f Test: add test script for e2e
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-25 16:52:06 +08:00
junxiang Mu 612404c47f Test: add e2e_test for s3select
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-25 15:07:44 +08:00
guojidan 85388262b3 Merge pull request #294 from guojidan/improve-sql
Refactor: DatabaseManagerSystem as global
2025-07-25 08:33:54 +08:00
junxiang Mu 25a4503285 fix: fmt
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-25 08:18:14 +08:00
安正超 526c4d5a61 refactor: 优化构建工作流,统一 latest 文件处理和简化制品上传 (#293) 2025-07-25 01:10:04 +08:00
junxiang Mu addc964d56 Refactor: DatabaseManagerSystem as global
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 17:12:51 +08:00
loverustfs 371119f733 GNU to MUSL modify Dockerfile 1.0.0-alpha.37 2025-07-24 16:36:15 +08:00
guojidan 021abc0398 Merge pull request #292 from guojidan/Arc
Chore: remove dirty file(cache.rs)
2025-07-24 16:32:20 +08:00
junxiang Mu 0672b6dd3e Chore: remove dirty file(cache.rs)
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 14:57:48 +08:00
guojidan 1372dc2857 Merge pull request #288 from guojidan/scanner
Refactor: Scanner
2025-07-24 14:42:54 +08:00
houseme 77bc9af109 Update Cargo.toml 2025-07-24 14:14:12 +08:00
junxiang Mu 91b1c84430 rebase
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:18:05 +08:00
junxiang Mu b667927216 fix fmt
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:28 +08:00
junxiang Mu 29795fac51 fix Cargo.toml
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:28 +08:00
junxiang Mu 2ce7e01f55 Chore: remove dirty file(heal)
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:27 +08:00
junxiang Mu 4fefd63a5b rebase
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu 2a8c46874d fix: auto heal when xl.meta lose
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu b8b5511b68 fix: heal data part lose
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu bdaee228db fix(ahm): adjust test expectations for missing xl.meta recovery scenario
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu d562620e99 fix: implement uses_data_dir method
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu 69b0c828c9 fix: scanner add heal bucket
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu 2bfd1efb9b Fix: fix add heal_manager into scanner when scanner start
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu 0854e6b921 Chore: rename init_heal_manager_with_channel
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu b907f4e61b refactor(ahm): remove obsolete scanner/data_usage.rs after data usage refactor
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu 6ec568459c chore: update admin handlers, lockfile, and minor fixes
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:05 +08:00
junxiang Mu ea210d52dc refactor(heal): unify heal request interface, add disk field, update ahm/ecstore/common for erasure set healing
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:14:03 +08:00
junxiang Mu 3d3c6e4e06 chore(protos): update proto definitions, remove ns_scanner, fix codegen and formatting
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu e7d0a8d4b9 feat: integrate global metrics system into AHM scanner
- Add global metrics system to common crate for cross-module usage
- Integrate global metrics collection into AHM scanner operations
- Update ECStore to use common metrics system instead of local implementation
- Add chrono dependency to AHM crate for timestamp handling
- Re-export IlmAction from common metrics in ECStore lifecycle module
- Update scanner methods to use global metrics for cycle, disk, and volume scans
- Maintain backward compatibility with local metrics collector
- Fix clippy warnings and ensure proper code formatting

This change enables unified metrics collection across the entire RustFS system,
allowing better monitoring and observability of scanner operations.

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu 7d3b2b774c fix heal disk
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu aed8f52423 refactor: integrate disk healing into erasure set healing
- Remove HealType::Disk and related disk-specific healing methods
- Integrate disk format healing into heal_erasure_set with include_format_heal option
- Update auto disk scanner to use ErasureSet heal type instead of Disk heal
- Fix disk status change event handling to use ErasureSet heal requests
- Add proper bucket list retrieval for auto healing scenarios
- Update data scanner to submit ErasureSet heal tasks for offline disks
- Remove duplicate healing logic between Disk and ErasureSet types
- Ensure all healing operations go through unified ErasureSet healing path
2025-07-24 12:12:49 +08:00
junxiang Mu c49414f6ac fix: resolve test conflicts and improve data scanner functionality
- Fix multi-threaded test conflicts in AHM heal integration tests
- Remove global environment sharing to prevent test state pollution
- Fix test_all_disk_method by clearing global disk map before test
- Improve data scanner and cache value implementations
- Update dependencies and resolve clippy warnings

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu 8e766b90cd feat: implement heal channel mechanism for admin-ahm communication
- Add global unbounded channel in common crate for heal requests
- Implement channel processor in ahm to handle heal commands
- Add Start/Query/Cancel commands support via channel
- Integrate heal manager initialization in main.rs
- Replace direct MRF calls with channel-based heal requests in ecstore
- Support advanced heal options including pool_index and set_index
- Enable admin handlers to send heal requests via channel
2025-07-24 12:12:49 +08:00
junxiang Mu 3409cd8dff feat(ahm): add HealingTracker support & complete fresh-disk healing
• Introduce ecstore HealingTracker into ahm crate; load/init/save tracker
• Re-implement heal_fresh_disk to use heal_erasure_set with tracker
• Enhance auto-disk scanner: detect unformatted disks via get_disk_id()
• Remove DataUsageCache handling for now
• Refactor imports & types, clean up duplicate constants
2025-07-24 12:12:49 +08:00
junxiang Mu f4973a681c feat: implement complete ahm heal system with ecstore integration
- Add comprehensive heal storage API with ECStore integration
- Implement heal object, bucket, disk, metadata, and EC decode operations
- Add heal task management with progress tracking and statistics
- Optimize heal manager by removing unnecessary workers
- Add integration tests for core heal functionality (heal_object, heal_bucket, heal_format)
- Integrate with ecstore's native heal commands for actual repair operations

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu 4fb3d187d0 feat: implement heal subsystem for automatic data repair
- Add heal module with core types (HealType, HealRequest, HealTask)
- Implement HealManager for task scheduling and execution
- Add HealStorageAPI trait and ECStoreHealStorage implementation
- Integrate heal capabilities into scanner for automatic repair
- Support multiple heal types: object, bucket, disk, metadata, MRF, EC decode
- Add progress tracking and event system for heal operations
- Merge heal and scanner error types for unified error handling
- Include comprehensive logging and metrics for heal operations

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
dandan 0aff736efd Chore: fix ref and fix comment
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
dandan 2aa7a631ef feat: refactor scanner module and add data usage statistics
- Move scanner code to scanner/ subdirectory for better organization
- Add data usage statistics collection and persistence
- Implement histogram support for size and version distribution
- Add global cancel token management for scanner operations
- Integrate scanner with ECStore for comprehensive data analysis
- Update error handling and improve test isolation
- Add data usage API endpoints and backend integration

Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
dandan b40ef147a9 refact: step 2
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
junxiang Mu 1f11a3167b fix: Refact heal and scanner design
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 12:12:49 +08:00
guojidan 18b0134ddf Merge pull request #290 from guojidan/feat/complete-lock-implementation
refactor: reimplement lock
2025-07-24 12:11:19 +08:00
junxiang Mu b48a5fdc94 fix fmt
Signed-off-by: junxiang Mu <1948535941@qq.com>
2025-07-24 11:52:57 +08:00