gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-27 08:38:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,212 Commits 14 Branches 117 Tags
ae50760fcc5703a1e4e1c05ab656fc0fff366f34
Commit Graph

63 Commits

Author SHA1 Message Date
weisd 359c9d2d26 Enhance Object Version Management and Replication Status Handling (#1413) 2026-01-07 10:44:35 +08:00
houseme 8d7cd4cb1b chore: upgrade dependencies and migrate to aws-lc-rs (#1333) 2026-01-02 00:02:34 +08:00
0xdx2 b19e8070a2 fix(tagging): fix e2e test_object_tagging failure (#1327) 2026-01-01 17:38:37 +08:00
houseme eb33e82b56 fix: Prevent panic in GetMetrics gRPC handler on invalid input (#1291) 
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2025-12-29 03:10:23 +08:00
houseme 7e75c9b1f5 remove unlinked file (#1258) 2025-12-24 23:37:43 +08:00
sunfkny e8fe9731fd Fix memory leak in Cache update method (#1143) 2025-12-15 10:04:14 +08:00
yihong 834025d9e3 docs: fix some dead link (#1053) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-12-08 11:23:24 +08:00
weisd 030d3c9426 fix filemeta nil versionid (#1002) 2025-12-05 20:30:08 +08:00
weisd fce4e64da4 Fix/objectdelete (#917) 
* fix getobject content length resp

* fix delete object

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2025-11-24 16:35:51 +08:00
shiro.lee 9b9bbb662b fix: removing the Limit on the Number of Object Versions (#819) (#892) 
removing the Limit on the Number of Object Versions (#819)
 2025-11-19 22:34:26 +08:00
weisd 85bc0ce2d5 fix: filemeta version handling and delete operations (#879) 
* fix filemeta version

* fix clippy

* fix delete version

* fix clippy/test
 2025-11-18 09:24:22 +08:00
Copilot 601f3456bc Fix large file upload freeze with adaptive buffer sizing (#869) 
* Initial plan

* Fix large file upload freeze by increasing StreamReader buffer size

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add comprehensive documentation for large file upload freeze fix

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade s3s version

* Fix compilation error: use BufReader instead of non-existent StreamReader::with_capacity

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Update documentation with correct BufReader implementation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add tokio feature `io-util`

* Implement adaptive buffer sizing based on file size

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Constants are managed uniformly and fmt code

* fix

* Fix: Trigger self-heal on read when shards missing from rejoined nodes (#871)

* Initial plan

* Fix: Trigger self-heal when missing shards detected during read

- Added proactive heal detection in get_object_with_fileinfo
- When reading an object, now checks if any shards are missing even if read succeeds
- Sends low-priority heal request to reconstruct missing shards on rejoined nodes
- This fixes the issue where data written during node outage is not healed when node rejoins

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* Unify CRC implementations to crc-fast (#873)

* Initial plan

* Replace CRC libraries with unified crc-fast implementation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fix: replace low to Normal

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-11-17 23:15:20 +08:00
houseme d934e3905b Refactor telemetry initialization for non-production environments (#789) 
* add dep `scopeguard`

* improve for tracing

* fix

* fix

* improve code for import

* add logger trace id

* fix

* fix

* fix

* fix

* fix
 2025-11-05 00:55:08 +08:00
安正超 dd47fcf2a8 fix: restore localized samples in tests (#749) 
* fix: restore required localized examples

* style: fix formatting issues
 2025-10-29 13:16:31 +08:00
likewu 2aca1f77af Fix/ilm (#721) 
* fix tip remote tier error
* fix transitioned_object
* fix filemeta
* add GCS R2
* add aliyun tencent huaweicloud azure gcs r2 backend tier
* fix signer
* change azure to s3
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-10-27 20:23:50 +08:00
weisd 6a59c0a474 fix: multipart upload checksum validation (#712) 
* fix multipart upload checksum
 2025-10-24 18:23:32 +08:00
weisd cd1e244c68 Refactor: Introduce content checksums and improve multipart/object metadata handling (#671) 
* feat:  adapt to s3s typed etag support

* refactor: move replication struct to rustfs_filemeta, fix filemeta transition bug

* add head_object checksum, filter object metadata output

* fix multipart checksum

* fix multipart checksum

* add content md5,sha256 check

* fix test

* fix cargo

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
 2025-10-20 23:46:13 +08:00
安正超 639bf0c233 Revert "feat(append): implement object append operations with state tracking (#599)" (#646) 
This reverts commit 4f73760a45.
 2025-10-12 23:47:51 +08:00
weisd 5689311cff fix:#630 (#633) 2025-10-10 15:16:28 +08:00
guojidan 4f73760a45 feat(append): implement object append operations with state tracking (#599) 
* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: rebase

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
 2025-09-27 20:06:26 -07:00
weisd 90f21a9102 refactor: Reimplement bucket replication system with enhanced architecture (#590) 
* feat:refactor replication

* use aws sdk for replication client

* refactor/replication

* merge main

* fix lifecycle test
 2025-09-26 14:27:53 +08:00
guojidan 9ddf6a011d feature: support kms && encryt (#573) 
* feat(kms): implement key management service with local and vault backends

Signed-off-by: junxiang Mu <1948535941@qq.com>

* feat(kms): enhance security with zeroize for sensitive data and improve key management

Signed-off-by: junxiang Mu <1948535941@qq.com>

* remove Hashi word

Signed-off-by: junxiang Mu <1948535941@qq.com>

* refactor: remove unused request structs from kms handlers

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
 2025-09-22 17:53:05 +08:00
guojidan 9d5ed1acac Feature/scanner performance optimization (#498) 
* Refactor: reimplement scanner

Signed-off-by: RustFS Developer <dandan@rustfs.com>

* comment lock

Signed-off-by: junxiang Mu <1948535941@qq.com>

* remove dirty file

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Fix: fix rebase

* fix(scanner): Improve error handling and logging

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: RustFS Developer <dandan@rustfs.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: RustFS Developer <dandan@rustfs.com>
 2025-09-08 18:35:45 +08:00
weisd 01b2623f66 Fix/response (#485) 
* fix:list_parts response

* fix:list_objects skip delete_marker
 2025-09-03 17:52:31 +08:00
weisd c9d26c6e88 Fix/delete version (#484) 
* fix:delete_version

* fix:test_lifecycle_expiry_basic

---------

Co-authored-by: likewu <likewu@126.com>
 2025-09-03 15:12:58 +08:00
likewu 7462be983a Feature up/ilm (#470) 
* fix delete-marker expiration. add api_restore.

* time retry object upload

* lock file

* make fmt

* restore object

* serde-rs-xml -> quick-xml

* scanner_item prefix object_name

* object_path

* object_name

* fi version_purge_status

* old_dir None

Co-authored-by: houseme <housemecn@gmail.com>
 2025-09-01 16:11:28 +08:00
weisd 387f4faf78 fix:rm object versions (#385) 2025-08-12 15:33:47 +08:00
guojidan d987686c14 feat(lifecycle): Implement object lifecycle management functionality (#358) 
* feat(lifecycle): Implement object lifecycle management functionality

Add a lifecycle module to automatically handle object expiration and transition during scanning
Modify the file metadata cache module to be publicly visible to support lifecycle operations
Adjust the scanning interval to a shorter time for testing lifecycle rules
Implement the parsing and execution logic for S3 lifecycle configurations
Add integration tests to verify the lifecycle expiration functionality
Update dependencies to support the new lifecycle features

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix cargo dependencies

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix fmt

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-08-08 10:51:02 +08:00
houseme 48a9707110 fix: add tokio-test (#363) 
* fix: add tokio-test

* fix: "called `unwrap` on `v` after checking its variant with `is_some`"

    = help: try using `if let` or `match`
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_unwrap
    = note: `-D clippy::unnecessary-unwrap` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(clippy::unnecessary_unwrap)]`

* fmt

* set toolchain 1.88.0

* fmt

* fix: cliip
 2025-08-08 10:23:22 +08:00
houseme 6c37e1cb2a refactor: replace lazy_static with LazyLock (#318) 
* refactor: replace `lazy_static` with `LazyLock`

Replace `lazy_static` with `LazyLock`.

Compile time may reduce a little.

See https://github.com/rust-lang-nursery/lazy-static.rs/issues/214

* fmt

* fix
 2025-07-31 14:25:39 +08:00
junxiang Mu d562620e99 fix: implement uses_data_dir method 
Signed-off-by: junxiang Mu <1948535941@qq.com>
 2025-07-24 12:14:05 +08:00
Chrislearn Young 823d4b6f79 Add typos github actions and fix typos (#265) 
* Add typo github actions and fix typos

* cargo fmt
 2025-07-19 22:08:50 +08:00
weisd a9d77a618f feat: implement list_parts API for S3 multipart upload compatibility (#209) 
* feat: add list_parts api
 2025-07-15 16:04:03 +08:00
houseme 36acb5bce9 feat(console): Enhance network address handling for WebUI (#129) 
* add crates homepage,description,keywords,categories,documentation

* add readme

* modify version 0.0.3

* cargo fmt

* fix: yaml.docker-compose.security.no-new-privileges.no-new-privileges-docker-compose.yml (#63)

* Feature up/ilm (#61)

* fix delete-marker expiration. add api_restore.

* remove target return 204

* log level

* fix: make lint build and clippy happy (#71)

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: make ci and local use the same toolchain (#72)

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* feat: optimize GitHub Actions workflows with performance improvements (#77)

* feat: optimize GitHub Actions workflows with performance improvements

- Rename workflows with more descriptive names
- Add unified setup action for consistent environment setup
- Optimize caching strategy with Swatinem/rust-cache@v2
- Implement skip-check mechanism to avoid duplicate builds
- Simplify matrix builds with better include/exclude logic
- Add intelligent build strategy checks
- Optimize Docker multi-arch builds
- Improve artifact naming and retention
- Add performance testing with benchmark support
- Enhance security audit with dependency scanning
- Change Chinese comments to English for better maintainability

Performance improvements:
- CI testing: ~35 min (42% faster)
- Build release: ~60 min (50% faster)
- Docker builds: ~45 min (50% faster)
- Security audit: ~8 min (47% faster)

* fix: correct secrets context usage in GitHub Actions workflow

- Move environment variables to job level to fix secrets access issue
- Fix unrecognized named-value 'secrets' error in if condition
- Ensure OSS upload step can properly check for required secrets

* fix: resolve GitHub API rate limit by adding authentication token

- Add github-token input to setup action to authenticate GitHub API requests
- Pass GITHUB_TOKEN to all setup action usages to avoid rate limiting
- Fix arduino/setup-protoc@v3 API access issues in CI/CD workflows
- Ensure protoc installation can successfully access GitHub releases API

* fix:make bucket err (#85)

* Rename DEVELOPMENT.md to CONTRIBUTING.md

* Create issue-translator.yml (#89)

Enable Issues Translator

* fix(dockerfile): correct env variable names for access/secret key and improve compatibility (#90)

* fix: restore Zig and cargo-zigbuild caching in GitHub Actions setup action (#92)

* fix: restore Zig and cargo-zigbuild caching in GitHub Actions setup action

Use mlugg/setup-zig and taiki-e/cache-cargo-install-action to speed up cross-compilation tool installation and avoid repeated downloads. All comments and code are in English.

* fix: use correct taiki-e/install-action for cargo-zigbuild

Use taiki-e/install-action@cargo-zigbuild instead of taiki-e/cache-cargo-install-action@v2 to match the original implementation from PR #77.

* refactor: remove explicit Zig version to use latest stable

* Create CODE_OF_CONDUCT.md

* Create SECURITY.md

* Update issue templates

* Create CLA.md

* docs: update PR template to English version

* fix: improve data scanner random sleep calculation

- Fix random number generation API usage
- Adjust sleep calculation to follow MinIO pattern
- Ensure proper random range for scanner cycles

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix: soupprt ipv6

* improve log

* add client ip log

* Update rustfs/src/console.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* improve code

* feat: unify package format to zip for all platforms

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: kira-offgrid <kira@offgridsec.com>
Co-authored-by: likewu <likewu@126.com>
Co-authored-by: laoliu <lygn128@163.com>
Co-authored-by: yihong <zouzou0208@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: Yone <zhiyu@live.cn>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-07-09 14:39:40 +08:00
安正超 3dbcaaa221 docs: simplify crates README files and enforce PR-only workflow (#110) 
* docs: simplify all crates README files

- Remove extensive code examples and detailed documentation
- Convert to minimal module introductions with core feature lists
- Direct users to main RustFS repository for comprehensive docs
- Updated 20 crate README files for consistency and brevity

Files updated:
- crates/rio/README.md (415→15 lines)
- crates/s3select-api/README.md (592→15 lines)
- crates/s3select-query/README.md (658→15 lines)
- crates/signer/README.md (407→15 lines)
- crates/utils/README.md (395→15 lines)
- crates/workers/README.md (463→15 lines)
- crates/zip/README.md (408→15 lines)

* docs: restore original headers in crates README files

- Add back RustFS logo image and CI badges
- Restore formatted headers and structured layout
- Keep simplified content with module introductions
- Maintain consistent documentation structure across all crates

All 20 crate README files now have proper headers while keeping
the simplified content that directs users to the main repository.

* rules: enforce PR-only workflow for main branch

- Strengthen rule that ALL changes must go through pull requests
- Explicitly forbid direct commits to main branch under any circumstances
- Add comprehensive PR requirements and enforcement guidelines
- Clarify that PRs are the ONLY way to merge to main branch
- Add requirement for PR approval before merging
- Include enforcement mechanisms for branch protection
 2025-07-08 21:10:07 +08:00
安正超 fa17f7b1e3 feat: add comprehensive README documentation for all RustFS submodules (#48) 2025-07-04 23:02:13 +08:00
houseme 5826396cd0 refactor: Restructure project layout and clean up dependencies (#30) 
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity.

Key changes include:
- Adjusted the directory layout for a more logical module organization.
- Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times.
- Updated import paths and configuration files to reflect the structural changes.
 2025-07-02 19:33:12 +08:00
houseme 2e14b32ccd chore: Add copyright and license headers (#23) 
* chore: Add copyright and license headers

This commit adds the Apache 2.0 license and a copyright notice to the header of all source files. This ensures that the licensing and copyright information is clearly stated within the codebase.

* cargo fmt

* fix

* fmt

* fix clippy
 2025-07-02 15:07:47 +08:00
weisd 20054d0a56 fix: simplify filemeta MessagePack serialization and improve code quality 
- Refactor `unmarshal_msg` to use `rmp_serde::from_slice` instead of manual parsing
- Add serde field renaming attributes to `FileMetaVersion` struct
- Remove 428 lines of manual MessagePack parsing code
- Improve string comparison using `!object.is_empty()` instead of `object != ""`
- Update volume directory numbering in run script from test{0..4} to test{1..4}
- Clean up unused imports and code

Fixes #517
 2025-06-29 22:39:36 +08:00
houseme 7bd8431006 cargo fmt 2025-06-27 18:53:51 +08:00
houseme 749537664f fix：Apply suggestions from clippy 1.88 2025-06-27 18:16:29 +08:00
weisd 7e17d7d729 refactor: improve object listing and version handling 
- Refactor find_version_index to accept Uuid directly instead of string
- Split from_meta_cache_entries_sorted into separate methods for versions and infos
- Add support for after_version_id filtering in version listing
- Fix S3 error code for replication configuration not found
- Improve error handling and logging levels
- Clean up import statements and remove debug code
 2025-06-24 11:10:23 +08:00
likewu 364c3098b2 Fix/main (#508) 
* fix error

* fix

* fix cargo

* fix openssl cfg

* fix clippy

* fix clippy

* fix

* fix clippy

* fix clippy
 2025-06-24 10:27:41 +08:00
likewu 81f0c9763f Merge branch 'main' of https://github.com/rustfs/s3-rustfs into feature/ilm 
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/utils/Cargo.toml
#	crates/utils/src/net.rs
#	ecstore/Cargo.toml
#	ecstore/src/set_disk.rs
#	rustfs/src/storage/ecfs.rs
 2025-06-23 16:42:18 +08:00
likewu 2c176dd864 fix clippy 2025-06-23 12:03:46 +08:00
likewu cc71f40a6d ilm feature add 2025-06-22 23:04:40 +08:00
houseme d3cc36f6e0 fix 2025-06-20 10:51:36 +08:00
houseme e6b019c29d Merge branch 'main' of github.com:rustfs/s3-rustfs into feature/observability-metrics 
# Conflicts:
#	.github/workflows/build.yml
#	.github/workflows/ci.yml
#	Cargo.lock
#	Cargo.toml
#	appauth/src/token.rs
#	crates/config/src/config.rs
#	crates/event-notifier/examples/simple.rs
#	crates/event-notifier/src/global.rs
#	crates/event-notifier/src/lib.rs
#	crates/event-notifier/src/notifier.rs
#	crates/event-notifier/src/store.rs
#	crates/filemeta/src/filemeta.rs
#	crates/notify/examples/webhook.rs
#	crates/utils/Cargo.toml
#	ecstore/Cargo.toml
#	ecstore/src/cmd/bucket_replication.rs
#	ecstore/src/config/com.rs
#	ecstore/src/disk/error.rs
#	ecstore/src/disk/mod.rs
#	ecstore/src/set_disk.rs
#	ecstore/src/store_api.rs
#	ecstore/src/store_list_objects.rs
#	iam/Cargo.toml
#	iam/src/manager.rs
#	policy/Cargo.toml
#	rustfs/src/admin/rpc.rs
#	rustfs/src/main.rs
#	rustfs/src/storage/mod.rs
 2025-06-19 13:16:48 +08:00
weisd c48ebd5149 feat: add compress support 2025-06-17 15:06:40 +08:00
weisd ca298b460c fix test 2025-06-16 11:40:15 +08:00