gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-27 16:48:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,244 Commits 15 Branches 117 Tags
45391ac776019bfdfd67335884a2cf2d55472cbe
Commit Graph

100 Commits

Author SHA1 Message Date
安正超 a63aaf933d fix(notify): normalize lock ordering to prevent deadlock (#3199) 
* fix(notify): normalize lock ordering to prevent deadlock

replace_targets() and shutdown() acquired target_list before replay_workers,
opposite to runtime_view.rs order. This is an ABBA deadlock pattern.

- Swap to replay_workers -> target_list order (canonical)
- Add lock order comments
- 82 notify tests pass

* docs(notify): update runtime facade lock order

* docs(notify): clarify runtime facade lock order

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-03 15:37:59 +00:00
安正超 1838922f07 fix(security): add deny_unknown_fields to deserialization structs (#3198) 
* fix(security): add deny_unknown_fields to deserialization structs

Prevent silent acceptance of malformed or adversarial payloads.

- policy: Policy, BucketPolicy, Statement, BPStatement, PrincipalObject
- notify: S3KeyFilter custom deserializer rejects unknown child elements
- update: VersionInfo (remote HTTP response)

26 policy + 82 notify tests pass.

* fix(update): keep version response forward compatible

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-03 15:33:28 +00:00
安正超 b5e565c0ce chore(agents): add Rust code quality rules and skill (#3144) 
* docs: update security advisory lessons

* chore(agents): add Rust code quality rules and skill

Add rules derived from full-project code review (48 findings across
7 dimensions) to prevent recurring issues in agent-generated code.

AGENTS.md changes:
- crates/AGENTS.md: error type design, concurrency, recursion safety,
  type casting, test quality rules
- root AGENTS.md: serde safety, naming conventions
- crates/ecstore/AGENTS.md: allocation discipline, lock ordering,
  recursion safety, dead code policy
- crates/notify/AGENTS.md: lock ordering for runtime_view/facade

Skill changes:
- code-change-verification: add Rust-specific checks (unwrap, as cast,
  clone, lock order, recursion, error types, test assertions)
- security-advisory-lessons: add serde deserialization safety pattern
- NEW rust-code-quality: automated scan + manual review checklist
 2026-05-31 22:19:36 +08:00
houseme 73bde843d6 refactor(s3): consolidate semantic boundaries and remove s3-common (#3012) 
* refactor(common): introduce rustfs-data-usage core crate

* refactor(concurrency): migrate workers crate into concurrency

* refactor(crypto): migrate appauth token APIs into crypto

* fix docs urls

* remove unused crate

* refactor(data-usage): switch consumers to rustfs-data-usage

* chore(fmt): apply cargo fmt and lockfile sync

* refactor(common): remove data_usage compatibility re-export

* refactor(capacity): move capacity_scope to object-capacity

* refactor(io-metrics): relocate internode metrics from common

* refactor(common): decouple scanner report from madmin

* chore(fmt): normalize import ordering after pre-commit

* refactor(s3): split s3 types and ops crates

* refactor(s3): centralize event version and safe parsing

* refactor(s3): add op-event compatibility guardrails

* refactor(s3): add runtime op-event mismatch observability

* refactor(s3): extract delete event mapping helper

* refactor(s3): extract put event mapping helper

* refactor(s3): consolidate remaining event semantic helpers

* refactor(s3): add op-event coverage checks and observability alerts

* refactor(s3-ops): consolidate op-event semantic mapping

* refactor(scanner): remove last_minute wrapper module

* refactor(scanner): consolidate duplicated data usage models
 2026-05-19 12:50:25 +00:00
houseme c4248078d6 chore: update dependencies (#2890) 
* chore: update dependencies

* build(deps): bump the dependencies group with 5 updates

* build(deps): bump the dependencies group with 6 updates (#2908)

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>

* fix(ecstore): narrow Windows URL drive path rewrite

* chore(deps): bump starshard to 1.2.0

* revert(ecstore): restore endpoint Windows path behavior

* up

* up

* up

* perf(notify): use cached snapshot mode for scans

* fix

* chore(deps): bump workspace dependency versions

* chore(deps): refresh pinned dependency references

* chore(ci): align profiling and decommission tooling updates

- enable tokio_unstable cfg in performance profiling build flags\n- bump Rust base image from 1.93 to 1.95 in source and decommission Dockerfiles\n- remove obsolete compose version key from docker-compose-simple.yml\n- add standard Apache-2.0 license header to docker-compose.decommission.yml

* chore(deps): bump the dependencies group across 1 directory with 7 updates (#2994)

Bumps the dependencies group with 7 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.39.1` | `0.39.2` |
| [dial9-tokio-telemetry](https://github.com/dial9-rs/dial9-tokio-telemetry) | `0.3.9` | `0.3.10` |
| [opentelemetry](https://github.com/open-telemetry/opentelemetry-rust) | `0.31.0` | `0.32.0` |
| [opentelemetry-otlp](https://github.com/open-telemetry/opentelemetry-rust) | `0.31.1` | `0.32.0` |
| [opentelemetry_sdk](https://github.com/open-telemetry/opentelemetry-rust) | `0.31.0` | `0.32.0` |
| [opentelemetry-semantic-conventions](https://github.com/open-telemetry/opentelemetry-rust) | `0.31.0` | `0.32.0` |
| [opentelemetry-stdout](https://github.com/open-telemetry/opentelemetry-rust) | `0.31.0` | `0.32.0` |



Updates `sysinfo` from 0.39.1 to 0.39.2
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/main/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.39.1...v0.39.2)

Updates `dial9-tokio-telemetry` from 0.3.9 to 0.3.10
- [Release notes](https://github.com/dial9-rs/dial9-tokio-telemetry/releases)
- [Changelog](https://github.com/dial9-rs/dial9/blob/main/CHANGELOG.md)
- [Commits](https://github.com/dial9-rs/dial9-tokio-telemetry/compare/dial9-tokio-telemetry-v0.3.9...dial9-tokio-telemetry-v0.3.10)

Updates `opentelemetry` from 0.31.0 to 0.32.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-rust/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-rust/blob/main/docs/release_0.32.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-rust/compare/opentelemetry-prometheus-0.31.0...opentelemetry-0.32.0)

Updates `opentelemetry-otlp` from 0.31.1 to 0.32.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-rust/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-rust/blob/main/docs/release_0.32.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-rust/compare/opentelemetry-otlp-0.31.1...opentelemetry-otlp-0.32.0)

Updates `opentelemetry_sdk` from 0.31.0 to 0.32.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-rust/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-rust/blob/main/docs/release_0.32.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-rust/compare/v0.31.0...opentelemetry_sdk-0.32.0)

Updates `opentelemetry-semantic-conventions` from 0.31.0 to 0.32.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-rust/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-rust/blob/main/docs/release_0.32.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-rust/compare/v0.31.0...opentelemetry-semantic-conventions-0.32.0)

Updates `opentelemetry-stdout` from 0.31.0 to 0.32.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-rust/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-rust/blob/main/docs/release_0.32.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-rust/compare/v0.31.0...opentelemetry-stdout-0.32.0)

---
updated-dependencies:
- dependency-name: sysinfo
  dependency-version: 0.39.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: dial9-tokio-telemetry
  dependency-version: 0.3.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: opentelemetry
  dependency-version: 0.32.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: opentelemetry-otlp
  dependency-version: 0.32.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: opentelemetry_sdk
  dependency-version: 0.32.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: opentelemetry-semantic-conventions
  dependency-version: 0.32.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: opentelemetry-stdout
  dependency-version: 0.32.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

* chore(deps): bump workspace dependency versions

* chore(deps): refresh lockfile windows crate graph

* chore(bench): align snapshot mode labels with coverage

* chore(bench): clarify tested snapshot modes

* fix(review): address PR2890 Copilot comments

---------

Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-05-19 04:58:42 +00:00
Henry Guo 17ae9f34c7 fix(notify): accept case-insensitive filter rule names (#2990) 2026-05-18 03:41:48 +00:00
houseme 6e12289339 fix(runtime): finalize issue 2941 profiling cleanup (#2983) 
* perf(runtime): narrow profiling support and upgrade starshard

* style(notify): normalize starshard imports

* perf(ecstore): reduce list_path_raw coordination overhead

* docs(scripts): add issue 2941 perf capture workflow

* fix(runtime): finalize issue 2941 profiling cleanup

* build(deps): bump quick-xml to 0.40.0

* chore(scripts): untrack local perf capture guide

* fix(scripts): honor label in perf capture output
 2026-05-16 11:09:04 +00:00
Henry Guo 11be278c40 test(notify): cover prefix suffix event dispatch (#2960) 2026-05-14 05:38:38 +00:00
houseme 81754d80b3 refactor(targets): unify queue/connectivity handling and coverage (#2953) 
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: marshawcoco <marshawcoco@gmail.com>
 2026-05-14 04:31:23 +00:00
Henry Guo c1bcee327c fix(notify): keep live listen events active when disabled (#2952) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-05-13 17:10:14 +00:00
安正超 28c9358482 test(notify): cover encoded key target union (#2934) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-05-12 15:20:06 +00:00
Henry Guo 07a01a1123 fix(notify): match filters against decoded event keys (#2921) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-05-11 15:01:37 +00:00
JaySon bafff6d207 feat(targets): add check_mysql_server_available probe function (#2884) 
Signed-off-by: JaySon-Huang <tshent@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-05-10 11:53:45 +00:00
houseme 53ec1b95d8 keep sftp e2e tests buildable (#2897) 2026-05-10 11:52:15 +00:00
houseme 81ad48dac2 feat(targets): add AMQP support for notify and audit (#2879) 
Co-authored-by: Hyesook Yun <74169420+suk13574@users.noreply.github.com>
 2026-05-09 01:56:26 +00:00
houseme 5431b9273d feat(targets): complete redis mysql postgres target wiring (#2842) 
Signed-off-by: jaehanbyun <awbrg789@naver.com>
Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: JaySon-Huang <tshent@qq.com>
Co-authored-by: jaehanbyun <awbrg789@naver.com>
Co-authored-by: Gunther Xing <jiengup@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: JaySon <tshent@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2026-05-07 10:00:59 +00:00
houseme e19b0dd997 fix(notify): improve webhook target diagnostics for env setup (#2810) 2026-05-06 00:28:46 +00:00
houseme b96ccfd110 build(deps): bump the dependencies group with 7 updates (#2676) 2026-04-25 02:15:14 +00:00
houseme 368ef0f16c refactor(targets): unify endpoint source/merge logic and bump rustfs-kafka-async to v1.2.0 (#2654) 
Co-authored-by: Filipe Monteiro <a22407332@alunos.ulht.pt>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: loverustfs <hello@rustfs.com>
 2026-04-23 09:14:36 +00:00
houseme 3ac1d2ab0b fix(security): redact target debug logs and remove eval-based bench hook (#2637) 2026-04-21 21:21:01 +00:00
houseme 3796b684f0 feat(targets): add NATS and Pulsar target support (#2618) 2026-04-21 00:28:19 +00:00
houseme 1cbf156559 refactor(obs): migrate metrics runtime/schema and tighten migration guards (#2584) 2026-04-18 07:51:15 +00:00
Tunglies 579b124726 lint: clippy rules or_fun_call (#2561) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-04-16 02:48:39 +00:00
Tunglies 49366ee200 chore(lint): clippy rules redundant_clone (#2554) 2026-04-15 13:54:07 +00:00
houseme 979626c370 refactor(utils): decouple config deps and move sys helpers (#2520) 
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2026-04-13 21:05:03 +08:00
houseme d6158c0481 feat(mqtt): migrate client and harden TLS config (#2413) 
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-04-07 15:28:29 +08:00
houseme d2901fd78c feat(admin): add audit target APIs and harden target source handling (#2350) 
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2026-04-04 09:07:22 +08:00
Andy Brown c4efb46827 fix(notify): emit delete webhooks for prefix deletes and align replication headers (#2383) 
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2026-04-03 21:09:05 +08:00
houseme 16db18216d fix: populate tagging notification principalId and object metadata (#2342) 2026-03-30 14:32:39 +08:00
weisd 28f57b228c feat(s3): advance parity coverage (#2278) 2026-03-24 17:29:33 +08:00
weisd 05dc131a49 perf(storage): optimize internode RPC transfer path (#2262) 
Co-authored-by: momoda693 <momoda693@gmail.com>
 2026-03-23 17:09:49 +08:00
heihutu 3c28f0a0ba feat(metrics): migrate system monitoring from rustfs-obs to rustfs-metrics (#2242) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-03-20 18:52:33 +08:00
weisd b9b7d86ae4 feat: improve legacy metadata and admin compatibility (#2202) 2026-03-18 21:05:09 +08:00
houseme b5d881f399 fix(notify): Fix XML Filter parsing and add comprehensive tests (#2191) 2026-03-17 23:08:07 +08:00
simon-escapecode f00d01ec2d fix: resolve silent failure in MQTT bucket event notifications (#2112) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-03-11 10:08:30 +08:00
安正超 bb4fbf5ae2 fix(notify): ignore disabled targets when sending events (#2117) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-03-11 00:37:30 +08:00
houseme 60aa47bf61 feat(storage): integrate S3Operation into OperationHelper for unified metrics and audit (#2103) 2026-03-08 17:57:33 +08:00
houseme d17d2083d4 feat(targets): enhance webhook TLS support with custom CA and skip-verify (#1994) 
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
 2026-02-27 21:24:49 +08:00
heihutu 74759b6e99 fix: missing object.key in S3 event notifications for multipart uploads #1609 (#1624) 
Co-authored-by: houseme <housemecn@gmail.com>
 2026-01-27 13:32:56 +08:00
houseme 27480f7625 Refactor Event Admin Handlers and Parallelize Target Status Probes (#1501) 2026-01-14 14:18:02 +08:00
Dominik Gašparić 8009ad5692 Fix event object structure according to AWS rules (#1379) 
Signed-off-by: Dominik Gašparić <56818232+codedoga@users.noreply.github.com>
 2026-01-05 01:51:14 +08:00
houseme 8d7cd4cb1b chore: upgrade dependencies and migrate to aws-lc-rs (#1333) 2026-01-02 00:02:34 +08:00
houseme 08f1a31f3f Fix notification event stream cleanup, add bounded send concurrency, and reduce overhead (#1224) 2025-12-22 00:57:05 +08:00
yxrxy 3bd96bcf10 fix: resolve event target deletion issue (#1219) 2025-12-21 12:43:48 +08:00
houseme 4abfc9f554 Fix/fix event 1216 (#1191) 
Signed-off-by: loverustfs <hello@rustfs.com>
Co-authored-by: loverustfs <hello@rustfs.com>
 2025-12-19 12:07:07 +08:00
mythrnr 94d5b1c1e4 fix: format of bucket event notifications (#1138) 2025-12-16 20:44:57 +08:00
houseme 1a4e95e940 chore: remove unused dependencies to optimize build (#1117) 2025-12-11 18:13:26 +08:00
yihong 834025d9e3 docs: fix some dead link (#1053) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-12-08 11:23:24 +08:00
houseme 51584986e1 feat(obs): unify metrics initialization and fix exporter move error (#851) 
* feat(obs): unify metrics initialization and fix exporter move error

- Fix Rust E0382 (use after move) by removing duplicate MetricExporter consumption.
- Consolidate MeterProvider construction into single Recorder builder path.
- Remove redundant Recorder::builder(...).install_global() call.
- Ensure PeriodicReader setup is performed only once (HTTP + optional stdout).
- Set global meter provider and metrics recorder exactly once.
- Preserve existing behavior for stdout/file vs HTTP modes.
- Minor cleanup: consistent resource reuse and interval handling.

* update telemetry.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix

* fix

* fix

* fix: modify logger level from error to event

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-11-14 00:50:07 +08:00
houseme 98be7df0f5 feat(storage): refactor audit and notification with OperationHelper (#825) 
* improve code for audit

* improve code ecfs.rs

* improve code

* improve code for ecfs.rs

* feat(storage): refactor audit and notification with OperationHelper

This commit introduces a significant refactoring of the audit logging and event notification mechanisms within `ecfs.rs`.

The core of this change is the new `OperationHelper` struct, which encapsulates and simplifies the logic for both concerns. It replaces the previous `AuditHelper` and manual event dispatching.

Key improvements include:

- **Unified Handling**: `OperationHelper` manages both audit and notification builders, providing a single, consistent entry point for S3 operations.
- **RAII for Automation**: By leveraging the `Drop` trait, the helper automatically dispatches logs and notifications when it goes out of scope. This simplifies S3 method implementations and ensures cleanup even on early returns.
- **Fluent API**: A builder-like pattern with methods such as `.object()`, `.version_id()`, and `.suppress_event()` makes the code more readable and expressive.
- **Context-Aware Logic**: The helper's `.complete()` method intelligently populates log details based on the operation's `S3Result` and only triggers notifications on success.
- **Modular Design**: All helper logic is now isolated in `rustfs/src/storage/helper.rs`, improving separation of concerns and making `ecfs.rs` cleaner.

This refactoring significantly enhances code clarity, reduces boilerplate, and improves the robustness of logging and notification handling across the storage layer.

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* improve code for audit and notify

* fix

* fix

* fix
 2025-11-10 17:30:50 +08:00