houseme 873a04aed0 Feature/bucket event notification (#347)
* init event notifer

* feat: implement event notification system

- Add core event notification interfaces
- Support multiple notification backends:
  - Webhook (default)
  - Kafka
  - MQTT
  - HTTP Producer
- Implement configurable event filtering
- Add async event dispatching with backpressure handling
- Provide serialization/deserialization for event payloads

This module enables system events to be published to various endpoints
with consistent delivery guarantees and failure handling.

* feat(event-notifier): improve notification system initialization safety

- Add READY atomic flag to track full initialization status
- Implement initialize_safe and start_safe methods with mutex protection
- Add wait_until_ready function with configurable timeout
- Create initialize_and_start_with_ready_check helper method
- Replace sleep-based waiting with proper readiness checks
- Add safety checks before sending events
- Replace chrono with std::time for time handling
- Update error handling to provide clear initialization status

This change reduces race conditions in multi-threaded environments
and ensures events are only processed when the system is fully ready.

* fix sql

Signed-off-by: junxiang Mu <1948535941@qq.com>

* move protobuf generate into bin crate

Signed-off-by: junxiang Mu <1948535941@qq.com>

* improve Cargo toml

* improve code

* improve code for global

* improve code

* feat(event-notifier): improve environment variable handling

- Fix deserialization error when parsing config from environment variables
- Add proper array format support for adapters configuration
- Update environment variable examples with correct format
- Improve documentation for configuration loading
- Implement helper functions for environment variable validation

This change fixes the "invalid type: map, expected a sequence" error
by ensuring proper formatting of array-type fields in environment variables.

* feat: integrate event-notifier system with rustfs

- Rename package from rustfs-event-notifier to event-notifier for consistency
- Add shutdown hooks for event notification system in main process
- Handle graceful termination of notification services on server shutdown
- Implement initialization and configuration loading for event notification
- Fix environment variable configuration to properly parse adapter arrays
- Update example code to demonstrate proper configuration usage

This change ensures proper integration between rustfs and the event
notification system, with clean startup and shutdown sequences to prevent
resource leaks during application lifecycle.

* feat: improve webhook server and run script integration

- Enhance webhook example with proper shutdown handling using tokio::select!
- Update run.sh to automatically start webhook server alongside main service
- Add event notification configuration to run.sh using environment variables
- Set proper port bindings to ensure webhook server starts on port 3000
- Improve console output for better debugging experience
- Fix race condition during service startup and shutdown

This change ensures proper integration between the webhook server and
the main rustfs service, providing a seamless development experience
with automatic service discovery and clean termination.

* improve for logger

* improve code for global.rs

* fix: modify webhook port

* fix

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
2025-04-22 23:08:03 +08:00
2025-03-14 23:26:54 +08:00
2025-04-08 22:32:24 +08:00
2025-04-22 23:06:43 +08:00
2025-04-11 20:53:51 +08:00
2025-04-22 22:59:31 +08:00
2025-04-09 02:41:06 +00:00
2025-04-11 20:53:51 +08:00
2025-04-11 20:53:51 +08:00
2025-04-21 15:27:58 +08:00
fix
2025-04-22 23:07:13 +08:00
2025-04-22 09:14:09 +08:00
2025-03-14 23:26:54 +08:00
2025-04-03 01:59:28 +08:00
2025-03-11 16:12:34 +08:00
2024-07-01 19:33:20 +08:00
2025-03-20 01:16:55 +08:00

How to compile RustFS

Must package Version download link
Rust 1.8.5 https://www.rust-lang.org/tools/install
protoc 30.2 protoc-30.2-linux-x86_64.zip
flatc 24.0+ Linux.flatc.binary.g++-13.zip

Download Links:

https://github.com/google/flatbuffers/releases/download/v25.2.10/Linux.flatc.binary.g++-13.zip

https://github.com/protocolbuffers/protobuf/releases/download/v30.2/protoc-30.2-linux-x86_64.zip

generate protobuf code:

cargo run --bin gproto

Or use Docker:

- uses: arduino/setup-protoc@v3
  with:
  version: "30.2"

- uses: Nugine/setup-flatc@v1
  with:
  version: "25.2.10"

How to add Console web

  1. wget https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip

  2. mkdir in this repos folder ./rustfs/static

  3. Compile RustFS

Star RustFS

Add Env Information:

export RUSTFS_VOLUMES="./target/volume/test"
export RUSTFS_ADDRESS="0.0.0.0:9000"
export RUSTFS_CONSOLE_ENABLE=true
export RUSTFS_CONSOLE_ADDRESS="0.0.0.0:9001"
# 具体路径修改为配置文件真实路径,obs.example.toml 仅供参考 其中`RUSTFS_OBS_CONFIG` 和下面变量二选一
export RUSTFS_OBS_CONFIG="./deploy/config/obs.example.toml"

# 如下变量需要必须参数都有值才可以,以及会覆盖配置文件`obs.example.toml`中的值
export RUSTFS__OBSERVABILITY__ENDPOINT=http://localhost:4317
export RUSTFS__OBSERVABILITY__USE_STDOUT=true
export RUSTFS__OBSERVABILITY__SAMPLE_RATIO=2.0
export RUSTFS__OBSERVABILITY__METER_INTERVAL=30
export RUSTFS__OBSERVABILITY__SERVICE_NAME=rustfs
export RUSTFS__OBSERVABILITY__SERVICE_VERSION=0.1.0
export RUSTFS__OBSERVABILITY__ENVIRONMENT=develop
export RUSTFS__OBSERVABILITY__LOGGER_LEVEL=info
export RUSTFS__SINKS__FILE__ENABLED=true
export RUSTFS__SINKS__FILE__PATH="./deploy/logs/rustfs.log"
export RUSTFS__SINKS__WEBHOOK__ENABLED=false
export RUSTFS__SINKS__WEBHOOK__ENDPOINT=""
export RUSTFS__SINKS__WEBHOOK__AUTH_TOKEN=""
export RUSTFS__SINKS__KAFKA__ENABLED=false
export RUSTFS__SINKS__KAFKA__BOOTSTRAP_SERVERS=""
export RUSTFS__SINKS__KAFKA__TOPIC=""
export RUSTFS__LOGGER__QUEUE_CAPACITY=10

You need replace your real data folder:

./rustfs /data/rustfs

How to deploy the observability stack

The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends.

  1. Enter the .docker/observability directory,
  2. Run the following command:
docker-compose -f docker-compose.yml up -d
  1. Access the Grafana dashboard by navigating to http://localhost:3000 in your browser. The default username and password are admin and admin, respectively.

  2. Access the Jaeger dashboard by navigating to http://localhost:16686 in your browser.

  3. Access the Prometheus dashboard by navigating to http://localhost:9090 in your browser.

Create a new Observability configuration file

1. Enter the deploy/config directory,

2. Copy obs.toml.example to obs.toml

3. Modify the obs.toml configuration file

3.1. Modify the endpoint value to the address of the OpenTelemetry Collector
3.2. Modify the service_name value to the name of the service
3.3. Modify the service_version value to the version of the service
3.4. Modify the environment value to the environment of the service
3.5. Modify the meter_interval value to export interval
3.6. Modify the sample_ratio value to the sample ratio
3.7. Modify the use_stdout value to export to stdout
3.8. Modify the logger_level value to the logger level
S
Description
2.3x faster than MinIO for 4KB object payloads. RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.
Readme Apache-2.0 111 MiB
Languages
Rust 94.7%
Shell 4.1%
Python 1%