Commit Graph

23 Commits

Author SHA1 Message Date
houseme 51584986e1 feat(obs): unify metrics initialization and fix exporter move error (#851)
* feat(obs): unify metrics initialization and fix exporter move error

- Fix Rust E0382 (use after move) by removing duplicate MetricExporter consumption.
- Consolidate MeterProvider construction into single Recorder builder path.
- Remove redundant Recorder::builder(...).install_global() call.
- Ensure PeriodicReader setup is performed only once (HTTP + optional stdout).
- Set global meter provider and metrics recorder exactly once.
- Preserve existing behavior for stdout/file vs HTTP modes.
- Minor cleanup: consistent resource reuse and interval handling.

* update telemetry.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix

* fix

* fix

* fix: modify logger level from error to event

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-11-14 00:50:07 +08:00
houseme 29056a767a Refactor Telemetry Initialization and Environment Utilities (#811)
* improve code for metrics

* improve code for metrics

* fix

* fix

* Refactor telemetry initialization and environment functions ordering

- Reorder functions in envs.rs by type size (8-bit to 64-bit, signed before unsigned) and add missing variants like get_env_opt_u16.
- Optimize init_telemetry to support three modes: stdout logging (default error level with span tracing), file rolling logs (size-based with retention), and HTTP-based observability with sub-endpoints (trace, metric, log) falling back to unified endpoint.
- Fix stdout logging issue by retaining WorkerGuard in OtelGuard to prevent premature release of async writer threads.
- Enhance observability mode with HTTP protocol, compression, and proper resource management.
- Update OtelGuard to include tracing_guard for stdout and flexi_logger_handles for file logging.
- Improve error handling and configuration extraction in OtelConfig.

* fix

* up

* fix

* fix

* improve code for obs

* fix

* fix
2025-11-07 20:01:54 +08:00
houseme d934e3905b Refactor telemetry initialization for non-production environments (#789)
* add dep `scopeguard`

* improve for tracing

* fix

* fix

* improve code for import

* add logger trace id

* fix

* fix

* fix

* fix

* fix
2025-11-05 00:55:08 +08:00
houseme 0714c7a9ca modify logger level from info to error (#744)
* modify logger level from `info` to `error`

* fix test

* improve tokio runtime config

* add rustfs helm chart files (#747)

* add rustfs helm chart files

* update readme file with helm chart

* delete helm chart license file

* fix typo in readme file

* fix: restore localized samples in tests (#749)

* fix: restore required localized examples

* style: fix formatting issues

* improve code for Observability

* upgrade crates version

* fix

* up

* fix

---------

Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2025-10-29 19:20:53 +08:00
houseme e22b24684f chore: bump dependencies, add metrics support, remove DNS resolver (#699)
* upgrade version

* add metrics

* remove dns resolver

* add metrics counter for create bucket

* fix

* fix

* fix
2025-10-24 00:16:17 +08:00
houseme 42d3645d6f fix(targets): make target removal and reload transactional; prevent reappearing entries (#662)
* feat: improve code for notify

* upgrade starshard version

* upgrade version

* Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)

* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>

* Improve lock (#596)

* improve lock

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <1948535941@qq.com>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <1948535941@qq.com>

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(append): implement object append operations with state tracking (#599)

* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: rebase

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>

* build(deps): upgrade s3s (#595)

Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* fix: validate mqtt broker

* improve code for `import`

* fix

* improve

* remove logger from `rustfs-obs` crate

* remove code for config Observability

* fix

* improve code

* fix comment

* up

* up

* upgrade version

* fix

* fmt

* upgrade tokio version to 1.48.0

* upgrade `datafusion` and `reed-solomon-simd` version

* fix

* fmt

* improve code for notify webhook example

* improve code

* fix

* fix

* fmt

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: Nugine <nugine@foxmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-10-17 15:34:53 +08:00
houseme a12a3bedc3 feat(obs): optimize WriteMode selection logic in init_telemetry (#546)
- Refactor WriteMode selection to ensure all variables moved into thread closures are owned types, preventing lifetime issues.
- Simplify and clarify WriteMode assignment for production and non-production environments.
- Improve code readability and maintainability for logger initialization.
2025-09-16 08:25:37 +08:00
houseme 6c37e1cb2a refactor: replace lazy_static with LazyLock (#318)
* refactor: replace `lazy_static` with `LazyLock`

Replace `lazy_static` with `LazyLock`.

Compile time may reduce a little.

See https://github.com/rust-lang-nursery/lazy-static.rs/issues/214

* fmt

* fix
2025-07-31 14:25:39 +08:00
houseme c5df1f92c2 refactor: replace lazy_static with LazyLock and notify crate registry create_targets_from_config (#311)
* improve code for notify

* improve code for logger and fix typo (#272)

* Add GNU to  build.yml (#275)

* fix unzip error

* fix url change error

fix url change error

* Simplify user experience and integrate console and endpoint

Simplify user experience and integrate console and endpoint

* Add gnu to  build.yml

* upgrade version

* feat: add `cargo clippy --fix --allow-dirty` to pre-commit command (#282)

Resolves #277

- Add --fix flag to automatically fix clippy warnings
- Add --allow-dirty flag to run on dirty Git trees
- Improves code quality in pre-commit workflow

* fix: the issue where preview fails when the path length exceeds 255 characters (#280)

* fix

* fix: improve Windows build support and CI/CD workflow (#283)

- Fix Windows zip command issue by using PowerShell Compress-Archive
- Add Windows support for OSS upload with ossutil
- Replace Chinese comments with English in build.yml
- Fix bash syntax error in package_zip function
- Improve code formatting and consistency
- Update various configuration files for better cross-platform support

Resolves Windows build failures in GitHub Actions.

* fix: update link in README.md leading to a 404 error (#285)

* add rustfs.spec for rustfs (#103)

add support on loongarch64

* improve cargo.lock

* build(deps): bump the dependencies group with 5 updates (#289)

Bumps the dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [hyper-util](https://github.com/hyperium/hyper-util) | `0.1.15` | `0.1.16` |
| [rand](https://github.com/rust-random/rand) | `0.9.1` | `0.9.2` |
| [serde_json](https://github.com/serde-rs/json) | `1.0.140` | `1.0.141` |
| [strum](https://github.com/Peternator7/strum) | `0.27.1` | `0.27.2` |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.36.0` | `0.36.1` |


Updates `hyper-util` from 0.1.15 to 0.1.16
- [Release notes](https://github.com/hyperium/hyper-util/releases)
- [Changelog](https://github.com/hyperium/hyper-util/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hyperium/hyper-util/compare/v0.1.15...v0.1.16)

Updates `rand` from 0.9.1 to 0.9.2
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/rand_core-0.9.1...rand_core-0.9.2)

Updates `serde_json` from 1.0.140 to 1.0.141
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.140...v1.0.141)

Updates `strum` from 0.27.1 to 0.27.2
- [Release notes](https://github.com/Peternator7/strum/releases)
- [Changelog](https://github.com/Peternator7/strum/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Peternator7/strum/compare/v0.27.1...v0.27.2)

Updates `sysinfo` from 0.36.0 to 0.36.1
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.36.0...v0.36.1)

---
updated-dependencies:
- dependency-name: hyper-util
  dependency-version: 0.1.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rand
  dependency-version: 0.9.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: serde_json
  dependency-version: 1.0.141
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: strum
  dependency-version: 0.27.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: sysinfo
  dependency-version: 0.36.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* improve code for logger

* improve

* upgrade

* refactor: 优化构建工作流,统一 latest 文件处理和简化制品上传 (#293)

* Refactor: DatabaseManagerSystem as global

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix: fmt

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Test: add e2e_test for s3select

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Test: add test script for e2e

Signed-off-by: junxiang Mu <1948535941@qq.com>

* improve code for registry and intergation

* improve code for registry `create_targets_from_config`

* fix

* Feature up/ilm (#305)

* fix

* fix

* fix

* fix delete-marker expiration. add api_restore.

* fix

* time retry object upload

* lock file

* make fmt

* fix

* restore object

* fix

* fix

* serde-rs-xml -> quick-xml

* fix

* checksum

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* transfer lang to english

* upgrade clap version from 4.5.41 to 4.5.42

* refactor: replace `lazy_static` with `LazyLock`

* add router

* fix: modify comment

* improve code

* fix typos

* fix

* fix: modify name and fmt

* improve code for registry

* fix test

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: Marco Orlandin <mipnamic@mipnamic.net>
Co-authored-by: zhangwenlong <zhangwenlong@loongson.cn>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: likewu <likewu@126.com>
2025-07-30 19:02:10 +08:00
houseme 36acb5bce9 feat(console): Enhance network address handling for WebUI (#129)
* add crates homepage,description,keywords,categories,documentation

* add readme

* modify version 0.0.3

* cargo fmt

* fix: yaml.docker-compose.security.no-new-privileges.no-new-privileges-docker-compose.yml (#63)

* Feature up/ilm (#61)

* fix delete-marker expiration. add api_restore.

* remove target return 204

* log level

* fix: make lint build and clippy happy (#71)

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: make ci and local use the same toolchain (#72)

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* feat: optimize GitHub Actions workflows with performance improvements (#77)

* feat: optimize GitHub Actions workflows with performance improvements

- Rename workflows with more descriptive names
- Add unified setup action for consistent environment setup
- Optimize caching strategy with Swatinem/rust-cache@v2
- Implement skip-check mechanism to avoid duplicate builds
- Simplify matrix builds with better include/exclude logic
- Add intelligent build strategy checks
- Optimize Docker multi-arch builds
- Improve artifact naming and retention
- Add performance testing with benchmark support
- Enhance security audit with dependency scanning
- Change Chinese comments to English for better maintainability

Performance improvements:
- CI testing: ~35 min (42% faster)
- Build release: ~60 min (50% faster)
- Docker builds: ~45 min (50% faster)
- Security audit: ~8 min (47% faster)

* fix: correct secrets context usage in GitHub Actions workflow

- Move environment variables to job level to fix secrets access issue
- Fix unrecognized named-value 'secrets' error in if condition
- Ensure OSS upload step can properly check for required secrets

* fix: resolve GitHub API rate limit by adding authentication token

- Add github-token input to setup action to authenticate GitHub API requests
- Pass GITHUB_TOKEN to all setup action usages to avoid rate limiting
- Fix arduino/setup-protoc@v3 API access issues in CI/CD workflows
- Ensure protoc installation can successfully access GitHub releases API

* fix:make bucket err (#85)

* Rename DEVELOPMENT.md to CONTRIBUTING.md

* Create issue-translator.yml (#89)

Enable Issues Translator

* fix(dockerfile): correct env variable names for access/secret key and improve compatibility (#90)

* fix: restore Zig and cargo-zigbuild caching in GitHub Actions setup action (#92)

* fix: restore Zig and cargo-zigbuild caching in GitHub Actions setup action

Use mlugg/setup-zig and taiki-e/cache-cargo-install-action to speed up cross-compilation tool installation and avoid repeated downloads. All comments and code are in English.

* fix: use correct taiki-e/install-action for cargo-zigbuild

Use taiki-e/install-action@cargo-zigbuild instead of taiki-e/cache-cargo-install-action@v2 to match the original implementation from PR #77.

* refactor: remove explicit Zig version to use latest stable

* Create CODE_OF_CONDUCT.md

* Create SECURITY.md

* Update issue templates

* Create CLA.md

* docs: update PR template to English version

* fix: improve data scanner random sleep calculation

- Fix random number generation API usage
- Adjust sleep calculation to follow MinIO pattern
- Ensure proper random range for scanner cycles

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix: soupprt ipv6

* improve log

* add client ip log

* Update rustfs/src/console.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* improve code

* feat: unify package format to zip for all platforms

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: kira-offgrid <kira@offgridsec.com>
Co-authored-by: likewu <likewu@126.com>
Co-authored-by: laoliu <lygn128@163.com>
Co-authored-by: yihong <zouzou0208@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: Yone <zhiyu@live.cn>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-09 14:39:40 +08:00
houseme 2e14b32ccd chore: Add copyright and license headers (#23)
* chore: Add copyright and license headers

This commit adds the Apache 2.0 license and a copyright notice to the header of all source files. This ensures that the licensing and copyright information is clearly stated within the codebase.

* cargo fmt

* fix

* fmt

* fix clippy
2025-07-02 15:07:47 +08:00
houseme 28c71cc351 fix 2025-06-09 12:25:56 +08:00
houseme 690ce7fae4 feat(obs): upgrade OpenTelemetry dependencies to latest version (#447)
- Update opentelemetry from 0.29.1 to 0.30.0
- Update related opentelemetry dependencies for compatibility
- Ensure compatibility with existing observability implementation
- Improve tracing and metrics collection capabilities

This upgrade provides better performance and stability for our observability stack.
2025-06-07 22:22:26 +08:00
houseme 77ddd1cfe0 Merge branch 'main' of github.com:rustfs/s3-rustfs into feature/observability-metrics
# Conflicts:
#	Cargo.lock
#	Cargo.toml
2025-06-06 15:52:32 +08:00
houseme 6302764512 improve code for reqwest feature 2025-06-06 14:35:40 +08:00
houseme a394a3163f upgrade version 2025-06-06 13:49:31 +08:00
houseme 7f983e72e7 fix: must_use 2025-05-30 15:11:51 +08:00
houseme 3ceec7b5f2 improve code for obs 2025-05-29 21:10:04 +08:00
houseme 571cedf4ce feat(obs): implement global OpenTelemetry guard management 2025-05-12 13:32:18 +08:00
houseme dd7da015e3 Feature/rustfs config (#396)
* init rustfs config

* improve code for rustfs-config crate

* add

* improve code for comment

* fix: modify rustfs-config crate name

* add default fn

* improve error logger

* fix: modify docker config yaml

* improve code for config

* feat: restrict kafka feature to Linux only

- Add target-specific feature configuration in Cargo.toml for obs and event-notifier crates
- Implement conditional compilation for kafka feature only on Linux systems
- Add appropriate error handling for non-Linux platforms
- Ensure backward compatibility with existing code

* refactor(ci): optimize build workflow for better efficiency

- Integrate GUI build steps into main build-rustfs job
- Add conditional GUI build execution based on tag releases
- Simplify workflow by removing redundant build-rustfs-gui job
- Copy binary directly to embedded-rustfs directory without downloading artifacts
- Update merge job dependency to only rely on build-rustfs
- Improve cross-platform compatibility for Windows binary naming (.exe)
- Streamline artifact uploading and OSS publishing process
- Maintain consistent conditional logic for release operations

* refactor(ci): optimize build workflow for better efficiency

- Integrate GUI build steps into main build-rustfs job
- Add conditional GUI build execution based on tag releases
- Simplify workflow by removing redundant build-rustfs-gui job
- Copy binary directly to embedded-rustfs directory without downloading artifacts
- Update merge job dependency to only rely on build-rustfs
- Improve cross-platform compatibility for Windows binary naming (.exe)
- Streamline artifact uploading and OSS publishing process
- Maintain consistent conditional logic for release operations

* fix(ci): add repo-token to setup-protoc action for authentication

- Add GITHUB_TOKEN parameter to arduino/setup-protoc@v3 action
- Ensure proper authentication for Protoc installation in CI workflow
- Maintain consistent setup across different CI environments

* modify config

* improve readme.md

* remove env config relation

* add allow(dead_code)
2025-05-12 01:17:31 +08:00
houseme e8f8a0872d modify Telemetry filter order 2025-04-27 09:50:32 +08:00
houseme 86353d98d5 feat: add TraceLayer for HTTP service and improve metrics (#361)
* improve code for opentelemetry and add system metrics

* feat: add TraceLayer for HTTP service and improve metrics

- Add TraceLayer to HTTP server for request tracing
- Implement system metrics for process monitoring
- Optimize init_telemetry method for better resource management
- Add graceful shutdown handling for telemetry components
- Fix GracefulShutdown ownership issues with Arc wrapper

* improve code for init_process_observer

* remove tomlfmt.toml

* Translation comment

* improve code for console CompressionLayer params
2025-04-24 19:04:57 +08:00
houseme a07ca8ae81 improve code for obs 2025-04-09 15:12:31 +08:00