792f2ef204
docs(kms): add observability dashboard, alert rules and runbook ( #5517 )
...
Add a Grafana dashboard for the four KMS backend operation metrics
emitted at the policy choke point, Prometheus alert rules with
conservative default thresholds (pending staging baseline calibration),
and an operations runbook documenting the metric contract and the
response procedure for each alert. Register the new alert rules file in
the observability READMEs.
Refs rustfs/backlog#1584 (part of rustfs/backlog#1562 )
2026-07-31 21:28:10 +00:00
46e43f608f
feat(observability): add Grafana dashboard for the object data cache ( #4761 )
...
The GET body cache exports 11 `rustfs_object_data_cache_*` metrics but no
bundled dashboard visualized them. This adds one so operators can see the
cache's behaviour without hand-writing PromQL.
New `grafana-object-data-cache.json` (auto-provisioned from the dashboards
directory, `${DS_PROMETHEUS}`, schema 38) with 19 panels covering every metric:
hit ratio and lookup outcomes, plan decisions and cacheable ratio, fill
outcomes and fill-duration quantiles, hit-vs-fill byte throughput, entries and
weighted bytes vs capacity, in-flight fills, memory-pressure skips,
invalidations by reason/outcome, and size-class breakdowns. PromQL matches each
metric's type — rate() for counters, histogram_quantile() for the fill-duration
histogram, direct reads for gauges.
The observability README (EN + ZH) dashboard table gains a matching row.
Refs: backlog#1107
Co-authored-by: heihutu <heihutu@gmail.com >
2026-07-12 11:45:32 +08:00
27468ebfa9
feat(get): consolidate GET performance optimization ( #3972 )
...
* feat(get): consolidate GET performance optimization
Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.
## Changes
### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support
### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
- delete_marker_votes tracking
- requested_version_id and matching_version_votes tracking
- version_early_stop_decision() method
- 6 new tests for version early-stop scenarios
### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default
### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration
### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)
### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label
### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants
### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs
### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |
## Rollback
All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(get): add stress test scripts for GET optimization validation
- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(ecstore): align file cache reclaim defaults
* chore(deps): update redis and erasure codec
* test(ecstore): align decode fill policy default
* test(ecstore): align metadata early-stop default
* fix(ecstore): keep metadata early stop opt-in
---------
Co-authored-by: heihutu <heihutu@gmail.com >
2026-06-28 07:14:07 +08:00
25c6bdf490
perf(filemeta): phase-1~3 rename_data metadata optimization ( #3011 )
...
* chore(perf): harden amd64 profiling benchmark flow
* fix(profiling): isolate bench buckets and map protobuf conflict
* perf: avoid blocking owned local writes
* style: format profile admin handler
* docs: clarify observability trace validation
* perf: reduce mkdir overhead on local writes
* perf: add rename_data meta microbenchmark
* perf(filemeta): fast-path data_dir decode in version meta
* perf(filemeta): collapse data-dir lookup into one scan
* perf(filemeta): reduce scan allocs and refresh meta bench
* perf(ecstore): skip mkdir path on read-only open
* perf(filemeta): single-pass unshared data-dir scan
* perf(filemeta): add two-key inline remove fast path
* perf(filemeta): compare remove-two keys by bytes first
* bench(ecstore): add remove_two-only micro benchmark
* bench(ecstore): stabilize rename_data meta benchmark timing
* bench(ecstore): align rename_data path with remove_two
* perf(filemeta): avoid uuid string alloc in remove_two
* perf(filemeta): add fast-path for empty inline data
* perf(filemeta): streamline add_version match branch
* perf(filemeta): fast-return remove_key on miss
* perf(filemeta): speed up add_version insertion lookup
* style(ecstore): normalize formatting in perf-tuning files
* refactor(filemeta): unify inline data removal paths
2026-05-19 10:20:24 +00:00
13b4500212
feat(obs): improve telemetry stack, replication metrics, and Grafana alignment ( #2672 )
...
Co-authored-by: Filipe Monteiro <a22407332@alunos.ulht.pt >
Co-authored-by: cxymds <Cxymds@qq.com >
Co-authored-by: weisd <im@weisd.in >
Co-authored-by: loverustfs <hello@rustfs.com >
Co-authored-by: 安正超 <anzhengchao@gmail.com >
2026-04-24 13:50:17 +00:00
fd1b903531
fix(obs): Update observability docker compose stack ( #2010 )
2026-03-01 03:03:50 +08:00
d983638391
build: update docker config and refine s3s region handling ( #1976 )
...
Co-authored-by: houseme <housemecn@gmail.com >
2026-02-27 01:21:12 +08:00
4411c625e2
feat(metrics): async collection with configurable intervals & graceful shutdown ( #1768 )
2026-02-10 21:37:24 +08:00
43df8b6927
improve readme.md
2025-04-28 14:37:28 +08:00
d12817a772
upgrade docker images
2025-03-27 22:21:10 +08:00
4795638763
feat(observability): add obs_config option and document stdout export
...
- Add obs_config parameter to config struct with default path
- Document how to modify use_stdout value in README.md
- Support configuring observability via file or environment variables
This change helps users configure telemetry output destination for better
observability options in different deployment scenarios.
2025-03-18 16:39:18 +08:00
c3ecfeae6c
improve logger entry for Observability
2025-03-18 16:36:23 +08:00
Zhengchao An