46e43f608f
feat(observability): add Grafana dashboard for the object data cache ( #4761 )
...
The GET body cache exports 11 `rustfs_object_data_cache_*` metrics but no
bundled dashboard visualized them. This adds one so operators can see the
cache's behaviour without hand-writing PromQL.
New `grafana-object-data-cache.json` (auto-provisioned from the dashboards
directory, `${DS_PROMETHEUS}`, schema 38) with 19 panels covering every metric:
hit ratio and lookup outcomes, plan decisions and cacheable ratio, fill
outcomes and fill-duration quantiles, hit-vs-fill byte throughput, entries and
weighted bytes vs capacity, in-flight fills, memory-pressure skips,
invalidations by reason/outcome, and size-class breakdowns. PromQL matches each
metric's type — rate() for counters, histogram_quantile() for the fill-duration
histogram, direct reads for gauges.
The observability README (EN + ZH) dashboard table gains a matching row.
Refs: backlog#1107
Co-authored-by: heihutu <heihutu@gmail.com >
2026-07-12 11:45:32 +08:00
27468ebfa9
feat(get): consolidate GET performance optimization ( #3972 )
...
* feat(get): consolidate GET performance optimization
Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.
## Changes
### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support
### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
- delete_marker_votes tracking
- requested_version_id and matching_version_votes tracking
- version_early_stop_decision() method
- 6 new tests for version early-stop scenarios
### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default
### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration
### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)
### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label
### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants
### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs
### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |
## Rollback
All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(get): add stress test scripts for GET optimization validation
- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(ecstore): align file cache reclaim defaults
* chore(deps): update redis and erasure codec
* test(ecstore): align decode fill policy default
* test(ecstore): align metadata early-stop default
* fix(ecstore): keep metadata early stop opt-in
---------
Co-authored-by: heihutu <heihutu@gmail.com >
2026-06-28 07:14:07 +08:00
25c6bdf490
perf(filemeta): phase-1~3 rename_data metadata optimization ( #3011 )
...
* chore(perf): harden amd64 profiling benchmark flow
* fix(profiling): isolate bench buckets and map protobuf conflict
* perf: avoid blocking owned local writes
* style: format profile admin handler
* docs: clarify observability trace validation
* perf: reduce mkdir overhead on local writes
* perf: add rename_data meta microbenchmark
* perf(filemeta): fast-path data_dir decode in version meta
* perf(filemeta): collapse data-dir lookup into one scan
* perf(filemeta): reduce scan allocs and refresh meta bench
* perf(ecstore): skip mkdir path on read-only open
* perf(filemeta): single-pass unshared data-dir scan
* perf(filemeta): add two-key inline remove fast path
* perf(filemeta): compare remove-two keys by bytes first
* bench(ecstore): add remove_two-only micro benchmark
* bench(ecstore): stabilize rename_data meta benchmark timing
* bench(ecstore): align rename_data path with remove_two
* perf(filemeta): avoid uuid string alloc in remove_two
* perf(filemeta): add fast-path for empty inline data
* perf(filemeta): streamline add_version match branch
* perf(filemeta): fast-return remove_key on miss
* perf(filemeta): speed up add_version insertion lookup
* style(ecstore): normalize formatting in perf-tuning files
* refactor(filemeta): unify inline data removal paths
2026-05-19 10:20:24 +00:00
13b4500212
feat(obs): improve telemetry stack, replication metrics, and Grafana alignment ( #2672 )
...
Co-authored-by: Filipe Monteiro <a22407332@alunos.ulht.pt >
Co-authored-by: cxymds <Cxymds@qq.com >
Co-authored-by: weisd <im@weisd.in >
Co-authored-by: loverustfs <hello@rustfs.com >
Co-authored-by: 安正超 <anzhengchao@gmail.com >
2026-04-24 13:50:17 +00:00
fd1b903531
fix(obs): Update observability docker compose stack ( #2010 )
2026-03-01 03:03:50 +08:00
d983638391
build: update docker config and refine s3s region handling ( #1976 )
...
Co-authored-by: houseme <housemecn@gmail.com >
2026-02-27 01:21:12 +08:00
4411c625e2
feat(metrics): async collection with configurable intervals & graceful shutdown ( #1768 )
2026-02-10 21:37:24 +08:00
43df8b6927
improve readme.md
2025-04-28 14:37:28 +08:00
d12817a772
upgrade docker images
2025-03-27 22:21:10 +08:00
4795638763
feat(observability): add obs_config option and document stdout export
...
- Add obs_config parameter to config struct with default path
- Document how to modify use_stdout value in README.md
- Support configuring observability via file or environment variables
This change helps users configure telemetry output destination for better
observability options in different deployment scenarios.
2025-03-18 16:39:18 +08:00
c3ecfeae6c
improve logger entry for Observability
2025-03-18 16:36:23 +08:00
houseme