perf(ecstore): add Vec<u8> buffer pool for EC operations (#6538)

* feat(mimalloc): add arena diagnostics and configuration

Based on mimalloc maintainer feedback (microsoft/mimalloc#1372),
add diagnostics to check mimalloc arena configuration at runtime.

Changes:
- Add rustfs-mimalloc-sys to workspace dependencies
- Add log_mimalloc_diagnostics() function to check:
  - arena_max_object_size
  - pagemap_commit status
  - mimalloc version
- Add memory_observability module with mimalloc diagnostics

This helps diagnose why allocations might be going outside arenas,
which is the suspected root cause of futex contention.

Ref: rustfs/backlog#2005
Ref: microsoft/mimalloc#1372

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): add Vec<u8> buffer pool for EC operations

Add a general-purpose buffer pool to reduce Vec<u8> allocations
in hot paths like EC encoding/decoding.

Changes:
- Add BufferPool struct in crates/ecstore/src/erasure/codec/buffer_pool.rs
- Thread-safe pool with capacity-based bucketing (power-of-two)
- Global EC_BUFFER_POOL instance with 16 buffers per bucket
- Add buffer_pool module to codec/mod.rs

Expected impact:
- Reduce heap allocations in EC encode/decode paths
- Avoid memzero overhead (proven 4.8% CPU saving in ShardBufferPool)
- Reduce mimalloc lock contention

Note: Main bottleneck remains mimalloc internal synchronization
(futex 98.64% time). Buffer pool provides modest improvement (+2-5%).

Ref: rustfs/backlog#2005

Co-Authored-By: heihutu <heihutu@gmail.com>

* style: apply cargo fmt to buffer pool and related files

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): add #[allow(dead_code)] to buffer pool

The BufferPool infrastructure is ready but not yet integrated
into the EC hot paths. Add #[allow(dead_code)] with clear
documentation about integration status.

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): integrate BufferPool into bitrot verify path

Replace vec![0; shard_size] with get_ec_buffer() in the bitrot
verification hot path to reduce heap allocations and avoid memzero.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style: apply cargo fmt to buffer pool and bitrot changes

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(ecstore): clean up buffer pool code

- Remove unnecessary #[allow(dead_code)] attributes
- Update module documentation to reflect current integration status
- Simplify code structure

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(runtime): cap default worker threads at 16

Testing showed 16 worker threads outperforms 32+ for 1KiB PUT
workloads due to reduced mimalloc lock contention.

A/B test results (testing 4-node cluster, c=64):
- worker_threads=32: 740 obj/s (baseline)
- worker_threads=16: 785 obj/s (+6.1%)

The default was detect_cores() which returned 32 on our testing
nodes. Cap at 16 for optimal small-object performance.

Ref: rustfs/backlog#2005

Co-Authored-By: heihutu <heihutu@gmail.com>

* style: apply cargo fmt to buffer pool and runtime changes

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): remove unused BufferPool::new() function

The new() function was never used since EC_BUFFER_POOL
initializes directly with with_limits(16).

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): update buffer_pool tests to use with_limits

Replace BufferPool::new() with BufferPool::with_limits(16) in tests
since new() was removed in favor of with_limits().

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: hector <hetor@rustfs.com>
Co-authored-by: heihutu <heihutu@gmail.com>

.docker/observability/prometheus.yml.bak-issue2007-node-exporter-20260823T172248Z

@@ -0,0 +1,84 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+global:
+  scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
+  evaluation_interval: 15s
+  external_labels:
+    cluster: 'rustfs-dev'  # Label to identify the cluster
+    replica: '1'  # Replica identifier
+
+rule_files:
+  - /etc/prometheus/rules/*.yml
+
+scrape_configs:
+  - job_name: 'otel-collector'
+    static_configs:
+      - targets: [ 'otel-collector:8888' ]  # Scrape metrics from Collector
+    scrape_interval: 10s
+
+  - job_name: 'rustfs-app-metrics'
+    static_configs:
+      - targets: [ 'otel-collector:8889' ]  # Application indicators
+    scrape_interval: 15s
+    metric_relabel_configs:
+      - source_labels: [ __name__ ]
+        regex: 'go_.*'
+        action: drop  # Drop Go runtime metrics if not needed
+
+  - job_name: 'tempo'
+    static_configs:
+      - targets: [ 'tempo:3200' ]  # Scrape metrics from Tempo
+
+  - job_name: 'jaeger'
+    static_configs:
+      - targets: [ 'jaeger:14269' ]  # Jaeger admin port (14269 is standard for admin/metrics)
+
+  - job_name: 'loki'
+    static_configs:
+      - targets: [ 'loki:3100' ]
+
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: [ 'localhost:9090' ]
+
+  - job_name: 'vulture'
+    static_configs:
+      - targets:
+          - 'vulture:8080'
+
+otlp:
+  promote_resource_attributes:
+    - service.instance.id
+    - service.name
+    - service.namespace
+    - cloud.availability_zone
+    - cloud.region
+    - container.name
+    - deployment.environment.name
+    - k8s.cluster.name
+    - k8s.container.name
+    - k8s.cronjob.name
+    - k8s.daemonset.name
+    - k8s.deployment.name
+    - k8s.job.name
+    - k8s.namespace.name
+    - k8s.pod.name
+    - k8s.replicaset.name
+    - k8s.statefulset.name
+  translation_strategy: NoUTF8EscapingWithSuffixes
+
+storage:
+  tsdb:
+    out_of_order_time_window: 30m