Compare commits

..

2 Commits

Author SHA1 Message Date
overtrue 6befa4d042 ci: correct the companion-workflow comments
The ci-docs-only header claimed the ruleset already requires "Quick Checks";
it does not, and this PR deliberately leaves it that way. Say what is true
today and why the job is mirrored ahead of the ruleset change.

Also move the byte-identical requirement onto ci.yml's quick-checks job, which
is the more likely edit site, instead of pointing at a comment that was not
there.
2026-08-01 10:49:34 +08:00
overtrue 06c70fcd14 ci: stop running expensive jobs that cannot inform the result
Three independent fixes that all avoid burning self-hosted runners on work
whose outcome is already determined. None of them changes what is tested.

- ci-docs-only: add a "Quick Checks" companion job. It is a prerequisite for
  gating ci.yml's expensive jobs behind quick-checks (rustfs/backlog#1599):
  once "Quick Checks" is a required check, a docs-only PR would otherwise wait
  on it forever. The steps are a byte-identical copy of ci.yml's quick-checks
  rather than an `echo`, so that on a mixed PR the two same-named check runs
  execute the same commands against the same merge ref and cannot disagree —
  GitHub has no written contract for how it picks between same-named required
  check runs, and the real job only takes 45-51s, leaving no timing margin to
  rely on.

- ci: guard uring-integration with the same `closed` check every other job
  already has. The pull_request trigger includes `closed` only so the
  concurrency group cancels in-flight runs; this job had no guard and no
  `needs`, so every closed or merged PR ran the full io_uring suite (4m17s,
  7m19s and 7m31s on runs 30678272341, 30678117601 and 30662728539).

- ci: gate s3-lifecycle-behavior-tests on e2e-tests, matching
  s3-implemented-tests. Both lanes only download the prebuilt debug binary, and
  s3-implemented-tests already finishes later, so a green PR's wall clock is
  unchanged; a red one stops holding a sm-standard-4 for up to 30 minutes.

Refs: rustfs/backlog#1598, rustfs/backlog#1599
2026-08-01 10:41:35 +08:00
609 changed files with 43050 additions and 147812 deletions
@@ -1,6 +1,6 @@
---
name: rustfs-logging-governance
description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use whenever a change adds or edits any `tracing` macro call (`error!`/`warn!`/`info!`/`debug!`/`trace!`/`#[instrument]`) — including a single log line added in passing while fixing unrelated logic, which is how most new log sites enter the repo — and when reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use when editing or reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
---
# RustFS Logging Governance
-10
View File
@@ -60,16 +60,6 @@ body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fai
@echo "🧱 Checking body-cache whitelist guard..."
./scripts/check_body_cache_whitelist.sh
.PHONY: s3s-footprint-check
s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
@echo "📦 Checking s3s footprint ratchet..."
./scripts/check_s3s_footprint.sh
.PHONY: fips-wording-check
fips-wording-check: ## Check outward docs do not make unsupported FIPS claims
@echo "📣 Checking FIPS wording guard..."
./scripts/check_fips_wording.sh
.PHONY: log-analyzer-rules-check
log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
@echo "🩺 Checking log-analyzer rule anchors..."
+3 -3
View File
@@ -19,13 +19,13 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
@echo "✅ Fast development checks passed!"
-2
View File
@@ -28,8 +28,6 @@ script-tests: ## Run shell script tests
./scripts/test_entrypoint_credentials.sh
./scripts/test_internode_grpc_ab_bench.sh
./scripts/test_object_batch_bench_enhanced.sh
./scripts/test_hotpath_warp_ab_gate.sh
./scripts/test_hotpath_warp_abba.sh
./scripts/test_exact_1mib_handoff_abba.sh
./scripts/test_pinned_paired_abba_bench.sh
./scripts/test_manual_transition_runbooks.sh
+16 -77
View File
@@ -9,8 +9,6 @@
# * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
# uses the shared multipart fixture and a deterministic uploadId-lock
# handoff, so it must not overlap another process mutating that fixture.
# * bucket::metadata_sys::tests::concurrent_config_writes_from_separate_nodes_do_not_lose_writes
# uses the shared transaction lock and must not overlap other ecstore tests.
#
# serial_test's #[serial] attribute does NOT serialize these across runs:
# nextest executes each test in its own process, where the in-process
@@ -29,8 +27,6 @@
[test-groups]
ecstore-serial-flaky = { max-threads = 1 }
embedded-test-ports = { max-threads = 1 }
e2e-vault = { max-threads = 1 }
# Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
# server and manipulate its disk directories at runtime (crates/e2e_test:
@@ -44,7 +40,7 @@ e2e-inline-boundaries = { max-threads = 1 }
# --- default profile (local): serialize the flaky groups, never retry --------
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
test-group = 'ecstore-serial-flaky'
# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
@@ -56,36 +52,12 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
test-group = 'ecstore-serial-flaky'
# The production-handler relocation regression builds an isolated 8-disk,
# 2-pool store and commits a 72 MiB multipart object. Keep that cross-disk IO
# from overlapping the ecstore commit fixtures above.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
test-group = 'ecstore-serial-flaky'
# Embedded integration-test binaries discover an ephemeral port and release
# the probe listener before RustFS binds it. Serialize that cross-process
# TOCTOU window; retries would only hide real startup failures.
[[profile.default.overrides]]
filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
test-group = 'embedded-test-ports'
# Serialize the durable manual-transition checkpoint test across nextest's
# process boundary; it mutates bucket lifecycle metadata and is not quarantined.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
test-group = 'ecstore-serial-flaky'
# Serialize the bucket-incarnation / lifecycle-fence tests. They drive
# init_bucket_metadata_sys and bucket_metadata_sys_of, i.e. process-global
# OnceLock state that serial_test's #[serial] cannot protect across nextest's
# process boundary, and they delete+recreate buckets — the same shape that
# raced into InsufficientWriteQuorum in backlog#937. Preventive only, no
# retries. The matching ci-profile override is after [profile.ci].
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
test-group = 'ecstore-serial-flaky'
# Serialize the 4-disk reliability / degraded-read e2e tests (see the
# e2e-reliability test-group note above). The matching ci-profile override is at
# the end of the file, after [profile.ci] is declared.
@@ -97,12 +69,6 @@ test-group = 'e2e-reliability'
filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
test-group = 'e2e-inline-boundaries'
# Vault KMS tests share the fixed dev-server port 8200. serial_test's #[serial]
# does not cross nextest process boundaries, so keep these tests in one group.
[[profile.default.overrides]]
filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
test-group = 'e2e-vault'
# ---------------------------------------------------------------------------
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
# ---------------------------------------------------------------------------
@@ -138,9 +104,9 @@ filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(
test-group = 'ecstore-serial-flaky'
retries = 2
# Keep deterministic ECStore write handoffs isolated across nextest processes.
# Keep the deterministic multipart handoff isolated across nextest processes.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
test-group = 'ecstore-serial-flaky'
# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
@@ -165,28 +131,12 @@ test-group = 'e2e-reliability'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
test-group = 'ecstore-serial-flaky'
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
test-group = 'ecstore-serial-flaky'
# Match the default-profile embedded test isolation without quarantining or
# retrying failures in CI.
[[profile.ci.overrides]]
filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
test-group = 'embedded-test-ports'
# Serialize the durable manual-transition checkpoint test under the ci profile
# too. No retries: failures stay visible.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
test-group = 'ecstore-serial-flaky'
# Serialize the bucket-incarnation / lifecycle-fence tests under the ci profile
# too (see the matching default-profile override near the top). No retries.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
test-group = 'ecstore-serial-flaky'
# ---------------------------------------------------------------------------
# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
# ---------------------------------------------------------------------------
@@ -218,7 +168,7 @@ test-group = 'ecstore-serial-flaky'
# the nightly profile derives its set as "the replication module MINUS this
# allowlist", so any new replication test lands in nightly by default (never
# silently unrun) until it is explicitly blessed as fast here. Keep the two
# regexes byte-identical. Count invariant: 20 here + 36 nightly = 56 total
# regexes byte-identical. Count invariant: 20 here + 28 nightly = 48 total
# (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
# (#4724) because they set a loopback (127.0.0.1) replication target that the
@@ -254,7 +204,7 @@ test-group = 'ecstore-serial-flaky'
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
| test(/^reliant::lifecycle::/)
| test(/^reliant::tiering::/)
@@ -262,17 +212,6 @@ default-filter = """
"""
fail-fast = false
[profile.e2e-smoke.junit]
path = "junit.xml"
# The pagination boundary cases can stall when a server/listing regression
# prevents the continuation request from completing. Keep the timeout scoped
# to those known failure modes so legitimate lifecycle/tiering waits retain
# their test-level timing budget.
[[profile.e2e-smoke.overrides]]
filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
# ---------------------------------------------------------------------------
# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
# ---------------------------------------------------------------------------
@@ -280,10 +219,10 @@ slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
# tests that are unfit for the per-PR e2e-smoke gate:
#
# * 2 remote-target TLS validation tests.
# * 13 bucket-replication data-plane/helper tests — they PUT/delete objects
# and poll until source and target converge; two replicate over HTTPS,
# four pin active SSE fail-closed contracts (SSE-C, SSE-S3, SSE-KMS, and
# the SSE-S3 resync path), and one guards event/history observers.
# * 12 bucket-replication data-plane/helper tests — they PUT/delete objects
# and poll until source and target converge; two replicate over HTTPS, two
# pin active SSE failure contracts, and one guards event/history observers.
# The SSE-S3 contract remains ignored under backlog#1291.
# * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
# servers and drives the cross-process site-replication control plane.
# * 1 `_real_three_node` site-replication test.
@@ -349,9 +288,9 @@ path = "junit.xml"
#
# Each e2e test spawns its own single-node rustfs server on a random port with
# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
# 4-disk reliability / degraded-read fault-injection tests and the fixed-port
# Vault tests, both serialized below.
# parallel-safe — the same property e2e-smoke relies on. The exception is the
# 4-disk reliability / degraded-read fault-injection tests, serialized below
# (identical to the ci profile) so several 4-disk servers never run at once.
# KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
# 341 ran / 32 failed on the suites' first automated run ever). Deterministic
# product failures cannot be quarantined away with retries, so each family is
@@ -361,6 +300,9 @@ path = "junit.xml"
# negative-path siblings of each family stay in as regression guards.
# * rustfs#4843 — over-limit archive entry paths hard-reject the whole
# archive even under ignore-errors semantics.
# * rustfs#4846 — distributed-lock quorum tests misclassify as timeout
# under parallel load (multi-node in-process clusters; natural home is
# ci-7's nightly cluster lane).
[profile.e2e-full]
default-filter = """
package(e2e_test)
@@ -369,6 +311,7 @@ default-filter = """
& !test(/^replication_extension_test::/)
& !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
& !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
& !test(/^reliant::lock::test_distributed_lock_(2_nodes_grpc_read_survives_failed_node|4_nodes_grpc_read_write_quorum_split_with_two_failed_nodes)$/)
"""
fail-fast = false
@@ -387,7 +330,3 @@ test-group = 'e2e-reliability'
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
test-group = 'e2e-inline-boundaries'
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
test-group = 'e2e-vault'
-15
View File
@@ -170,10 +170,6 @@ Important behavior notes:
- Logs and metrics usually appear during startup, so seeing those two signals
first is expected.
- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
those attributes as structured metadata, and the Collector also mirrors the
common troubleshooting fields into the log line so simple line filters can
find them.
- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
startup, because request-path spans are created on demand.
- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
@@ -199,17 +195,6 @@ curl -I http://127.0.0.1:9000/health/ready
# Jaeger: http://localhost:16686
```
For a structured RustFS log such as an inter-node RPC authentication failure,
the Loki line now includes fields such as `event`, `component`, `subsystem`,
`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
LogQL checks:
```logql
{service_name="RustFS"} |= "RPC signature verification failed"
{service_name="RustFS"} |= "failure_reason="
{service_name="RustFS"} | failure_reason != ""
```
If logs and metrics are present but traces are sparse, the most common cause is
"no real request traffic yet" or "`info` level filtered nested spans", not an
OTLP routing failure.
-9
View File
@@ -169,7 +169,6 @@ RustFS 会自动在该基础 URL 后补全:
需要注意:
- 启动阶段通常会先看到日志和指标,因此“先有日志/指标、后有 trace”是正常现象。
- OpenTelemetry bridge 会把 `tracing` 字段作为日志 attributes 发送。Loki 会将这些 attributes 存为 structured metadata,同时 Collector 会把常用排障字段镜像进日志行,方便用简单的行内容过滤直接查到。
- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量,因为请求路径上的 span 是按需创建的。
- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span,但会过滤掉很多 `debug` 级别的嵌套 span。
如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏,建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`,再判断是否是 collector 或 Tempo 问题。
@@ -193,14 +192,6 @@ curl -I http://127.0.0.1:9000/health/ready
# Jaeger: http://localhost:16686
```
对于 RustFS 结构化日志,例如节点间 RPC 鉴权失败,Loki 日志行现在会包含 `event``component``subsystem``failure_reason``rpc_service``rpc_method``expected_audience` 等字段。常用 LogQL 检查:
```logql
{service_name="RustFS"} |= "RPC signature verification failed"
{service_name="RustFS"} |= "failure_reason="
{service_name="RustFS"} | failure_reason != ""
```
如果日志和指标已经正常,但 trace 仍然稀疏,最常见的原因通常是
“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”,而不是 OTLP 路由失败。
@@ -11500,831 +11500,6 @@
],
"title": "Compression Operations Rate",
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 332
},
"id": 531,
"panels": [],
"title": "Metrics Dimensions Drilldown",
"type": "row"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
},
"unit": "ops"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 333
},
"id": 532,
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.3.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "A",
"expr": "sum by (server, name, type) (rate(rustfs_api_requests_requests_total_by_server{job=~\"$job\",server=~\"$server\",name=~\"$api\"}[$__rate_interval]))",
"legendFormat": "{{server}} | {{name}} | {{type}}"
}
],
"title": "API Requests by Server and API",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
},
"unit": "s"
},
"overrides": [
{
"matcher": {
"id": "byFrameRefID",
"options": "A"
},
"properties": [
{
"id": "unit",
"value": "none"
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 333
},
"id": 533,
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.3.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "A",
"expr": "max by (server, drive, pool_index, set_index, drive_index, state) (rustfs_system_drive_runtime_state{job=~\"$job\",server=~\"$server\",drive=~\"$drive\"})",
"legendFormat": "{{server}} | {{drive}} | p{{pool_index}}/s{{set_index}}/d{{drive_index}} | {{state}}"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "B",
"expr": "max by (server, drive, pool_index, set_index, drive_index) (rustfs_system_drive_offline_duration_seconds{job=~\"$job\",server=~\"$server\",drive=~\"$drive\"})",
"legendFormat": "{{server}} | {{drive}} | offline seconds"
}
],
"title": "Drive Runtime State and Offline Duration",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
},
"unit": "ops"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 341
},
"id": 534,
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.3.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "A",
"expr": "sum by (server, drive, pool_index, set_index, drive_index, api) (rate(rustfs_system_drive_api_calls_total{job=~\"$job\",server=~\"$server\",drive=~\"$drive\",api=~\"$drive_api\"}[$__rate_interval]))",
"legendFormat": "{{server}} | {{drive}} | p{{pool_index}}/s{{set_index}}/d{{drive_index}} | {{api}}"
}
],
"title": "Drive API Calls by Operation",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
},
"unit": "ops"
},
"overrides": [
{
"matcher": {
"id": "byFrameRefID",
"options": "B"
},
"properties": [
{
"id": "unit",
"value": "short"
},
{
"id": "custom.axisPlacement",
"value": "right"
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 341
},
"id": 535,
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.3.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "A",
"expr": "sum by (server, source, state) (rate(rustfs_scanner_source_work_total{job=~\"$job\",server=~\"$server\",source=~\"$scanner_source\"}[$__rate_interval]))",
"legendFormat": "{{server}} | {{source}} | {{state}}"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "B",
"expr": "sum by (server, cycle_scope, source, state) (rustfs_scanner_cycle_source_work{job=~\"$job\",server=~\"$server\",source=~\"$scanner_source\"})",
"legendFormat": "{{server}} | {{cycle_scope}} | {{source}} | {{state}}"
}
],
"title": "Scanner Source Work by Server",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
},
"unit": "ops"
},
"overrides": [
{
"matcher": {
"id": "byFrameRefID",
"options": "B"
},
"properties": [
{
"id": "unit",
"value": "short"
},
{
"id": "custom.axisPlacement",
"value": "right"
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 349
},
"id": 536,
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.3.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "A",
"expr": "sum by (server, bucket, drive, result) (rate(rustfs_scanner_bucket_drive_result_total{job=~\"$job\",server=~\"$server\",bucket=~\"$bucket\",drive=~\"$drive\",result=~\"$scanner_result\"}[$__rate_interval]))",
"legendFormat": "{{server}} | {{bucket}} | {{drive}} | {{result}}"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "B",
"expr": "sum by (server, cycle_scope, bucket, drive, result) (rustfs_scanner_cycle_bucket_drive_result{job=~\"$job\",server=~\"$server\",bucket=~\"$bucket\",drive=~\"$drive\",result=~\"$scanner_result\"})",
"legendFormat": "{{server}} | {{cycle_scope}} | {{bucket}} | {{drive}} | {{result}}"
}
],
"title": "Scanner Bucket Drive Results",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
},
"unit": "ops"
},
"overrides": [
{
"matcher": {
"id": "byFrameRefID",
"options": "B"
},
"properties": [
{
"id": "unit",
"value": "Bps"
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 349
},
"id": 537,
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.3.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "A",
"expr": "sum by (bucket, target_arn) (rate(rustfs_bucket_replication_target_sent_count{job=~\"$job\",bucket=~\"$bucket\",target_arn=~\"$target_arn\"}[$__rate_interval]))",
"legendFormat": "sent objects | {{bucket}} | {{target_arn}}"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "B",
"expr": "sum by (bucket, target_arn) (rate(rustfs_bucket_replication_target_sent_bytes{job=~\"$job\",bucket=~\"$bucket\",target_arn=~\"$target_arn\"}[$__rate_interval]))",
"legendFormat": "sent bytes | {{bucket}} | {{target_arn}}"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "C",
"expr": "sum by (bucket, target_arn) (rate(rustfs_bucket_replication_target_total_failed_count{job=~\"$job\",bucket=~\"$bucket\",target_arn=~\"$target_arn\"}[$__rate_interval]))",
"legendFormat": "failed objects | {{bucket}} | {{target_arn}}"
}
],
"title": "Bucket Replication Target Flow",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "smooth",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"showValues": false,
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": 0
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 357
},
"id": 538,
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "right",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "12.3.2",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "A",
"expr": "max by (server, target_id) (rustfs_audit_target_queue_length_by_server{job=~\"$job\",server=~\"$server\"})",
"legendFormat": "audit queue | {{server}} | {{target_id}}"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"range": true,
"refId": "B",
"expr": "max by (server, action, state) (rustfs_ilm_action_tasks{job=~\"$job\",server=~\"$server\"})",
"legendFormat": "ilm | {{server}} | {{action}} | {{state}}"
}
],
"title": "Audit and ILM by Server",
"type": "timeseries"
}
],
"preload": false,
@@ -12376,32 +11551,6 @@
"sort": 1,
"type": "query"
},
{
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(rustfs_system_drive_api_calls_total,api)",
"includeAll": true,
"label": "Drive API",
"multi": true,
"name": "drive_api",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(rustfs_system_drive_api_calls_total,api)",
"refId": "PrometheusVariableQueryEditor-drive_api"
},
"refresh": 2,
"regex": "",
"sort": 1,
"type": "query"
},
{
"allValue": ".*",
"current": {
@@ -12521,136 +11670,6 @@
"regex": "",
"sort": 1,
"type": "query"
},
{
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(rustfs_api_requests_requests_total_by_server,server)",
"includeAll": true,
"label": "Server",
"multi": true,
"name": "server",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(rustfs_api_requests_requests_total_by_server,server)",
"refId": "PrometheusVariableQueryEditor-server"
},
"refresh": 2,
"regex": "",
"sort": 1,
"type": "query"
},
{
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(rustfs_api_requests_requests_total_by_server,name)",
"includeAll": true,
"label": "API",
"multi": true,
"name": "api",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(rustfs_api_requests_requests_total_by_server,name)",
"refId": "PrometheusVariableQueryEditor-api"
},
"refresh": 2,
"regex": "",
"sort": 1,
"type": "query"
},
{
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values({__name__=\"rustfs_bucket_replication_target_sent_count\",bucket=~\"$bucket\"},target_arn)",
"includeAll": true,
"label": "Target ARN",
"multi": true,
"name": "target_arn",
"options": [],
"query": {
"qryType": 1,
"query": "label_values({__name__=\"rustfs_bucket_replication_target_sent_count\",bucket=~\"$bucket\"},target_arn)",
"refId": "PrometheusVariableQueryEditor-target_arn"
},
"refresh": 2,
"regex": "",
"sort": 1,
"type": "query"
},
{
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(rustfs_scanner_source_work_total,source)",
"includeAll": true,
"label": "Scanner Source",
"multi": true,
"name": "scanner_source",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(rustfs_scanner_source_work_total,source)",
"refId": "PrometheusVariableQueryEditor-scanner_source"
},
"refresh": 2,
"regex": "",
"sort": 1,
"type": "query"
},
{
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(rustfs_scanner_bucket_drive_result_total,result)",
"includeAll": true,
"label": "Scanner Result",
"multi": true,
"name": "scanner_result",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(rustfs_scanner_bucket_drive_result_total,result)",
"refId": "PrometheusVariableQueryEditor-scanner_result"
},
"refresh": 2,
"regex": "",
"sort": 1,
"type": "query"
}
]
},
@@ -29,27 +29,11 @@ processors:
limit_mib: 1024
spike_limit_mib: 256
transform/logs:
error_mode: ignore
log_statements:
- context: log
statements:
- set(attributes["message"], body.string) where IsString(body)
- set(attributes["log.body"], body.string) where IsString(body)
- set(body, Concat([body, " event=", attributes["event"]], "")) where IsString(body) and attributes["event"] != nil
- set(body, Concat([body, " component=", attributes["component"]], "")) where IsString(body) and attributes["component"] != nil
- set(body, Concat([body, " subsystem=", attributes["subsystem"]], "")) where IsString(body) and attributes["subsystem"] != nil
- set(body, Concat([body, " state=", attributes["state"]], "")) where IsString(body) and attributes["state"] != nil
- set(body, Concat([body, " result=", attributes["result"]], "")) where IsString(body) and attributes["result"] != nil
- set(body, Concat([body, " reason=", attributes["reason"]], "")) where IsString(body) and attributes["reason"] != nil
- set(body, Concat([body, " failure_reason=", attributes["failure_reason"]], "")) where IsString(body) and attributes["failure_reason"] != nil
- set(body, Concat([body, " rpc_path=", attributes["rpc_path"]], "")) where IsString(body) and attributes["rpc_path"] != nil
- set(body, Concat([body, " rpc_service=", attributes["rpc_service"]], "")) where IsString(body) and attributes["rpc_service"] != nil
- set(body, Concat([body, " rpc_method=", attributes["rpc_method"]], "")) where IsString(body) and attributes["rpc_method"] != nil
- set(body, Concat([body, " expected_audience=", attributes["expected_audience"]], "")) where IsString(body) and attributes["expected_audience"] != nil
- set(body, Concat([body, " peer_addr=", attributes["peer_addr"]], "")) where IsString(body) and attributes["peer_addr"] != nil
- set(body, Concat([body, " replay_scope_bootstrap_allowed=", attributes["replay_scope_bootstrap_allowed"]], "")) where IsString(body) and attributes["replay_scope_bootstrap_allowed"] != nil
- set(body, Concat([body, " error=", attributes["error"]], "")) where IsString(body) and attributes["error"] != nil
- set(body, Concat([body, " exception_message=", attributes["exception.message"]], "")) where IsString(body) and attributes["exception.message"] != nil
- set(attributes["message"], body.string)
- set(attributes["log.body"], body.string)
exporters:
otlp/tempo:
@@ -17,9 +17,9 @@
# =============================================================================
#
# Metric source: the KMS operation-policy choke point in
# crates/kms/src/policy.rs. All label values are bounded static strings
# (operation, op_class, outcome, error_class, backend, scope); key identifiers,
# key material, and tokens never appear in labels.
# crates/kms/src/policy.rs. All label values are static enum strings
# (operation, op_class, outcome, error_class); key identifiers, key material,
# and tokens never appear in labels.
#
# Response procedures: docs/operations/kms-observability-runbook.md
#
@@ -70,9 +70,8 @@ groups:
# ------------------------------------------------------------------
# 2. KmsBackendHighErrorRate
# Sustained share of operations terminating without success
# (fatal, budget/deadline exhaustion, admission backpressure,
# or an open circuit). The cancelled outcome is excluded because
# shutdowns legitimately produce it.
# (fatal, budget_exhausted, deadline_exceeded). The cancelled
# outcome is excluded because shutdowns legitimately produce it.
# The traffic guard keeps a single failure on a near-idle
# cluster from firing the alert.
# Threshold: 5% for 10m — conservative default, calibrate
@@ -95,11 +94,9 @@ groups:
summary: "KMS backend non-success ratio above 5% for 10m"
description: >-
{{ $value | humanizePercentage }} of KMS backend operations
are terminating in fatal, budget_exhausted,
deadline_exceeded, backpressure_timeout,
backpressure_rejected, or circuit_open. Object encryption
and decryption paths depending on the KMS are degraded or
failing.
are terminating in fatal, budget_exhausted, or
deadline_exceeded. Object encryption and decryption paths
depending on the KMS are degraded or failing.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
# ==========================================================================
@@ -189,26 +186,3 @@ groups:
Retryable failures are outlasting the retry budget, so
callers are seeing hard failures.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
# ------------------------------------------------------------------
# 6. KmsBackendCircuitOpen
# Direct circuit-state signal, independent of operation traffic.
# A transient open can recover on its first half-open probe; alert
# only when the circuit remains open or half-open for one minute.
# ------------------------------------------------------------------
- alert: KmsBackendCircuitOpen
expr: |
rustfs_kms_backend_circuit_open > 0
for: 1m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend circuit open ({{ $labels.backend }}/{{ $labels.scope }})"
description: >-
The KMS backend circuit for {{ $labels.backend }} scope
{{ $labels.scope }} has remained open or half-open for one
minute. Operations in this scope can terminate as
circuit_open until the half-open probe succeeds or returns
a non-retryable failure.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"
+12 -43
View File
@@ -25,13 +25,9 @@ inputs:
required: false
default: "rustfs-deps"
cache-save-if:
description: >-
Whether to save the cache. The fail-safe default is 'false': a caller that
wants to populate a cache must opt in explicitly, so a forgotten input
costs a cold cache (minutes) rather than silently consuming the
repository-wide 10GB Actions cache quota and evicting other lanes.
description: "Condition for saving cache"
required: false
default: "false"
default: "true"
install-cross-tools:
description: "Install cross-compilation tools"
required: false
@@ -40,43 +36,28 @@ inputs:
description: "Target architecture to add"
required: false
default: ""
install-build-packaging-tools:
description: >-
Install musl-tools/zip/unzip, needed for musl linking and release
packaging. Off for CI test lanes, which use none of them.
github-token:
description: "GitHub token for API access"
required: false
default: "true"
install-test-tools:
description: >-
Install cargo-nextest and the rustfmt/clippy components. Off for release
and audit lanes, which run no tests and no lints.
required: false
default: "true"
default: ""
runs:
using: "composite"
steps:
# protobuf-compiler is deliberately absent: the setup-protoc step below
# installs 34.1 into the tool cache and prepends it to PATH, so the apt
# build (older, and never version-matched) was shadowed on every run and
# simply never used.
- name: Install system dependencies (Ubuntu)
if: runner.os == 'Linux'
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y \
musl-tools \
build-essential \
pkg-config \
libssl-dev \
ripgrep
# musl-gcc is needed by the native musl release leg, and zip/unzip by the
# release packaging steps. No CI test lane touches any of them.
- name: Install packaging and cross-linking dependencies (Ubuntu)
if: runner.os == 'Linux' && inputs.install-build-packaging-tools == 'true'
shell: bash
run: sudo apt-get install -y musl-tools zip unzip
ripgrep \
unzip \
zip \
protobuf-compiler
- name: Install protoc
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
@@ -94,7 +75,7 @@ runs:
with:
toolchain: ${{ inputs.rust-version }}
targets: ${{ inputs.target }}
components: ${{ inputs.install-test-tools == 'true' && 'rustfmt, clippy' || '' }}
components: rustfmt, clippy
- name: Install Zig
if: inputs.install-cross-tools == 'true'
@@ -105,24 +86,12 @@ runs:
uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
- name: Install cargo-nextest
if: inputs.install-test-tools == 'true'
uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
# false is rust-cache's own default. With true, cleanup.ts returns
# *before* pruning ~/.cargo/registry/src, and config.ts archives the
# whole registry — so every cache carried the unpacked source tree of
# every dependency, not just "a few extra crates".
#
# No coverage is lost: getPackages runs `cargo metadata --all-features`,
# a strict superset of any single lane's feature closure, and -sys crates
# are explicitly exempted from pruning (their src timestamps would
# otherwise trigger rebuilds). Anything pruned is re-unpacked from the
# .crate files still in registry/cache, whose mtimes crates.io
# normalises, so cargo fingerprints stay valid.
cache-all-crates: false
cache-all-crates: true
cache-on-failure: true
shared-key: ${{ inputs.cache-shared-key }}
save-if: ${{ inputs.cache-save-if }}
@@ -37,7 +37,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -46,11 +45,8 @@ jobs:
name: Architecture Migration Rules
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
run: |
+4 -75
View File
@@ -24,7 +24,6 @@ on:
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/security/check_performance_ab_workflow.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
pull_request:
@@ -37,16 +36,10 @@ on:
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/security/check_performance_ab_workflow.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
schedule:
# Daily, not weekly. This schedule exists to catch RustSec advisories
# published against an unchanged dependency tree; at weekly cadence a new
# advisory could sit unnoticed for seven days. The check list is unchanged —
# splitting it into a light daily advisories-only run and a weekly full run
# would create runs where sources/bans/licenses go unverified.
- cron: '0 3 * * *' # Daily 03:00 UTC (staggered after the midnight ci/build crons)
- cron: '0 3 * * 0' # Weekly on Sunday 03:00 UTC (staggered after the midnight ci/build crons)
workflow_dispatch:
permissions:
@@ -66,7 +59,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -82,32 +74,11 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# cargo-deny compiles nothing, so the full setup composite (apt packages,
# protoc, flatc, nextest, rustfmt/clippy) was pure overhead here. It does
# still need a real cargo: `cargo deny check` runs `cargo metadata`, and
# Cargo.toml pins datafusion and s3s as git dependencies, which must be
# materialised into ~/.cargo/git — a cold clone is hundreds of MB, so the
# cache stays.
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
# Was relying on the composite's default, which used to be "true": every
# PR touching Cargo.toml/Cargo.lock saved a second, PR-scoped copy of this
# cache and pushed the main-scoped lanes out of the 10GB quota. The
# default is now "false", but state it explicitly — see
# scripts/security/check_cache_save_if.sh.
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
# Same reasoning as the setup composite: true archives every
# dependency's unpacked source tree.
cache-all-crates: false
cache-on-failure: true
shared-key: rustfs-cargo-deny
save-if: ${{ github.ref == 'refs/heads/main' }}
cache-shared-key: rustfs-cargo-deny
- name: Install cargo-deny
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
@@ -125,31 +96,16 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
- name: Check setup cache-save-if is explicit
run: ./scripts/security/check_cache_save_if.sh
- name: Check every job declares a timeout
run: ./scripts/security/check_job_timeouts.sh
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check performance A/B workflow trust boundary
run: ./scripts/security/check_performance_ab_workflow.sh
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
timeout-minutes: 30
if: github.event_name == 'pull_request' && github.event.action != 'closed'
permissions:
contents: read
@@ -157,8 +113,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Dependency Review
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5
@@ -171,28 +125,3 @@ jobs:
# conscious re-review of the license/provenance claim (backlog#1181).
allow-dependencies-licenses: pkg:cargo/rustfs-uring@0.1.0
comment-summary-in-pr: always
alert-on-failure:
name: Alert on scheduled failure
# dependency-review is deliberately excluded: it only runs on pull_request,
# so it can never contribute a failure to a scheduled run.
needs: [cargo-deny, workflow-pin-report]
# A scheduled cargo-deny failure usually means the dependency tree just
# matched a newly published advisory — the single most important signal this
# workflow produces, and until now it was only visible to whoever happened to
# open the Actions tab. Same ci-8 mechanism coverage.yml and
# e2e-replication-nightly.yml already use.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
+5 -40
View File
@@ -50,18 +50,12 @@ on:
- "**/*.svg"
- ".gitignore"
- ".dockerignore"
- "flake.lock"
schedule:
- cron: "0 1 * * 0" # Weekly on Sunday 01:00 UTC (staggered after the ci.yml midnight cron)
workflow_dispatch:
inputs:
build_docker:
# Advisory only. docker.yml triggers on workflow_run and its job-level
# condition requires the triggering event to be a tag push, so a manual
# dispatch of this workflow never produces images regardless of this
# value. Kept because the summary step reports it; wiring it up would
# mean teaching docker.yml's version parser a second event shape.
description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
description: "Build and push Docker images after binary build"
required: false
default: true
type: boolean
@@ -89,7 +83,6 @@ jobs:
build-check:
name: Build Strategy Check
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
build_type: ${{ steps.check.outputs.build_type }}
@@ -99,8 +92,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Determine build strategy
id: check
@@ -173,7 +164,6 @@ jobs:
name: Prepare Platform Matrix
needs: build-check
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
matrix: ${{ steps.select.outputs.matrix }}
selected: ${{ steps.select.outputs.selected }}
@@ -181,14 +171,10 @@ jobs:
- name: Select target platforms
id: select
shell: bash
env:
# via env, not interpolation: a dispatch input is free-form text and
# would otherwise be pasted into the script for bash to evaluate.
RAW_PLATFORMS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}
run: |
set -euo pipefail
selected="$RAW_PLATFORMS"
selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
@@ -259,7 +245,6 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Setup Rust environment
@@ -268,17 +253,9 @@ jobs:
rust-version: stable
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}
# main only. A cache saved on refs/tags/X is scoped to that tag: no
# other tag, no main run and no PR can restore it, so every release
# cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
# six legs each) that nobody could read, evicting the hot lanes from
# the repo-wide 10GB quota. Tag builds still restore the main-scoped
# cache, since default-branch caches are readable from every ref.
# The one real cost: re-running a failed leg of the same tag no longer
# finds that tag's own warm cache and falls back to main's.
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
install-cross-tools: ${{ matrix.cross }}
install-test-tools: 'false'
- name: Download static console assets
shell: bash
@@ -725,14 +702,9 @@ jobs:
needs: [ build-check, build-rustfs ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Build completion summary
shell: bash
env:
# dispatch input via env: free-form text must not be pasted into the
# script for bash to evaluate.
INPUT_BUILD_DOCKER: ${{ github.event.inputs.build_docker }}
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
@@ -774,7 +746,7 @@ jobs:
echo "🐳 Docker Images:"
if [[ "$BUILD_TYPE" == "preview" ]]; then
echo "⏭️ Preview tags do not publish Docker images"
elif [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
elif [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
echo "⏭️ Docker image build was skipped (binary only build)"
elif [[ "$BUILD_STATUS" == "success" ]]; then
echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
@@ -788,7 +760,6 @@ jobs:
needs: [ build-check, build-rustfs ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
outputs:
@@ -798,7 +769,6 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Create GitHub Release
@@ -849,15 +819,12 @@ jobs:
needs: [ build-check, build-rustfs, create-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -943,7 +910,6 @@ jobs:
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Update latest.json
env:
@@ -1003,7 +969,6 @@ jobs:
needs: [ build-check, create-release, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
steps:
-265
View File
@@ -1,265 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Sole writer of the Rust dependency caches that ci.yml restores.
#
# Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
# concurrency group cancels in-progress runs on main pushes, and merges land far
# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
# 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
# Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
# so the writer lanes were saving nothing and every PR paid a cold restore —
# 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
#
# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
# superseded runs (which is correct — nobody needs test results for a commit
# that is already three merges behind) while the caches still get written.
#
# The group below deliberately does NOT cancel in progress; see the comment on
# it for how that bounds concurrency and why it is scoped by event.
#
# Each job below owns exactly one shared-key and is the only place that sets
# cache-save-if to anything but 'false' for it; every lane in ci.yml reads.
# scripts/security/check_cache_save_if.sh keeps the declarations explicit.
#
# The builds are supersets of what the reading lanes compile, because a reader
# restores only what the writer saved. Feature resolution matters here: a lane
# built with e2e-test-hooks resolves dependency features differently, which
# changes -Cmetadata, so the plain build does not cover it. See
# rustfs/backlog#1600.
name: Cache Warm
on:
push:
branches: [ main ]
# Mirrors ci.yml's push paths-ignore: if a commit cannot change what ci.yml
# compiles, it cannot change what ci.yml needs restored either.
paths-ignore:
- "**.md"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "scripts/probe.sh"
- "LICENSE*"
- ".gitignore"
- ".dockerignore"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
workflow_dispatch:
inputs:
emit_timings:
description: >-
Also emit cargo --timings for the ci-dev build and upload it. Used to
decide whether sccache is worth adopting (rustfs/backlog#1601 gate).
required: false
default: false
type: boolean
permissions:
contents: read
# Scoped by event. A push run and a dispatch run do not compete: GitHub keeps
# one running plus one pending per group, so with a single shared group a
# manually dispatched run was displaced as pending by the next merge and
# cancelled — observed three times in a row, which made the --timings gate in
# rustfs/backlog#1601 effectively impossible to trigger while main was busy.
#
# Still no cancel-in-progress: a burst of merges collapses into "current run
# finishes, newest queued run follows" rather than a pile-up, which is what
# bounds this workflow to one self-hosted runner per event type.
#
# The two paths can now overlap and race to save the same key. That is benign:
# the loser finds the key already present and skips, and both builds produce the
# same artifacts from the same commit.
concurrency:
group: cache-warm-${{ github.event_name }}
cancel-in-progress: false
env:
CARGO_TERM_COLOR: always
jobs:
# Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'true'
install-build-packaging-tools: 'false'
# rustfs/backlog#1601 gate. sccache can only cache compilation units whose
# --emit includes link, so it covers workspace rlibs and nothing else:
# clippy is metadata-only, and the ~100 test binaries, the rustfs bin and
# every build script invoke the system linker. Before spending a bucket,
# credentials and a supply-chain boundary on it, measure how much of the
# build is actually rlib codegen.
#
# Read from the report: workspace lib codegen as a share of the build, and
# s3select-query's own rlib as a share. The plan adopts sccache only above
# 50% and 25% respectively; if linking dominates instead, the answer is
# mold/lld plus split-debuginfo, which is exactly the part sccache cannot
# touch. Off by default — this doubles the ci-dev build.
- name: Build ci-dev superset (with --timings)
if: inputs.emit_timings
env:
CARGO_BUILD_JOBS: "2"
run: cargo build --workspace --all-targets --timings
- name: Upload cargo timings report
if: inputs.emit_timings
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: cargo-timings-ci-dev
path: target/cargo-timings/
retention-days: 30
if-no-files-found: error
# --all-targets covers the test binaries nextest builds, including
# e2e_test, which test-and-lint's own run excludes. The second build adds
# the e2e-test-hooks feature resolution that build-rustfs-debug-binary uses
# and that no lint lane enables.
- name: Build ci-dev superset
env:
# Same limit ci.yml puts on its nextest step: this builds the same
# ~100 workspace test binaries, and three concurrent links saturate the
# self-hosted runner's overlay I/O and can wedge Cargo (#5394).
CARGO_BUILD_JOBS: "2"
run: |
cargo build --workspace --all-targets
cargo build -p rustfs --bins --features e2e-test-hooks
# Runs before rust-cache's post step, so these are the sizes it is about
# to archive. Reported so the cache-all-crates decision stays evidence-led:
# registry/src is what that flag prunes, registry/cache is what the pruned
# sources are re-unpacked from. See rustfs/backlog#1600.
- name: Report cache input sizes
if: always()
run: |
# tee, not a plain redirect: sent only to $GITHUB_STEP_SUMMARY these
# numbers are readable in the UI but absent from the job log, and the
# REST API exposes the log, not the summary — which made the figures
# unreachable for exactly the scripted comparison they exist for.
sizes="$(du -sh ~/.cargo/registry/src ~/.cargo/registry/cache \
~/.cargo/registry/index ~/.cargo/git target 2>/dev/null || true)"
echo "cache-input-sizes-begin"
printf '%s\n' "$sizes"
echo "cache-input-sizes-end"
{
echo "### Cache input sizes (ci-dev)"
echo '```'
printf '%s\n' "$sizes"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
# Readers: test-and-lint-rio-v2, build-rustfs-debug-binary-rio-v2.
warm-ci-feat-rio:
name: Warm ci-feat-rio
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-rio
cache-save-if: 'true'
install-build-packaging-tools: 'false'
- name: Build ci-feat-rio superset
run: |
cargo build -p rustfs -p rustfs-ecstore --all-targets --features rio-v2
cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
# Readers: the swift and sftp legs of test-and-lint-protocols. Built in
# sequence rather than as `--features swift,sftp`, which is a combination no
# lane actually compiles; running both leaves the union in target/.
warm-ci-feat-proto:
name: Warm ci-feat-proto
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-proto
cache-save-if: 'true'
install-build-packaging-tools: 'false'
- name: Build ci-feat-proto superset
run: |
cargo build -p rustfs -p rustfs-protocols --all-targets --features swift
cargo build -p rustfs -p rustfs-protocols --all-targets --features sftp
# Reader: uring-integration. Runs on ubuntu-latest to match it: rust-cache's
# key covers runner.os and arch but not the runner label or image, so a cache
# written on sm-standard-4 would be restored by the hosted runner as if it
# belonged to it.
warm-ci-uring:
name: Warm ci-uring
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-uring
cache-save-if: 'true'
install-build-packaging-tools: 'false'
- name: Install build dependencies
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
- name: Build ci-uring superset
run: cargo build -p rustfs-ecstore --all-targets
+2 -19
View File
@@ -23,8 +23,8 @@
# https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks
#
# "Quick Checks" is mirrored here ahead of the ruleset change that will make it
# required too (rustfs/backlog#1599). Until that change lands this job is
# inert; mirroring it first is what lets the ruleset change happen without
# required too (rustfs/backlog#1599). It is not required yet, so today this job
# is inert; adding it first is what lets that ruleset change land without
# stranding docs-only PRs on a check nobody reports.
#
# Keep the paths list below in sync with the pull_request paths-ignore list
@@ -53,7 +53,6 @@ on:
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- "flake.lock"
permissions:
contents: read
@@ -79,8 +78,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
run: sudo apt-get update && sudo apt-get install -y ripgrep
@@ -102,9 +99,6 @@ jobs:
- name: Check architecture migration rules
run: ./scripts/check_architecture_migration_rules.sh
- name: Check logging guardrails
run: ./scripts/check_logging_guardrails.sh
- name: Check tokio io-uring feature guard
run: ./scripts/check_no_tokio_io_uring.sh
@@ -114,18 +108,9 @@ jobs:
- name: Check body-cache whitelist guard
run: ./scripts/check_body_cache_whitelist.sh
- name: Check s3s footprint ratchet
run: ./scripts/check_s3s_footprint.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
- name: Check CI paths stay in sync
run: ./scripts/check_ci_paths_sync.sh
- name: Check io_uring lane --lib precondition
run: ./scripts/check_uring_lane_lib_only.sh
test-and-lint:
name: Test and Lint
runs-on: ubuntu-latest
@@ -133,8 +118,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Docs-only PRs skip the full code CI, but they are exactly where a
# planning-type document could be slipped in (git add -f bypasses
+72 -265
View File
@@ -33,7 +33,6 @@ on:
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- "flake.lock"
pull_request:
types: [ opened, synchronize, reopened, closed ]
branches: [ main ]
@@ -55,7 +54,6 @@ on:
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- "flake.lock"
merge_group:
types: [ checks_requested ]
schedule:
@@ -83,7 +81,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -92,11 +89,8 @@ jobs:
name: Typos
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Typos check with custom config file
uses: crate-ci/typos@37bb98842b0d8c4ffebdb75301a13db0267cef89 # master
@@ -114,8 +108,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
run: sudo apt-get update && sudo apt-get install -y ripgrep
@@ -137,9 +129,6 @@ jobs:
- name: Check architecture migration rules
run: ./scripts/check_architecture_migration_rules.sh
- name: Check logging guardrails
run: ./scripts/check_logging_guardrails.sh
- name: Check tokio io-uring feature guard
run: ./scripts/check_no_tokio_io_uring.sh
@@ -149,54 +138,27 @@ jobs:
- name: Check body-cache whitelist guard
run: ./scripts/check_body_cache_whitelist.sh
- name: Check s3s footprint ratchet
run: ./scripts/check_s3s_footprint.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
- name: Check CI paths stay in sync
run: ./scripts/check_ci_paths_sync.sh
- name: Check io_uring lane --lib precondition
run: ./scripts/check_uring_lane_lib_only.sh
test-and-lint:
name: Test and Lint
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 90
# Both lines are required. Job-level `permissions` replaces the workflow
# block rather than merging with it, so declaring only `actions: write`
# would drop `contents: read` and break this job's checkout and the
# repo-token the setup action hands to setup-protoc.
permissions:
contents: read
actions: write
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
# This job's token can cancel runs and delete Actions caches. Checkout
# otherwise writes it into .git/config, where a PR's own build.rs or
# proc-macro could read it back out.
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
# Every lane in this workflow reads its cache and none writes it.
# cache-warm.yml is the sole writer for all four keys: this workflow
# cancels superseded runs on main, and a cancelled run never reaches
# rust-cache's post step, so writing from here saved nothing (12 of 15
# consecutive main-push runs were cancelled). See rustfs/backlog#1600.
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-test
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Prepare test evidence
run: |
@@ -211,54 +173,43 @@ jobs:
# Clippy runs before the test pass: lint failures are the most common
# CI-only breakage and should surface in minutes, not after 20+ minutes
# of tests.
# Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
# experiment, since --all-targets is check-only for workspace members and
# never links the ~100 test binaries the limit exists to throttle.
- name: Run clippy lints
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo clippy --all-targets -- -D warnings
run: cargo clippy --all-targets -- -D warnings
- name: Run nextest tests
env:
# #5394 mitigation, now under a measured experiment (backlog#1601).
#
# 2 was chosen when three concurrent workspace test links were believed
# to saturate the runner's overlay I/O and wedge Cargo until the 75m
# timeout. cgroup v2 readings from the sampler show the pod actually
# has 14 CPUs and 28GB (peak use 2.1GB), so 2 throttles compilation to
# a seventh of what is available and memory was never the constraint —
# the label name "sm-standard-4" had led everyone, including the
# original mitigation, to assume 4 cores.
#
# Raised to 3 on main pushes and manual dispatches; PRs keep 2 so the
# merge path is untouched while the experiment runs.
#
# Dispatch is included because push alone cannot supply the samples:
# this workflow cancels superseded runs on main, and only 4 of the last
# 20 push-triggered Test and Lint jobs reached a terminal state — at
# that rate ten samples would take roughly fifty merges. The
# concurrency group is scoped by event_name, so a dispatched run has
# its own group and is not cancelled by merge traffic, which makes the
# sample collectable on demand rather than by waiting.
#
# Baseline over 17 samples at 2:
# median nextest/clippy step ratio 1.95, spread 1.85-2.06. The gate-2
# criterion is that ratio dropping at least 10% (below ~1.76) with no
# 75m timeout and no run showing three consecutive samples of
# rustc/collect2/rust-lld in D state. If it does not, the conclusion is
# "this limit is not the bottleneck" — fix it back at 2 and record the
# experiment, which is a result, not a failure.
#
# Must stay step-level: rust-cache hashes CARGO/CC/CFLAGS/CXX/CMAKE/RUST
# prefixed variables from process.env into the cache key, so promoting
# this to job level would rotate every key on this lane.
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
# Three concurrent workspace test links saturate the self-hosted
# runner's overlay I/O and can wedge Cargo until the 75m timeout.
CARGO_BUILD_JOBS: "2"
run: |
mkdir -p artifacts/test-and-lint
./scripts/ci/resource_sampler.sh start nextest
trap './scripts/ci/resource_sampler.sh stop' EXIT
# Evidence sampler for issue #5394: the post-mortem pgrep below runs
# only after `timeout` has already TERM'd the whole cargo process
# group, so it cannot name a wedged process. Sample system and
# process state every 60s instead; the last samples before the
# timeout show what was stuck (rustc, linker, build script, memory
# pressure, ...). The log rides along in the existing artifact.
(
while true; do
{
echo "=== $(date --utc --iso-8601=seconds)"
echo "--- load"; cat /proc/loadavg
echo "--- psi"; grep -H . /proc/pressure/* 2>/dev/null || true
echo "--- mem"; free -m
echo "--- disk"; df -h / /home/runner 2>/dev/null || df -h /
echo "--- top-rss"
ps -eo pid,ppid,stat,etime,rss,pcpu,args --sort=-rss | head -15
echo "--- build/test processes"
ps -eo pid,ppid,stat,etime,rss,pcpu,args | grep -E '[c]argo|[r]ustc|[n]extest|[c]ollect2|rust-ll[d]|[b]uild-script|deps[/]' || true
echo "--- d-state (uninterruptible IO)"
ps -eo pid,stat,etime,args | awk 'NR > 1 && $2 ~ /D/' || true
echo
} >> artifacts/test-and-lint/sampler.log 2>&1 || true
sleep 60
done
) &
sampler_pid=$!
trap 'kill "${sampler_pid}" 2>/dev/null || true' EXIT
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 75m \
cargo nextest run --profile ci --all --exclude e2e_test \
@@ -330,50 +281,6 @@ jobs:
- name: Run rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh
# Early stop. Once this job has failed the PR cannot merge, so the sibling
# lanes are burning runners on a result nobody can act on: on run
# 30674613104 three lanes had already failed while Test and Lint and the
# rio-v2 variant kept going past 70 minutes.
#
# Only this job may cancel. The lanes that are NOT required checks
# (protocols, ILM, e2e, s3-tests) must never hold that power: a flake in
# one of them would turn the required "Test and Lint" into `cancelled`,
# which blocks the merge. Today a maintainer can merge with sftp red, and
# that has to stay true.
#
# These steps run last so the `if: always()` artifact upload above still
# captures logs and diagnostics before the run goes away.
- name: Annotate early-stop reason
if: failure() && github.event_name == 'pull_request'
run: |
{
echo "## CI early-stop"
echo "Job \`${GITHUB_JOB}\` (Test and Lint) failed; cancelling run ${GITHUB_RUN_ID} to free runners."
echo "Sibling jobs showing **cancelled** were stopped by this job, not by their own failure."
} >> "$GITHUB_STEP_SUMMARY"
# curl rather than `gh`: every existing `gh` call in this repo runs on
# ubuntu-latest, and the sm-standard-* images are custom and trimmed (they
# ship no C toolchain, see the e2e job below), so `gh` is not known to
# exist here.
#
# Fork PRs are excluded explicitly instead of relying on the error path:
# their GITHUB_TOKEN is forced read-only and job-level permissions cannot
# raise it, so the call would always 403. Skipping keeps their logs clean.
- name: Cancel run on failure (same-repo PR only)
if: >-
failure() && github.event_name == 'pull_request'
&& github.event.pull_request.head.repo.full_name == github.repository
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
curl -fsS -X POST \
-H "Authorization: Bearer ${GH_TOKEN}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/cancel" || true
# Dedicated serial lane for the ILM / lifecycle integration tests. These tests
# drive the object layer through process-global singletons (the GLOBAL_ENV
# ECStore, the global tier-config manager, background-expiry workers) and bind
@@ -387,7 +294,6 @@ jobs:
test-ilm-integration-serial:
name: ILM Integration (serial)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 45
env:
@@ -395,16 +301,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-ilm-serial
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
@@ -427,7 +331,6 @@ jobs:
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 60
env:
@@ -435,16 +338,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-rio
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-test-rio-v2
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run rio-v2 clippy lints
run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
@@ -457,17 +358,10 @@ jobs:
test-and-lint-protocols:
name: "Test and Lint (${{ matrix.features.name }})"
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 60
strategy:
# On a PR, one failing protocol leg is enough to know the PR is not ready,
# so stop the sibling leg instead of paying another ~40 minutes for it.
# Everywhere else (main pushes, the merge queue, the weekly schedule) keep
# the full signal: there we want to know whether swift AND sftp are broken,
# not just whichever failed first. This is the only part of the early-stop
# work that also covers fork PRs, since it needs no token.
fail-fast: ${{ github.event_name == 'pull_request' }}
fail-fast: false
matrix:
features:
- name: swift
@@ -479,16 +373,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-proto
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-test-${{ matrix.features.name }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run clippy with ${{ matrix.features.name }}
run: |
@@ -501,7 +393,6 @@ jobs:
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 30
env:
@@ -509,16 +400,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-rustfs-debug-binary
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build debug binary
run: cargo build -p rustfs --bins --features e2e-test-hooks
@@ -534,7 +423,6 @@ jobs:
build-rustfs-debug-binary-rio-v2:
name: Build RustFS Debug Binary (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 30
env:
@@ -542,16 +430,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-rio
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-rustfs-debug-binary-rio-v2
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build debug binary with rio-v2
run: cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
@@ -573,7 +459,6 @@ jobs:
# suite (measured 4m17s / 7m19s / 7m31s on runs 30678272341 / 30678117601 /
# 30662728539) and kept the cancellation run in progress for minutes.
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
# GitHub-hosted ubuntu-latest runs a recent kernel with io_uring and, unlike
# a container, applies no seccomp filter that would block io_uring_setup — so
# the probe succeeds and the tests exercise the real UringBackend/FdCache/
@@ -584,24 +469,17 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
# Keeps its own key rather than joining ci-dev. rust-cache's key is
# built from runner.os/arch plus rustc and lockfile fingerprints — it
# does NOT include the runner label or image. ubuntu-latest and
# sm-standard-4 are therefore indistinguishable to it, so sharing a key
# would let two different system images overwrite each other's
# artifacts, and would make a 2-core hosted runner unpack ci-dev's ~3GB
# instead of this lane's ~1.3GB. cache-warm.yml warms this key on
# ubuntu-latest for the same reason.
cache-shared-key: ci-uring
cache-save-if: 'false'
install-build-packaging-tools: 'false'
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install build dependencies
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
# ext4 supports O_DIRECT; the runner's default TMPDIR may sit on tmpfs or
# overlayfs, where open(O_DIRECT) returns EINVAL/EOPNOTSUPP and the native
@@ -628,17 +506,7 @@ jobs:
RUSTFS_IO_URING_READ_ENABLE: "true"
RUSTFS_URING_TESTS_MUST_RUN: "1"
TMPDIR: /mnt/rustfs-odirect
# --lib narrows what gets compiled, not what gets run: every selected
# test lives in the lib target. The 7 integration binaries under
# crates/ecstore/tests/ each reported "running 0 tests" here, so they
# were compiled and linked for nothing.
#
# The `uring_` filter must stay exactly as it is. libtest matches on
# substring, so it also selects names containing `during_` — 6 of the 18
# selected tests are such incidental matches. Narrowing the filter to
# `io_uring` would silently drop them, which is a coverage change.
# scripts/check_uring_lane_lib_only.sh guards the --lib precondition.
run: cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture
run: cargo test -p rustfs-ecstore uring_ -- --test-threads=1 --nocapture
e2e-tests:
name: End-to-End Tests
@@ -648,8 +516,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Full setup with dependency caching: the smoke-suite step below
# compiles the e2e_test crate, which pulls in most of the workspace.
@@ -658,9 +524,9 @@ jobs:
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-e2e
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
@@ -673,17 +539,15 @@ jobs:
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
# Build the e2e test graph once. The archive is reused by the security
# count-floor check and the smoke run below, avoiding a second compile of
# the same e2e_test target on cold runners (backlog#1645).
- name: Archive e2e smoke test binaries
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
run: |
cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
# Guard the security negative-auth smoke subset (backlog#1151 sec-5)
# against a rename or deletion silently dropping it out of the e2e-smoke
# filter. The script lists what the profile selects and fails if the count
# of security auth-rejection tests falls below the committed floor in
# .config/security-smoke-floor.txt (infra-12 count-floor mechanism). Run
# before the smoke suite so a thinned gate fails fast; the `nextest list`
# here compiles the e2e_test binaries the run below reuses.
- name: Check security smoke subset count floor
run: ./scripts/check_security_smoke_count.sh check
# PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
# profile.e2e-smoke default-filter in .config/nextest.toml is the single
@@ -691,30 +555,7 @@ jobs:
# adding new e2e jobs here. Each test spawns its own rustfs server on a
# random port and reuses the downloaded debug binary above.
- name: Run e2e smoke suite
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
run: |
cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
--status-level all --final-status-level all --failure-output final
- name: Upload e2e smoke diagnostics
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: e2e-smoke-diagnostics-${{ github.run_number }}
path: |
${{ runner.temp }}/rustfs-e2e-smoke-logs/
${{ runner.temp }}/rustfs-e2e-smoke-list.json
if-no-files-found: warn
- name: Upload e2e smoke JUnit report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: e2e-smoke-junit-${{ github.run_number }}
path: target/nextest/e2e-smoke/junit.xml
if-no-files-found: warn
run: cargo nextest run --profile e2e-smoke -p e2e_test
- name: Install s3s-e2e test tool
uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
@@ -759,42 +600,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
- name: Verify awscurl
run: test -x "$AWSCURL_PATH"
- name: Install Vault
run: |
VAULT_VERSION="1.17.6"
VAULT_ARCHIVE="vault_${VAULT_VERSION}_linux_amd64.zip"
curl -fsSLo "$RUNNER_TEMP/$VAULT_ARCHIVE" "https://releases.hashicorp.com/vault/${VAULT_VERSION}/${VAULT_ARCHIVE}"
echo "0cddc1fbbb88583b5ba5b845f9f8fae47c6fb39a6d48cd543c6ba6fd3ac1a669 $RUNNER_TEMP/$VAULT_ARCHIVE" | sha256sum --check --status
unzip -q "$RUNNER_TEMP/$VAULT_ARCHIVE" -d "$RUNNER_TEMP/vault-bin"
echo "RUSTFS_TEST_VAULT_BIN=$RUNNER_TEMP/vault-bin/vault" >> "$GITHUB_ENV"
- name: Verify Vault
run: |
"$RUSTFS_TEST_VAULT_BIN" version
cache-shared-key: ci-e2e
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
@@ -830,8 +643,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Clean up previous test run
run: |
@@ -886,8 +697,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -960,8 +769,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+4 -23
View File
@@ -22,18 +22,11 @@ on:
issue_comment:
types: [created, edited]
# Least privilege at the top, widened per job below. This workflow runs on
# pull_request_target and issue_comment, so it holds full secrets on every fork
# PR and on any comment anyone writes — the one place in this repository where a
# compromised action would be handed a repo-write token. It does not check out
# or execute PR code, so there is no pwn-request path today, but the blast
# radius should not depend on that staying true.
#
# contents: write in particular was never used: the signature records are
# written to rustfs/cla through the scoped app token created below, and nothing
# here writes to this repository's contents.
permissions:
contents: read
contents: write
pull-requests: write
issues: write
checks: write
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.event.issue.number || github.ref }}
@@ -43,26 +36,14 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request_target' && github.event.action == 'closed'
# Echoes one line; the run exists only so the concurrency group cancels the
# in-flight run of a closed PR.
permissions: {}
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
cla:
if: ${{ (github.event_name != 'issue_comment' || github.event.issue.pull_request) && (github.event_name != 'pull_request_target' || github.event.action != 'closed') }}
# checks: write reports the merge-queue check run; pull-requests and issues
# let cla-bot comment and label. contents stays read — see the note above.
permissions:
contents: read
checks: write
issues: write
pull-requests: write
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Report CLA result for merge queue
if: github.event_name == 'merge_group'
+1 -5
View File
@@ -62,16 +62,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-coverage
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: 'false'
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
@@ -118,8 +116,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
+20 -34
View File
@@ -85,7 +85,6 @@ jobs:
github.event.workflow_run.head_branch != 'main' &&
!contains(github.event.workflow_run.head_branch, '-preview'))
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
@@ -98,18 +97,11 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
@@ -210,9 +202,9 @@ jobs:
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
input_version="${{ github.event.inputs.version }}"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_push="${{ github.event.inputs.push_images }}"
should_build=true
# Get short SHA
@@ -220,7 +212,7 @@ jobs:
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🔧 Force rebuild: ${{ github.event.inputs.force_rebuild }}"
echo " 🚀 Push images: $should_push"
case "$input_version" in
@@ -306,8 +298,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
@@ -343,28 +333,32 @@ jobs:
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
# Convert version format for Dockerfile compatibility
case "$VERSION" in
"latest")
# For stable latest, use RELEASE=latest + release CHANNEL
DOCKER_RELEASE="latest"
DOCKER_CHANNEL="release"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
DOCKER_CHANNEL="release"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
DOCKER_CHANNEL="release"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "docker_channel=$DOCKER_CHANNEL" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
echo " - Docker CHANNEL: $DOCKER_CHANNEL"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
@@ -418,24 +412,18 @@ jobs:
push: ${{ needs.build-check.outputs.should_push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
cache-from: |
type=gha,scope=docker-${{ matrix.variant }}
cache-to: |
type=gha,mode=max,scope=docker-${{ matrix.variant }}
build-args: |
BUILDTIME=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
VERSION=${{ needs.build-check.outputs.version }}
BUILD_TYPE=${{ needs.build-check.outputs.build_type }}
REVISION=${{ github.sha }}
RELEASE=${{ steps.meta.outputs.docker_release }}
CHANNEL=${{ steps.meta.outputs.docker_channel }}
BUILDKIT_INLINE_CACHE=1
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
@@ -451,7 +439,6 @@ jobs:
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
@@ -506,7 +493,6 @@ jobs:
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
@@ -64,16 +64,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-e2e-repl
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: 'false'
# awscurl lets the STS dual-node test actually exercise its path. Without
# it the test skips gracefully with a visible log line
@@ -126,8 +124,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
-11
View File
@@ -45,13 +45,6 @@
# The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker
# via DEPLOY_MODE=binary and defers all pip setup to run.sh's self-bootstrap.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: e2e-s3tests
on:
@@ -142,8 +135,6 @@ jobs:
TEST_MODE: ${{ matrix.test-mode }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Provision Python explicitly rather than trusting the runner image to
# ship a working pip (ci-1: a bare python3 without pip is what broke the
@@ -363,8 +354,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
+1 -16
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Fuzz
on:
@@ -66,7 +59,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -87,14 +79,13 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: nightly
cache-shared-key: fuzz-${{ hashFiles('fuzz/Cargo.lock') }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
- name: Install cargo-fuzz
@@ -154,8 +145,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -211,8 +200,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -260,8 +247,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
+7 -32
View File
@@ -32,7 +32,6 @@ permissions:
jobs:
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
@@ -50,26 +49,16 @@ jobs:
steps:
- name: Checkout helm chart repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
RAW="${{ github.event.inputs.version }}"
else
RAW="$RAW_BRANCH"
RAW="${{ github.event.workflow_run.head_branch }}"
fi
case "$RAW" in
@@ -84,13 +73,10 @@ jobs:
./scripts/helm_chart_version.sh "$RAW_TAG"
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
@@ -115,7 +101,6 @@ jobs:
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
@@ -123,8 +108,6 @@ jobs:
- name: Checkout helm package repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
@@ -140,19 +123,11 @@ jobs:
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git config --global user.name "${{ secrets.USERNAME }}"
git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
git push origin main
-8
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: "issue-translator"
on:
issue_comment:
@@ -33,7 +26,6 @@ permissions:
jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: usthe/issues-translate-action@b41f55ddc81d7d54bd542a4f289fe28ec081898e # v2.7
with:
+5 -58
View File
@@ -18,25 +18,11 @@
# This is NOT a PR gate. The fixtures are real MinIO backend trees generated on
# the fly (they are gitignored, never committed), so the job regenerates them
# each run with Docker and then runs the `#[ignore]` reader tests in
# rustfs/src/storage/minio_generated_read_test.rs.
#
# Scope: end-to-end MinIO-to-RustFS SSE interop is NOT implemented yet. Both
# envelope parsers reject MinIO's own wrapped-DEK shape — see
# `is_data_key_envelope` in crates/kms/src/encryption/dek.rs and the
# `deny_unknown_fields` `LocalSseDekEnvelope` in rustfs/src/storage/sse.rs — and
# closing that gap is tracked in rustfs/backlog#1638. Treat this job as the
# harness for #1638, not as standing evidence that a MinIO migration reads back.
# crates/ecstore/tests/minio_generated_read_test.rs.
#
# Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
# unlike the self-hosted fleet, whose pods drift in Docker/pip availability
# (see the infra note in e2e-s3tests.yml). Nightly + manual only.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: minio-interop
on:
@@ -62,62 +48,23 @@ jobs:
env:
# Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
RUSTFS_MINIO_STATIC_KMS_KEY_B64: IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g=
# Single definition of "the interop tests", shared by the guard step and
# the run step so the two cannot drift apart.
#
# These used to live in crates/ecstore/tests/minio_generated_read_test.rs
# and were selected with `-p rustfs-ecstore -E
# 'binary(minio_generated_read_test)'`. #5435 moved them into the `rustfs`
# crate as a `#[cfg(test)] mod`, which deleted that test binary; the
# selector was never updated and has selected zero interop tests ever
# since (cargo-nextest 0.9.140 now rejects it outright: "operator didn't
# match any binary names", exit 94).
INTEROP_PACKAGE: rustfs
INTEROP_FEATURES: rio-v2
INTEROP_FILTER: "test(minio_generated_read_test::)"
INTEROP_REQUIRED_TESTS: '["reads_minio_generated_sse_s3_multipart_fixture", "reads_minio_generated_sse_kms_multipart_fixture", "rejects_minio_generated_sse_s3_fixture_with_wrong_kms_key", "rejects_minio_generated_sse_s3_fixture_with_truncated_ciphertext"]'
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-minio-interop
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Generate real MinIO fixtures via Docker
run: bash crates/rio-v2/tests/minio_fixture_lab/capture_via_docker.sh
# `binary(...)` at least dies loudly when nothing matches, but `test(...)`
# is a perfectly valid filterset that matches zero tests, so the next
# rename or module move would leave this job selecting nothing and
# reporting success without executing a single interop assertion. Count
# the selection and require every core reader test, while allowing new
# reader cases to be added without changing this guard.
#
# Count only `filter-match.status == "matches"`: the top-level
# `test-count` in the JSON is the package total and ignores `-E` entirely.
- name: Assert the interop selector still matches tests
run: |
set -euo pipefail
selection="$(cargo nextest list --run-ignored ignored-only \
-p "$INTEROP_PACKAGE" --features "$INTEROP_FEATURES" \
-E "$INTEROP_FILTER" --message-format json \
| python3 -c 'import json,os,sys; d=json.load(sys.stdin); required=json.loads(os.environ["INTEROP_REQUIRED_TESTS"]); matched=[name for suite in d.get("rust-suites", {}).values() for name,test in suite.get("testcases", {}).items() if test.get("filter-match", {}).get("status") == "matches"]; missing=[test for test in required if not any(name.endswith("minio_generated_read_test::" + test) for name in matched)]; print(len(matched)); print(",".join(missing))')"
count="$(printf '%s\n' "$selection" | sed -n '1p')"
missing="$(printf '%s\n' "$selection" | sed -n '2p')"
echo "interop tests selected: ${count}"
if [ -n "${missing}" ]; then
echo "::error::Selector '${INTEROP_FILTER}' in package '${INTEROP_PACKAGE}' is missing required tests: ${missing}. The MinIO interop reader tests have moved or been renamed; fix the selector instead of running an incomplete matrix. Context: rustfs/backlog#1638."
exit 1
fi
- name: Run MinIO interop reader tests
run: |
cargo nextest run --run-ignored ignored-only --no-tests=fail \
-p "$INTEROP_PACKAGE" --features "$INTEROP_FEATURES" \
-E "$INTEROP_FILTER"
cargo nextest run --run-ignored ignored-only \
-p rustfs-ecstore --features rio-v2 \
-E 'binary(minio_generated_read_test)'
-11
View File
@@ -45,13 +45,6 @@
# docker-capable self-hosted `dind-sm-standard-2` label was the alternative but
# has fewer cores and reintroduces fleet-state risk for no reliability gain.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: mint
on:
@@ -125,8 +118,6 @@ jobs:
timeout-minutes: 120
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Enable buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
@@ -272,8 +263,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
-57
View File
@@ -1,57 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Nightly GNU Build
on:
schedule:
- cron: "0 0 * * *"
timezone: "Asia/Shanghai"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: nightly-gnu-build-main-${{ github.event_name }}
cancel-in-progress: ${{ github.event_name == 'workflow_dispatch' }}
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
jobs:
build:
name: Build x86_64 GNU
runs-on: sm-standard-2
timeout-minutes: 150
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout main branch
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
ref: main
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: build-x86_64-unknown-linux-gnu
cache-save-if: 'false'
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Build RustFS
run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
+2 -9
View File
@@ -19,12 +19,9 @@ on:
schedule:
- cron: '0 5 * * 0' # Weekly on Sunday 05:00 UTC (staggered after the midnight ci/build crons)
# GITHUB_TOKEN only needs to read the repository here: the branch push and the
# pull request are both created by update-flake-lock using the
# FLAKE_UPDATE_TOKEN PAT below, not by this token. Leaving write on it hands a
# repo-write credential to an unattended weekly job that does not use it.
permissions:
contents: read
contents: write
pull-requests: write
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -40,10 +37,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
# persist-credentials-exempt: update-flake-lock pushes the branch and opens
# the PR. It passes FLAKE_UPDATE_TOKEN to create-pull-request itself rather
# than reusing .git/config, but that is unverified — exempt until a
# workflow_dispatch run confirms it (rustfs/backlog#1602).
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@629b284231c2a82554b724e357e47fc6020833c8 # v3
-10
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Nix CI
on:
@@ -53,7 +46,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -71,8 +63,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@4eea0b33e3d1f02ecfe37cf16e7204c424009606 # v3.21.0
-463
View File
@@ -1,463 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2.
#
# Trigger:
# - release published: automatically package when a GitHub release is published
# - workflow_dispatch: manual trigger with optional tag/run_id
#
# Flow:
# 1. Find the Build workflow run for the release tag
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
# 3. Build DEB packages for amd64 and arm64
# 4. Build RPM packages for x86_64 and aarch64
# 5. Upload all packages to Cloudflare R2
name: Package DEB/RPM
permissions:
contents: read
actions: read
on:
release:
types: [ published ]
workflow_dispatch:
inputs:
tag:
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
required: false
type: string
build_run_id:
description: "Build workflow run ID (overrides tag lookup)"
required: false
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.release.tag_name || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
build_type: ${{ steps.resolve.outputs.build_type }}
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
tag: ${{ steps.resolve.outputs.tag }}
steps:
- name: Resolve build run
id: resolve
shell: bash
env:
GH_TOKEN: ${{ github.token }}
INPUT_TAG: ${{ github.event.inputs.tag }}
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
run: |
set -euo pipefail
# Determine tag
if [[ "${{ github.event_name }}" == "release" ]]; then
TAG="${{ github.event.release.tag_name }}"
elif [[ -n "$INPUT_TAG" ]]; then
TAG="$INPUT_TAG"
else
TAG=""
fi
echo "Tag: ${TAG:-<none>}"
# Determine build run ID
BUILD_RUN_ID=""
if [[ -n "$INPUT_RUN_ID" ]]; then
# Explicit run ID takes priority
BUILD_RUN_ID="$INPUT_RUN_ID"
echo "Using explicit build run ID: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
# Find the build run that produced this tag
echo "Looking for build run for tag: $TAG"
BUILD_RUN_ID=$(gh api \
"repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=${TAG}&status=success&per_page=1" \
--jq '.workflow_runs[0].id' 2>/dev/null || echo "")
if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
# Tag might not be a branch; try event=push with head_branch matching
BUILD_RUN_ID=$(gh api \
"repos/${{ github.repository }}/actions/workflows/build.yml/runs?event=push&status=success&per_page=100" \
--jq ".workflow_runs[] | select(.head_branch == \"$TAG\") | .id" 2>/dev/null | head -1 || echo "")
fi
if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
echo "❌ No successful build run found for tag: $TAG"
exit 1
fi
echo "Found build run: $BUILD_RUN_ID"
else
# No tag — latest successful main build
echo "No tag specified, looking for latest main build"
BUILD_RUN_ID=$(gh api \
"repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=main&status=success&per_page=1" \
--jq '.workflow_runs[0].id' 2>/dev/null || echo "")
if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
echo "❌ No successful main build found"
exit 1
fi
echo "Latest main build: $BUILD_RUN_ID"
fi
# Determine version and build type
if [[ -n "$TAG" ]]; then
VERSION="$TAG"
if [[ "$TAG" == *"-preview"* ]]; then
BUILD_TYPE="preview"
elif [[ "$TAG" == *"alpha"* || "$TAG" == *"beta"* || "$TAG" == *"rc"* ]]; then
BUILD_TYPE="prerelease"
else
BUILD_TYPE="release"
fi
else
SHORT_SHA=$(gh api "repos/${{ github.repository }}/actions/runs/${BUILD_RUN_ID}" \
--jq '.head_sha' 2>/dev/null | head -c 7)
VERSION="dev-${SHORT_SHA}"
BUILD_TYPE="development"
fi
{
echo "version=$VERSION"
echo "build_type=$BUILD_TYPE"
echo "build_run_id=$BUILD_RUN_ID"
echo "tag=${TAG}"
} >> "$GITHUB_OUTPUT"
echo "📊 Resolved:"
echo " Version: $VERSION"
echo " Build type: $BUILD_TYPE"
echo " Build run ID: $BUILD_RUN_ID"
# Build DEB and RPM packages for each architecture
package:
name: Package (${{ matrix.arch }})
needs: resolve
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
- arch: x86_64
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
- arch: aarch64
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download binary artifact from build run
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
run-id: ${{ needs.resolve.outputs.build_run_id }}
github-token: ${{ github.token }}
merge-multiple: true
- name: Extract binary
id: binary
shell: bash
run: |
set -euo pipefail
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
if [[ -z "$ZIP_FILE" ]]; then
echo "❌ No binary artifact found"
ls -la ./binary-artifact/ || true
exit 1
fi
echo "Found artifact: $ZIP_FILE"
mkdir -p ./bin
unzip -o "$ZIP_FILE" -d ./bin
if [[ ! -f ./bin/rustfs ]]; then
echo "❌ rustfs binary not found in archive"
exit 1
fi
chmod +x ./bin/rustfs
ls -lh ./bin/rustfs
echo "✅ Binary extracted"
- name: Build DEB package
id: deb
shell: bash
run: |
set -euo pipefail
VERSION="${{ needs.resolve.outputs.version }}"
DEB_ARCH="${{ matrix.deb_arch }}"
# DEB version: replace - with ~ (1.0.0-beta.12 -> 1.0.0~beta.12)
DEB_VERSION="${VERSION/-/~}"
PKG_DIR="rustfs_${DEB_VERSION}_${DEB_ARCH}"
echo "Building DEB: ${PKG_DIR}.deb"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
Depends: libc6 (>= 2.31)
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
POSTINST
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}"
DEB_FILE="${PKG_DIR}.deb"
ls -lh "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
- name: Build RPM package
id: rpm
shell: bash
run: |
set -euo pipefail
VERSION="${{ needs.resolve.outputs.version }}"
RPM_ARCH="${{ matrix.rpm_arch }}"
echo "Building RPM for ${RPM_ARCH}"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential
sudo gem install fpm
fpm -s dir -t rpm \
--name rustfs \
--version "$VERSION" \
--architecture "$RPM_ARCH" \
--depends "glibc >= 2.31" \
--maintainer "RustFS Team <support@rustfs.com>" \
--description "High-performance distributed object storage" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--after-install <(cat <<'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTINST
) \
--before-remove <(cat <<'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
) \
--after-remove <(cat <<'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
RPM_FILE=$(ls -1 rustfs-*.rpm 2>/dev/null | head -1)
if [[ -z "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
ls -lh "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Upload packages to artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: packages-${{ matrix.arch }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "⚠️ R2 credentials missing, skipping upload"
exit 0
fi
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
BUILD_TYPE="${{ needs.resolve.outputs.build_type }}"
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/release"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
echo "📤 Uploading to $R2_PATH"
DEB_FILE="${{ steps.deb.outputs.deb_file }}"
RPM_FILE="${{ steps.rpm.outputs.rpm_file }}"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading: $f"
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Upload complete"
# Also upload as latest for release/prerelease
if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading latest: $(basename "$f")"
aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Latest packages updated"
fi
# Summary
summary:
name: Summary
needs: [ resolve, package ]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Print summary
shell: bash
run: |
echo "## 📦 Package Summary" >> "$GITHUB_STEP_SUMMARY"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "| Item | Value |" >> "$GITHUB_STEP_SUMMARY"
echo "|------|-------|" >> "$GITHUB_STEP_SUMMARY"
echo "| Version | \`${{ needs.resolve.outputs.version }}\` |" >> "$GITHUB_STEP_SUMMARY"
echo "| Build Type | ${{ needs.resolve.outputs.build_type }} |" >> "$GITHUB_STEP_SUMMARY"
echo "| Build Run | #${{ needs.resolve.outputs.build_run_id }} |" >> "$GITHUB_STEP_SUMMARY"
echo "| Package Status | ${{ needs.package.result }} |" >> "$GITHUB_STEP_SUMMARY"
+72 -85
View File
@@ -17,18 +17,11 @@
# Two entry points, honestly scoped:
# * schedule (nightly, on main): post-merge detection — catches a regression
# within 24h of landing, not before merge.
# * workflow_dispatch: an explicitly selected trusted ref.
# The dispatch input can run the gate with --allow-regression so a deliberate
# correctness cost (e.g. the #4221 fsync durability fix) is recorded, not
# blocked (rustfs/backlog#935 correction 1).
# * pull_request labeled `perf-ab`: opt-in pre-merge gate for a specific PR.
# The `perf-deliberate-tradeoff` label runs the gate with --allow-regression so
# a deliberate correctness cost (e.g. the #4221 fsync durability fix) is
# recorded but does not block (rustfs/backlog#935 correction 1).
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Performance A/B
on:
@@ -46,6 +39,8 @@ on:
required: false
default: false
type: boolean
pull_request:
types: [labeled, synchronize, reopened]
push:
# Every main commit pre-builds and caches its release binary (perf-3) so the
# nightly A/B restores a ready baseline instead of paying the double build.
@@ -53,6 +48,14 @@ on:
permissions:
contents: read
pull-requests: write
# Per-PR: a new push cancels the previous (up to 90-minute) A/B run instead of
# stacking them. Nightly schedule and manual dispatch get a unique group and
# always run to completion.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
CARGO_TERM_COLOR: always
@@ -60,8 +63,8 @@ env:
jobs:
# perf-3: on every push to main, build the release binary once and cache it
# keyed by commit SHA (rustfs-baseline-<sha>). The warp-ab measurements
# restore this instead of paying the ~32min-per-side source
# keyed by commit SHA (rustfs-baseline-<sha>). The nightly A/B (and, later, the
# perf-7 PR gate) restore this instead of paying the ~32min-per-side source
# build. That double build is what pushed the expanded 24-cell nightly past its
# ceiling — 2026-07-11..07-14 all cancelled on the 120min timeout. Incremental
# builds off the shared cargo cache keep each push cheap, and building on the
@@ -89,8 +92,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
@@ -98,6 +99,7 @@ jobs:
rust-version: stable
cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build release rustfs
run: cargo build --release --bin rustfs
@@ -116,11 +118,17 @@ jobs:
warp-ab:
name: Warp A/B budget gate
# Always run on schedule / manual dispatch. Never on push — that event only
# feeds build-baseline-cache above.
# Always run on schedule / manual dispatch. Opt-in on PRs: only when the
# `perf-ab` label is present, and for `labeled` events only when the label
# being added is `perf-ab` itself (adding an unrelated label to an opted-in
# PR must not re-run the gate). Never on push — that event only feeds
# build-baseline-cache above.
if: >-
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
contains(github.event.pull_request.labels.*.name, 'perf-ab') &&
(github.event.action != 'labeled' || github.event.label.name == 'perf-ab'))
runs-on: sm-standard-2
# With perf-3's cached baseline binary the common (cache-hit) nightly is
# measurement-only and finishes well under 50min. This ceiling stays
@@ -134,7 +142,6 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
fetch-depth: 0 # baseline is built from origin/main
- name: Setup Rust environment
@@ -143,6 +150,7 @@ jobs:
rust-version: stable
cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install warp
run: |
@@ -154,11 +162,13 @@ jobs:
- name: Decide exemption
id: exempt
env:
INPUT_ALLOW_REGRESSION: ${{ github.event.inputs.allow_regression }}
run: |
allow="false"
if [[ "$INPUT_ALLOW_REGRESSION" == "true" ]]; then
if [[ "${{ github.event_name }}" == "pull_request" ]] \
&& ${{ contains(github.event.pull_request.labels.*.name, 'perf-deliberate-tradeoff') }}; then
allow="true"
fi
if [[ "${{ github.event.inputs.allow_regression }}" == "true" ]]; then
allow="true"
fi
echo "allow_regression=$allow" >> "$GITHUB_OUTPUT"
@@ -205,34 +215,8 @@ jobs:
cp target/release/rustfs baseline-bin/rustfs
echo "built=true" >> "$GITHUB_OUTPUT"
- name: Build baseline on cache miss (different candidate)
id: baseline_build
if: >-
steps.baseline_cache.outputs.cache-hit != 'true' &&
steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
run: |
set -euo pipefail
baseline_root="$RUNNER_TEMP/rustfs-baseline-${{ github.run_id }}"
baseline_target="$RUNNER_TEMP/rustfs-baseline-target-${{ github.run_id }}"
git worktree add --detach "$baseline_root" "${{ steps.commits.outputs.baseline_sha }}"
cargo build --release --manifest-path "$baseline_root/Cargo.toml" --bin rustfs --target-dir "$baseline_target"
mkdir -p baseline-bin
cp "$baseline_target/release/rustfs" baseline-bin/rustfs
git worktree remove --force "$baseline_root"
echo "built=true" >> "$GITHUB_OUTPUT"
- name: Build candidate binary
id: candidate_build
if: steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
run: |
set -euo pipefail
cargo build --release --bin rustfs
mkdir -p candidate-bin
cp target/release/rustfs candidate-bin/rustfs
echo "built=true" >> "$GITHUB_OUTPUT"
- name: Save self-healed baseline to cache
if: steps.selfheal.outputs.built == 'true' || steps.baseline_build.outputs.built == 'true'
if: steps.selfheal.outputs.built == 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
with:
path: baseline-bin/rustfs
@@ -240,66 +224,62 @@ jobs:
- name: Run warp A/B and gate
id: ab
env:
INPUT_DURATION: ${{ github.event.inputs.duration }}
run: |
set -euo pipefail
# The formal runner executes A1 baseline -> B1 candidate -> B2 candidate
# -> A2 baseline for each workload and drive-sync cell. It requires three
# rounds per leg to emit tail latency and error-rate evidence.
# Budget note: with perf-3's cached baseline the nightly does no source
# build on a cache hit, so the wall-clock is dominated by the short warp
# matrix — duration/rounds/cooldown are kept small to fit all 24 cells
# (6 workloads x 2 phases x 2 drive-sync) rather than dropping cells.
# --health-timeout 180 outlasts the server's own 120s startup-readiness
# budget, which the rig's previous 60s health poll undershot (the first
# two nightly failures). perf-6 recalibrates these once the noise study
# lands.
duration="${INPUT_DURATION:-12s}"
duration="${{ github.event.inputs.duration || '12s' }}"
baseline_sha="${{ steps.commits.outputs.baseline_sha }}"
candidate_sha="${{ steps.commits.outputs.candidate_sha }}"
baseline_hit="${{ steps.baseline_cache.outputs.cache-hit }}"
selfheal_built="${{ steps.selfheal.outputs.built }}"
baseline_built="${{ steps.baseline_build.outputs.built }}"
candidate_built="${{ steps.candidate_build.outputs.built }}"
args=(--duration "$duration" --rounds 3 --cooldown 5 --health-timeout 180 --baseline-revision "$baseline_sha" --candidate-revision "$candidate_sha")
args=(--duration "$duration" --rounds 2 --cooldown 5 --health-timeout 180)
if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" || "$baseline_built" == "true" ]]; then
if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" ]]; then
chmod +x baseline-bin/rustfs
base_bin="$PWD/baseline-bin/rustfs"
args+=(--baseline-bin "$base_bin")
if [[ "$baseline_hit" == "true" ]]; then
base_src="actions-cache (rustfs-baseline-$baseline_sha)"
elif [[ "$selfheal_built" == "true" ]]; then
base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
else
base_src="isolated origin/main source build (saved as rustfs-baseline-$baseline_sha)"
base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
fi
if [[ "$candidate_sha" == "$baseline_sha" ]]; then
# Nightly on main: the candidate is the same commit as the baseline,
# so reuse the one binary for both phases and skip all builds.
args+=(--candidate-bin "$base_bin")
args+=(--candidate-bin "$base_bin" --skip-build)
cand_src="same binary as baseline (same commit)"
elif [[ "$candidate_built" == "true" ]]; then
chmod +x candidate-bin/rustfs
args+=(--candidate-bin "$PWD/candidate-bin/rustfs")
cand_src="source build of the checked-out ref"
else
echo "::error::candidate binary was not built" >&2
exit 2
cand_src="source build of the checked-out ref"
fi
else
echo "::error::baseline binary was not restored or built" >&2
exit 2
# Cache miss with candidate != baseline (opt-in PR gate only): fall
# back to the source double-build. With the post-#4806 LTO profile
# this will overrun the job budget and alert; rerun once the push
# cache build for origin/main has completed, or wait for perf-7's
# merge-base caching.
args+=(--baseline-ref origin/main)
base_src="source build of origin/main (cache miss)"
cand_src="source build of the checked-out ref"
fi
echo "baseline binary: $base_src"
echo "candidate binary: $cand_src"
args+=(--provenance-note "baseline commit: $baseline_sha - $base_src")
args+=(--provenance-note "candidate commit: $candidate_sha - $cand_src")
if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
args+=(--allow-regression --exemption-reason "workflow dispatch override")
args+=(--allow-regression --exemption-reason "labeled perf-deliberate-tradeoff / dispatch override")
fi
# Do not let a gate FAIL abort the job here; capture status and surface
# it after the step summary is written.
# it after the PR comment is posted.
set +e
bash scripts/run_hotpath_warp_abba.sh "${args[@]}"
bash scripts/run_hotpath_warp_ab.sh "${args[@]}"
echo "status=$?" >> "$GITHUB_OUTPUT"
set -e
# Locate the newest run dir + gate.md for the summary/comment/artifact
@@ -307,10 +287,10 @@ jobs:
# holds server-logs/ for diagnosis.
# Run dirs are UTC-timestamp names (no special chars); ls is safe here.
# shellcheck disable=SC2012
run_dir="$(ls -td target/hotpath-abba/*/ 2>/dev/null | head -n1 || true)"
run_dir="$(ls -td target/hotpath-ab/*/ 2>/dev/null | head -n1 || true)"
echo "run_dir=${run_dir%/}" >> "$GITHUB_OUTPUT"
# shellcheck disable=SC2012
gate_md="$(ls -t target/hotpath-abba/*/candidate_gate.md 2>/dev/null | head -n1 || true)"
gate_md="$(ls -t target/hotpath-ab/*/gate.md 2>/dev/null | head -n1 || true)"
echo "gate_md=$gate_md" >> "$GITHUB_OUTPUT"
- name: Upload A/B results
@@ -318,10 +298,10 @@ jobs:
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: hotpath-warp-ab-${{ github.run_number }}
# Includes per-cell median_summary.csv / baseline_compare.csv, both gates,
# Includes per-cell median_summary.csv / baseline_compare.csv, gate.md,
# and server-logs/ (rustfs.log + startup env per phase) so a failed run
# is diagnosable. Short retention: this is churny nightly debug data.
path: target/hotpath-abba/
path: target/hotpath-ab/
if-no-files-found: warn
retention-days: 14
@@ -362,6 +342,13 @@ jobs:
fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Comment gate result on PR
if: always() && github.event_name == 'pull_request' && steps.ab.outputs.gate_md != ''
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr comment "${{ github.event.pull_request.number }}" --body-file "${{ steps.ab.outputs.gate_md }}"
# Scheduled failure alerting is handled by the alert-on-failure job below
# (perf-2 consuming ci-8's schedule-failure-issue composite action).
@@ -370,7 +357,7 @@ jobs:
run: |
status="${{ steps.ab.outputs.status }}"
if [[ "$status" != "0" ]]; then
echo "::error::warp A/B budget gate failed (exit $status). See the step summary / gate.md artifact." >&2
echo "::error::warp A/B budget gate failed (exit $status). See the step summary / PR comment / gate.md artifact." >&2
exit "$status"
fi
echo "warp A/B budget gate passed."
@@ -380,12 +367,14 @@ jobs:
needs: [warp-ab]
# `always()` is required: without it this job is skipped when a needed
# job fails. Alerts only for scheduled (nightly) runs (backlog#1149
# ci-8); manual dispatch failures are already watched by a human.
# ci-8); PR and manual dispatch failures are already watched by a human.
# `cancelled` is included alongside `failure` on purpose: a job that hits
# timeout-minutes ends as `cancelled`, and the 2026-07-11..07-14 nightly
# timeouts went silent precisely because the guard was failure-only. The
# composite action already reports cancelled/timed-out jobs in the issue
# body.
# body. (Scheduled runs get a unique concurrency group with
# cancel-in-progress off, so a cancellation here means a timeout/manual
# abort, never a superseding run.)
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
@@ -396,8 +385,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
-81
View File
@@ -1,81 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Asserts that the self-hosted runners are still ephemeral — one job per pod.
#
# This repository is public and its pull_request jobs run on those runners,
# executing the PR's own build.rs, proc-macros and tests. The only thing keeping
# that code from reaching a later job is that each ARC pod handles exactly one
# job and is then destroyed. That guarantee lives in the ARC scale-set
# configuration, outside this repository, where it can be changed without any PR
# — so it is asserted here from the outside, against real run data, instead of
# being assumed.
#
# Monthly rather than per-PR: the property changes only when someone
# reconfigures the scale set, and the check costs a few dozen API calls.
# See docs/ci/runners.md and rustfs/backlog#1602.
name: Runner Hygiene
on:
schedule:
- cron: "0 6 1 * *" # Monthly, 1st at 06:00 UTC (after the daily audit cron)
workflow_dispatch:
permissions:
contents: read
concurrency:
group: runner-hygiene
cancel-in-progress: false
jobs:
check-ephemerality:
name: Check runner ephemerality
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Exit 2 (inconclusive / broken) is deliberately not a pass: a window
# where every sm-* job was still queued would otherwise look identical to
# a clean bill of health.
- name: Assert one job per self-hosted runner
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: ./scripts/ci/check_runner_ephemerality.sh 40
alert-on-failure:
name: Alert on scheduled failure
needs: [check-ephemerality]
# Same ci-8 mechanism as coverage.yml, audit.yml and the nightly lanes:
# scheduled runs file a tracking issue, manual dispatch stays quiet so
# debugging never produces a spurious alert.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -24,13 +24,6 @@
# The run itself is expected to end red (the forced failure); only the
# alert-on-failure job result matters.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Schedule Failure Alert Drill
on:
@@ -63,8 +56,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
-8
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: "Mark stale issues"
on:
schedule:
@@ -27,7 +20,6 @@ on:
jobs:
stale:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/stale@5bef64f19d7facfb25b37b414482c7164d639639 # v9
with:
-1
View File
@@ -15,7 +15,6 @@ concurrency:
jobs:
update:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: overtrue/repo-visuals-action@72f34d24769ff5d341956da2f23952594ef2f1e2 # v1.3.0
with:
-86
View File
@@ -1,86 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Windows Filesystem Tests
on:
push:
branches: [ main ]
paths:
- "crates/ecstore/src/disk/**"
- "crates/ecstore/src/store/init_format.rs"
- "crates/ecstore/Cargo.toml"
- "Cargo.toml"
- "Cargo.lock"
- ".github/actions/setup/**"
- ".github/workflows/windows-filesystem.yml"
pull_request:
branches: [ main ]
paths:
- "crates/ecstore/src/disk/**"
- "crates/ecstore/src/store/init_format.rs"
- "crates/ecstore/Cargo.toml"
- "Cargo.toml"
- "Cargo.lock"
- ".github/actions/setup/**"
- ".github/workflows/windows-filesystem.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
env:
CARGO_TERM_COLOR: always
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUST_BACKTRACE: 1
jobs:
rename-safety:
name: Rename Safety
runs-on: windows-latest
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: build-x86_64-pc-windows-msvc
cache-save-if: 'false'
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Test guarded rename publication
shell: pwsh
run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture
- name: Test Windows handle guards
shell: pwsh
run: cargo test -p rustfs-ecstore --lib windows_ -- --nocapture
- name: Test startup temporary-directory cleanup
shell: pwsh
run: cargo test -p rustfs-ecstore --lib cleanup_tmp_on_startup_ -- --nocapture
- name: Test fresh format publication
shell: pwsh
run: cargo test -p rustfs-ecstore --lib fresh_format_load_initializes_all_disks -- --nocapture
-4
View File
@@ -83,7 +83,3 @@ worktrees/*
# Local AI-agent review artifacts (omo evidence dumps)
.omo/
# insta scratch files; the accepted .snap files ARE the assertions and are committed
*.snap.new
*.pending-snap
-116
View File
@@ -1,116 +0,0 @@
---
name: issue-triage
description: Triage a GitHub issue — determine if it is already fixed, needs implementation, or should be closed. Searches related commits and PRs, verifies implementation status, and posts a triage comment or closes the issue. Use when the user provides an issue URL and asks whether it can be closed or needs work.
---
# Issue Triage
Use this skill when the user provides a GitHub issue URL and asks "can this be closed?", "is this already implemented?", "check completion status", or similar triage questions.
## Workflow
### 1. Fetch issue context
```bash
gh issue view <N> --repo <owner/repo> --json title,body,state,comments,labels,updatedAt
```
Read the issue body to understand what was requested. Extract:
- The specific feature/fix/behavior described.
- Any linked PRs or commits mentioned in the body or comments.
- Any checklist items or sub-issues.
### 2. Search for related work
Search git history for commits referencing the issue:
```bash
git log --oneline --all --grep="<N>" | head -30
```
Search for related PRs:
```bash
gh pr list --search "fixes #<N> OR closes #<N> OR #<N>" --state all --json number,title,state,mergedAt
```
If the issue mentions specific PRs, check their status:
```bash
gh pr view <PR_N> --json state,mergedAt,title
```
### 3. Verify implementation
For each linked or related PR that is merged, verify the fix is actually present on the current main branch:
```bash
git log --oneline main | grep -i "<keyword>"
# or
git log --oneline main --grep="<PR_N>"
```
If the issue describes a specific defect, check the relevant code to confirm the fix is in place:
```bash
grep -n "<pattern>" crates/<relevant>/src/<file>.rs
```
For issues with checklists, verify each item individually. If sub-items are tracked as separate issues, check those too:
```bash
gh issue view <SUB_N> --repo <owner/repo> --json state
```
### 4. Determine verdict
- **All items fixed and merged**: Close with a summary comment listing what was fixed and which PRs.
- **Some items fixed, some remaining**: Comment with status of each item. Do not close.
- **Not yet implemented**: Comment with a summary of what remains. Do not close.
- **Superseded or no longer relevant**: Close with explanation.
### 5. Take action
Close with comment:
```bash
gh issue close <N> --repo <owner/repo> --comment "<body>"
```
Comment without closing:
```bash
gh issue comment <N> --repo <owner/repo> --body-file /tmp/triage.md
```
Update issue labels if needed:
```bash
gh issue edit <N> --repo <owner/repo> --add-label "completed" --remove-label "needs-triage"
```
Always use `--body-file` for multiline content, never inline `--body`.
### 6. Handle multi-issue batches
When the user asks to check multiple issues (e.g., "check all issues by user X" or "scan backlog for closable issues"):
1. List the issues: `gh issue list --repo <repo> --author <user> --state open --json number,title,updatedAt`
2. For each issue, run steps 1-5 above.
3. Report a summary table of all triaged issues with verdicts.
## Output format
### Issue Triage: #<N> — <title>
**State**: OPEN / CLOSED
**Linked PRs**: <list with merge status>
#### Assessment
<what was requested vs what is implemented>
#### Verdict
- Close — all items resolved by <PR list>
- Keep open — <remaining items>
- Not started — <what needs to be done>
#### Action taken
- Closed with comment / Commented / No action
## Notes
- The user may ask in Chinese ("是否可以关闭", "检查完成情况"); respond in the same language.
- When closing, always include a summary of what was fixed and which PRs resolved it — this creates a useful audit trail.
- For issues in `rustfs/backlog`, use `--repo rustfs/backlog`.
- For issues in `rustfs/rustfs`, use `--repo rustfs/rustfs`.
- If the issue has sub-issues (GitHub sub-issues API), check each one's state before declaring the parent complete.
-147
View File
@@ -1,147 +0,0 @@
---
name: pr-review
description: Review a GitHub PR end-to-end from a URL or number — fetch metadata, inspect the diff, run multi-role adversarial review, check CI status, and post the review comment. Use when the user provides a PR link and asks to review it.
---
# PR Review
Use this skill when the user provides a GitHub PR URL or number and asks to review it. This covers the full review lifecycle: data gathering, code review, CI verification, and posting the result.
## Prerequisites
- Read `AGENTS.md` for the repository's adversarial validation policy and change-style rules.
- The `adversarial-validation` skill handles the review role playbooks; this skill orchestrates the workflow around it.
## Workflow
### 1. Gather PR context
```bash
gh pr view <N> --json title,author,state,body,additions,deletions,changedFiles,commits,baseRefName,headRefName
gh pr diff <N> --name-only
```
Read the PR body and linked issues to understand the change's purpose. If the PR references an issue, fetch that too:
```bash
gh issue view <ISSUE> --json title,body,state
```
### 2. Fetch the diff and classify the change
```bash
git fetch origin pull/<N>/head:pr-<N>
git diff main...pr-<N> --stat
```
Classify the change by risk tier (per AGENTS.md):
- **Exempt**: docs/comments/instruction-only, formatting, typos.
- **Mechanical**: renames, file moves, test-only or tooling changes.
- **Standard** (default): any behavior change.
- **High risk**: locking, erasure coding, quorum/heal, replication, multipart, RPC, lifecycle/tiering, metadata formats, persistence/fsync, IAM/KMS/auth, on-disk/on-wire formats, S3 API-visible behavior.
### 3. Cluster changed files and delegate review
Group the changed files into logical clusters (by crate or functional area). For each cluster, spawn a subagent with a focused review prompt that includes:
- The cluster's changed files and their diffs.
- The applicable adversarial role probes (from the `adversarial-validation` skill).
- The repository's AGENTS.md rules relevant to that domain.
For standard-tier changes: correctness adversary + simplicity adversary + test-coverage skeptic, plus every role whose domain the diff touches.
For high-risk changes: run all seven roles.
Each subagent must produce findings (concrete failure scenario with file:line) or a null report ("attacked X, Y, Z — no break found").
### 4. Check CI status
```bash
gh pr checks <N>
```
If any checks fail, investigate:
```bash
gh run view --log-failed --job=<JOB_ID>
```
Determine whether failures are pre-existing (on main), flaky, or caused by the PR.
### 5. Synthesize findings
Combine all subagent findings into a structured review:
- **Summary**: one-paragraph overview of the change and overall assessment.
- **Findings**: each finding with severity (critical/major/minor/nit), file:line, concrete failure scenario, and suggested fix.
- **CI status**: pass/fail with notes on any failures.
- **Verdict**: APPROVE, REQUEST_CHANGES, or COMMENT.
### 6. Post the review
Write the review body to a temp file and post via CLI:
```bash
# Request changes
gh pr review <N> --request-changes --body-file /tmp/pr_review.md
# Approve
gh pr review <N> --approve --body-file /tmp/pr_review.md
# Comment only (no verdict)
gh pr review <N> --comment --body-file /tmp/pr_review.md
```
For inline comments on specific lines, use the GitHub API:
```bash
cat > /tmp/pr_review.json <<'EOF'
{
"body": "review body",
"event": "REQUEST_CHANGES",
"comments": [
{
"path": "crates/foo/src/bar.rs",
"line": 42,
"body": "finding description"
}
]
}
EOF
gh api --method POST /repos/{owner}/{repo}/pulls/<N>/reviews --input /tmp/pr_review.json
```
Always use `--body-file` or `--input`, never inline multiline `--body`.
### 7. Handle follow-up
If the review requests changes:
- Monitor for new commits: `gh pr view <N> --json commits`
- Re-review changed files only: `git diff pr-<N>..origin/pull/<N>/head`
- Update the review when findings are addressed.
If CI was failing due to pre-existing main breakage:
- Comment on the PR noting the failure is pre-existing.
- Suggest updating the branch: `gh pr update-branch <N>`
## Output format
### PR Review: #<N> — <title>
**Author**: <author>
**Risk tier**: exempt | mechanical | standard | high-risk
**Changed files**: <count> across <cluster count> clusters
#### Summary
<one-paragraph overview>
#### Findings
| Severity | Location | Finding |
|----------|----------|---------|
| critical | file:line | concrete failure scenario |
#### CI Status
- All checks pass / Failing: <details>
#### Verdict
APPROVE / REQUEST_CHANGES / COMMENT
## Notes
- The user may ask for review in Chinese; respond in the same language but keep the review body in English per AGENTS.md rules.
- When the user asks for "多角色对抗 review", run the full adversarial validation protocol — this skill's step 3 covers that.
- If the PR is from a fork, check `maintainerCanModify` before attempting to push fixes.
- For very large PRs (>50 files), cluster aggressively and delegate in parallel to keep review time reasonable.
-43
View File
@@ -21,22 +21,6 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
## Worktree and Disk Hygiene
- Unless the requester explicitly says otherwise, treat every new implementation task as isolated work: fetch the latest `origin/main`, confirm the requested change is not already present there, and create a dedicated feature branch and worktree from that exact upstream commit before editing. Do not implement new work directly in the primary checkout or reuse a worktree from another task.
- Check available disk space before creating the worktree or starting dependency downloads, builds, tests, coverage, or other artifact-heavy commands. For long-running or artifact-heavy work, re-check disk usage at natural phase boundaries and before broad validation; if remaining space may not safely accommodate the next command, stop and reclaim task-owned artifacts before continuing.
- Keep cleanup scoped and safe: remove generated build/test/coverage artifacts and temporary files created by the task when they are no longer needed, and never delete another task's worktree or uncommitted files. Prefer shared dependency caches where supported instead of duplicating large artifacts across worktrees.
- At handoff, report the disk-space checks, cleanup performed, and any retained worktree or artifacts with the reason they are still needed.
## PR Lifecycle Monitoring
- Creating or updating a PR is not the terminal state. Unless the requester explicitly limits the task to PR creation, monitor the PR through its terminal state: merged, closed, or explicitly handed off because progress requires user or maintainer action.
- While the task is active, monitor CI/check runs, review decisions and unresolved threads, mergeability and conflicts, and unexpected head/base changes. Prefer event-driven or bounded waits provided by the current environment over frequent polling; report only state changes, actionable failures, or meaningful prolonged delays.
- Investigate every failing check and review comment before changing code. Fix failures attributable to the task, run the verification required for the new diff, push the update, respond to or resolve the corresponding review threads, and resume monitoring. Do not weaken checks, dismiss valid feedback, or retry flaky failures merely to obtain a green result.
- Treat opening, green CI, approval, and mergeability as intermediate states. Never merge without the required reviewer approval or explicit authority. If progress depends on credentials, infrastructure, a maintainer decision, or another external action, report the exact blocker and the evidence already collected.
- If the current execution environment cannot remain active until the next PR event, use a supported automation, monitor, or thread wakeup when available and within scope. Otherwise leave an explicit handoff containing the PR, current state, next event to observe, and pending cleanup; do not imply that background monitoring exists when none is scheduled.
- After observing a merge, verify the commits are preserved on the upstream base, ensure the worktree is clean, remove the dedicated worktree, prune stale worktree metadata, and delete the local task branch when it is no longer in use. For a closed or abandoned PR, preserve any unmerged work unless deletion was explicitly authorized. Do not delete remote branches unless explicitly requested or repository automation owns that cleanup.
## Autonomy and Approval Boundaries
- Inquiry tasks (answer, explain, review, diagnose, plan): report findings; do not change files unless a fix is explicitly requested.
@@ -322,28 +306,6 @@ High risk: all seven roles.
- Use environment variables or vault tooling for sensitive configuration.
- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
## Logging
Applies to **every** `tracing` macro you add or edit, including a single line
added in passing while fixing something else — not only to log-focused changes.
- Fields first, message second: `event`, `component`, `subsystem`,
`result`/`state`, then key context. The message is a short label, not a
sentence with values interpolated into it.
- Reuse the existing `EVENT_*` / `LOG_COMPONENT_*` / `LOG_SUBSYSTEM_*`
constants of the module you are editing; match the shape of the log sites
already in that file rather than introducing a second style next to them.
- Level policy: `error` for behavior/security-affecting failures, `warn` for
degraded or fallback paths, `info` for low-frequency lifecycle, `debug` for
targeted diagnostics, `trace` for hot paths. Per-object and per-request
success paths are `trace`.
- Never log secrets, tokens, credential payloads, or merged config dumps.
- `scripts/check_logging_guardrails.sh` enforces a subset of this on the files
it lists; passing it is a floor, not evidence the log matches the house style.
See `.agents/skills/rustfs-logging-governance/SKILL.md` for the full event
model, level policy, and guardrail-update checklist.
## Tools
### xl.meta decode tool Quick Use
@@ -369,11 +331,6 @@ cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
absent, empty, and nil all mean "no value", never `Uuid::nil()`.
- A remote-tier version of `None`/`""` means the tier bucket is unversioned:
send **no** `versionId` on tier GET/DELETE.
- Structs persisted in the scanner data-usage cache (`DataUsageCacheInfo`,
`DataUsageEntry`) carry a hand-written map-encoded `Serialize`. MessagePack
encodes derived structs as arrays, where an appended field makes the whole
cache a decode error for older readers — keep new fields `#[serde(default)]`
and keep the map encoding rather than reverting to `derive(Serialize)`.
## Naming Conventions
Generated
+392 -543
View File
File diff suppressed because it is too large Load Diff
+61 -62
View File
@@ -69,7 +69,7 @@ edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.97.1"
version = "1.0.0-rc.1"
version = "1.0.0-beta.12"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -86,52 +86,52 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.0-rc.1" }
rustfs-heal = { path = "crates/heal", version = "1.0.0-rc.1" }
rustfs-audit = { path = "crates/audit", version = "1.0.0-rc.1" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.0-rc.1" }
rustfs-common = { path = "crates/common", version = "1.0.0-rc.1" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-rc.1" }
rustfs-config = { path = "./crates/config", version = "1.0.0-rc.1" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-rc.1" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.0-rc.1" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.0-rc.1" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-rc.1" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-rc.1" }
rustfs-iam = { path = "crates/iam", version = "1.0.0-rc.1" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.0-rc.1" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-rc.1" }
rustfs-kms = { path = "crates/kms", version = "1.0.0-rc.1" }
rustfs-lock = { path = "crates/lock", version = "1.0.0-rc.1" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.0-rc.1" }
rustfs-notify = { path = "crates/notify", version = "1.0.0-rc.1" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-rc.1" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.0-rc.1" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-rc.1" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-rc.1", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-rc.1" }
rustfs-obs = { path = "crates/obs", version = "1.0.0-rc.1" }
rustfs-policy = { path = "crates/policy", version = "1.0.0-rc.1" }
rustfs-protos = { path = "crates/protos", version = "1.0.0-rc.1" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.0-rc.1" }
rustfs-replication = { path = "crates/replication", version = "1.0.0-rc.1" }
rustfs-rio = { path = "crates/rio", version = "1.0.0-rc.1" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-rc.1" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-rc.1" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-rc.1" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-rc.1" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-rc.1" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.0-rc.1" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-rc.1" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-rc.1" }
rustfs-signer = { path = "crates/signer", version = "1.0.0-rc.1" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-rc.1" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-rc.1" }
rustfs-targets = { path = "crates/targets", version = "1.0.0-rc.1" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-rc.1" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-rc.1" }
rustfs-utils = { path = "crates/utils", version = "1.0.0-rc.1" }
rustfs-zip = { path = "./crates/zip", version = "1.0.0-rc.1" }
rustfs = { path = "./rustfs", version = "1.0.0-beta.12" }
rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.12" }
rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.12" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.12" }
rustfs-common = { path = "crates/common", version = "1.0.0-beta.12" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.12" }
rustfs-config = { path = "./crates/config", version = "1.0.0-beta.12" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.12" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.12" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.12" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.12" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.12" }
rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.12" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.12" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.12" }
rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.12" }
rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.12" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.12" }
rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.12" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.12" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.12" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.12" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.12" }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.12" }
rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.12" }
rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.12" }
rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.12" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.12" }
rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.12" }
rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.12" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.12" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.12" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.12" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.12" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.12" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.12" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.12" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.12" }
rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.12" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.12" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.12" }
rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.12" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.12" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.12" }
rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.12" }
rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.12" }
# Async Runtime and Networking
async-channel = "2.5.0"
@@ -139,7 +139,7 @@ async_zip = { default-features = false, version = "0.0.18" }
mysql_async = { default-features = false, version = "0.37" }
async-compression = { version = "0.4.43" }
async-recursion = "1.1.1"
async-trait = "0.1.92"
async-trait = "0.1.91"
async-nats = { version = "0.50.0", default-features = false }
axum = "0.8.9"
futures = "0.3.33"
@@ -173,7 +173,7 @@ tower-http = { version = "0.7.0" }
# Serialization and Data Formats
apache-avro = "0.21.0"
bytes = { version = "1.12.1" }
bytesize = "2.7.0"
bytesize = "2.4.2"
byteorder = "1.5.0"
flatbuffers = "25.12.19"
form_urlencoded = "1.2.2"
@@ -212,7 +212,7 @@ zeroize = { version = "1.9.0" }
chrono = { version = "0.4.45" }
humantime = "2.4.0"
jiff = { version = "0.2.35" }
time = { version = "0.3.55" }
time = { version = "0.3.54" }
# Database
deadpool-postgres = { version = "0.14" }
@@ -227,16 +227,15 @@ atoi = "3.1.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.10.1" }
aws-credential-types = { version = "1.3.0" }
aws-sdk-kms = { default-features = false, version = "1.114.0" }
aws-sdk-s3 = { default-features = false, version = "1.141.0" }
aws-sdk-s3 = { default-features = false, version = "1.140.0" }
aws-sdk-sts = { default-features = false, version = "1.110.0" }
aws-smithy-http-client = { default-features = false, version = "1.2.0" }
aws-smithy-runtime-api = { version = "1.14.0" }
aws-smithy-types = { version = "1.6.1" }
base64 = "0.23.1"
base64 = "0.23.0"
base64-simd = "0.8.0"
brotli = "8.0.4"
clap = { version = "4.6.6" }
clap = { version = "4.6.4" }
const-str = { version = "1.1.0" }
convert_case = "0.11.0"
criterion = { version = "0.8" }
@@ -244,7 +243,7 @@ crossbeam-queue = "0.3.13"
crossbeam-channel = "0.5.16"
crossbeam-deque = "0.8.7"
crossbeam-utils = "0.8.22"
datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "e08aed1e5de41dcf81d529140dae07723b942a5e" }
datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
#datafusion = { default-features = false, version = "54.1.0" }
derive_builder = "0.20.2"
enumset = "1.1.14"
@@ -274,6 +273,7 @@ num_cpus = { version = "1.17.0" }
nvml-wrapper = "0.12.1"
parking_lot = "0.12.5"
path-absolutize = "4.0.1"
path-clean = "1.0.1"
percent-encoding = "2.3.2"
pin-project-lite = "0.2.17"
pretty_assertions = "1.4.1"
@@ -302,7 +302,7 @@ sysinfo = "0.39.6"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.3.1"
thiserror = "2.0.20"
thiserror = "2.0.19"
tracing = { version = "0.1.44" }
tracing-appender = "0.2.5"
tracing-core = "0.1.36"
@@ -340,21 +340,20 @@ libunftp = { version = "0.23.0" }
unftp-core = "0.1.0"
suppaftp = { version = "10.0.1" }
rcgen = { version = "0.14.8", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.62.5" }
russh-sftp = "2.4.0"
russh = { version = "0.62.4" }
russh-sftp = "2.3.0"
# WebDAV
dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "ce6338661179c8be22e516b00af7483f151485a7" }
libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "ce6338661179c8be22e516b00af7483f151485a7", features = ["extended"] }
hotpath = { version = "0.23.1", default-features = false }
mimalloc = "0.1.52"
hotpath = { version = "0.22.0", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
[workspace.metadata.cargo-shear]
ignored = ["hotpath", "rustfs"]
ignored = ["rustfs"]
[profile.dev]
# Full debuginfo roughly doubles compile+link time and produces multi-GB
+1 -1
View File
@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.1
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
+1 -1
View File
@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.1
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
-3
View File
@@ -47,9 +47,6 @@ consts = "consts"
Hashi = "Hashi" # HashiCorp
# Accept alternate spelling used in parser/XML comments.
unparseable = "unparseable"
# Disaster-recovery objectives: recovery time and recovery point.
RTO = "RTO"
rto = "rto"
[files]
extend-exclude = []
+1 -1
View File
@@ -55,10 +55,10 @@ hotpath.workspace = true
rustfs-targets = { workspace = true }
rustfs-config = { workspace = true, features = ["audit", "server-config-model"] }
rustfs-s3-types = { workspace = true }
chrono = { workspace = true, features = ["serde"] }
const-str = { workspace = true, features = ["std", "proc"] }
futures = { workspace = true }
hashbrown = { workspace = true, features = ["serde", "rayon"] }
jiff = { workspace = true, features = ["serde"] }
metrics = { workspace = true }
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true, features = ["raw_value"] }
+5 -24
View File
@@ -12,8 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use chrono::{DateTime, Utc};
use hashbrown::HashMap;
use jiff::Timestamp;
use rustfs_s3_types::EventName;
use serde::{Deserialize, Serialize};
use serde_json::Value;
@@ -151,8 +151,8 @@ pub struct AuditEntry {
pub deployment_id: Option<String>,
#[serde(rename = "siteName", skip_serializing_if = "Option::is_none")]
pub site_name: Option<String>,
#[serde(with = "jiff::fmt::serde::timestamp::millisecond::required")]
pub time: Timestamp,
#[serde(with = "chrono::serde::ts_milliseconds")]
pub time: DateTime<Utc>,
pub event: EventName,
#[serde(rename = "type", skip_serializing_if = "Option::is_none")]
pub entry_type: Option<String>,
@@ -198,7 +198,7 @@ impl AuditEntryBuilder {
pub fn new(version: impl Into<String>, event: EventName, trigger: impl Into<String>, api: ApiDetails) -> Self {
Self(AuditEntry {
version: version.into(),
time: Timestamp::now(),
time: Utc::now(),
event,
trigger: trigger.into(),
api,
@@ -232,7 +232,7 @@ impl AuditEntryBuilder {
self
}
pub fn time(mut self, time: Timestamp) -> Self {
pub fn time(mut self, time: DateTime<Utc>) -> Self {
self.0.time = time;
self
}
@@ -342,23 +342,4 @@ mod tests {
assert_eq!(value["requestID"], Value::String("req-audit-123".to_string()));
assert!(value.get("request_id").is_none(), "historical audit contract must not expose request_id");
}
#[test]
fn audit_entry_time_serializes_as_epoch_milliseconds() {
let entry = AuditEntryBuilder::new(
"1",
EventName::ObjectCreatedPut,
"s3",
ApiDetailsBuilder::new()
.name("PutObject")
.status("OK")
.status_code(200)
.build(),
)
.time(Timestamp::from_millisecond(1_711_423_698_870).expect("timestamp should be valid"))
.build();
let value = serde_json::to_value(entry).expect("audit entry should serialize");
assert_eq!(value["time"], Value::Number(1_711_423_698_870_i64.into()));
}
}
+3 -3
View File
@@ -97,7 +97,7 @@ async fn test_audit_log_dispatch_performance() {
return; // Alternatively: assert!(false, "AuditSystem failed to start");
}
use jiff::Timestamp;
use chrono::Utc;
use rustfs_targets::EventName;
use serde_json::json;
use std::collections::HashMap;
@@ -136,7 +136,7 @@ async fn test_audit_log_dispatch_performance() {
version: "1".to_string(),
deployment_id: Some(format!("test-deployment-{id}")),
site_name: Some("test-site".to_string()),
time: Timestamp::now(),
time: Utc::now(),
event: EventName::ObjectCreatedPut,
entry_type: Some("object".to_string()),
trigger: "api".to_string(),
@@ -298,7 +298,7 @@ fn test_performance_requirements() {
for i in 0..3000 {
// Simulate event name parsing and processing
let _event_id = format!("s3:ObjectCreated:Put_{i}");
let _timestamp = jiff::Timestamp::now().to_string();
let _timestamp = chrono::Utc::now().to_rfc3339();
// Simulate basic audit entry creation overhead
let _entry_size = 512; // bytes
@@ -264,7 +264,7 @@ fn create_sample_audit_entry() -> AuditEntry {
}
fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
use jiff::Timestamp;
use chrono::Utc;
use rustfs_targets::EventName;
use serde_json::json;
@@ -301,7 +301,7 @@ fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
version: "1".to_string(),
deployment_id: Some(format!("test-deployment-{id}")),
site_name: Some("test-site".to_string()),
time: Timestamp::now(),
time: Utc::now(),
event: EventName::ObjectCreatedPut,
entry_type: Some("object".to_string()),
trigger: "api".to_string(),
-4
View File
@@ -39,15 +39,11 @@ tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
tonic = { workspace = true, features = ["gzip", "deflate"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
chrono = { workspace = true, features = ["serde"] }
jiff = { workspace = true, features = ["serde"] }
metrics = { workspace = true }
serde = { workspace = true, features = ["derive"] }
rmp-serde = { workspace = true }
s3s = { workspace = true, features = ["minio"] }
tracing = { workspace = true }
[dev-dependencies]
serde_json = { workspace = true }
[lib]
doctest = false
-4
View File
@@ -356,8 +356,6 @@ pub struct HealChannelRequest {
pub recursive: Option<bool>,
/// Whether to dry run
pub dry_run: Option<bool>,
/// Whether to skip namespace locking
pub no_lock: Option<bool>,
/// Timeout in seconds (optional)
pub timeout_seconds: Option<u64>,
/// Origin of the request for operational status and queue accounting
@@ -562,7 +560,6 @@ pub fn create_heal_request(
update_parity: None,
recursive: None,
dry_run: None,
no_lock: None,
timeout_seconds: None,
source: HealRequestSource::Internal,
disk: None,
@@ -721,7 +718,6 @@ pub async fn send_heal_disk(set_disk_id: String, priority: Option<HealChannelPri
update_parity: None,
recursive: None,
dry_run: None,
no_lock: None,
timeout_seconds: None,
source: HealRequestSource::AutoHeal,
};
+30 -438
View File
@@ -15,10 +15,9 @@
use crate::heal_channel::HealScanMode;
use crate::last_minute::{AccElem, LastMinuteLatency};
use chrono::{DateTime, Utc};
use jiff::Timestamp;
use serde::{Deserialize, Serialize};
use std::{
collections::{BTreeSet, HashMap},
collections::HashMap,
fmt::Display,
future::Future,
pin::Pin,
@@ -670,7 +669,7 @@ impl LockedLastMinuteLatency {
#[derive(Clone, Debug)]
struct CurrentPathState {
path: String,
updated_at: Timestamp,
updated_at: DateTime<Utc>,
}
struct CurrentPathTracker {
@@ -679,10 +678,10 @@ struct CurrentPathTracker {
impl CurrentPathTracker {
fn new(initial_path: String) -> Self {
Self::new_at(initial_path, Timestamp::now())
Self::new_at(initial_path, Utc::now())
}
fn new_at(initial_path: String, updated_at: Timestamp) -> Self {
fn new_at(initial_path: String, updated_at: DateTime<Utc>) -> Self {
Self {
state: Arc::new(RwLock::new(CurrentPathState {
path: initial_path,
@@ -694,7 +693,7 @@ impl CurrentPathTracker {
async fn update_path(&self, path: String) {
let mut state = self.state.write().await;
state.path = path;
state.updated_at = Timestamp::now();
state.updated_at = Utc::now();
}
async fn get_state(&self) -> CurrentPathState {
@@ -702,36 +701,6 @@ impl CurrentPathTracker {
}
}
fn chrono_to_jiff_timestamp(dt: DateTime<Utc>) -> Timestamp {
let seconds = dt.timestamp();
let nanoseconds = match i32::try_from(dt.timestamp_subsec_nanos()) {
Ok(nanoseconds) => nanoseconds,
Err(_) => {
return if seconds < 0 { Timestamp::MIN } else { Timestamp::MAX };
}
};
match Timestamp::new(seconds, nanoseconds) {
Ok(timestamp) => timestamp,
Err(_) => {
if seconds < 0 {
Timestamp::MIN
} else {
Timestamp::MAX
}
}
}
}
fn timestamp_elapsed_seconds_since(now: Timestamp, earlier: Timestamp) -> u64 {
let duration = now.duration_since(earlier);
if duration.is_negative() {
return 0;
}
u64::try_from(duration.as_secs()).map_or(u64::MAX, |seconds| seconds)
}
#[derive(Clone, Copy, Debug, Default)]
struct ScannerDiskBucketScanState {
concurrency_limit: u64,
@@ -739,48 +708,6 @@ struct ScannerDiskBucketScanState {
active: u64,
}
type ScannerDiskBucketScanKey = (String, String);
#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
pub struct ScannerDiskBucketScanSnapshot {
pub pool: String,
pub set: String,
pub concurrency_limit: u64,
pub queued: u64,
pub active: u64,
}
#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
struct ScannerBucketDriveResultKey {
bucket: String,
drive: String,
result: String,
}
impl ScannerBucketDriveResultKey {
fn new(bucket: impl Into<String>, drive: impl Into<String>, result: impl Into<String>) -> Self {
Self {
bucket: bucket.into(),
drive: drive.into(),
result: result.into(),
}
}
}
const MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS: usize = 4096;
#[derive(Debug, Default)]
struct ScannerBucketDriveResults {
counts: HashMap<ScannerBucketDriveResultKey, ScannerBucketDriveResultValue>,
eviction_index: BTreeSet<(u64, ScannerBucketDriveResultKey)>,
}
#[derive(Clone, Copy, Debug)]
struct ScannerBucketDriveResultValue {
count: u64,
last_seen: u64,
}
// ---------------------------------------------------------------------------
// Metrics
// ---------------------------------------------------------------------------
@@ -811,11 +738,7 @@ pub struct Metrics {
scanner_set_scan_concurrency_limit: AtomicU64,
scanner_set_scans_queued: AtomicU64,
scanner_set_scans_active: AtomicU64,
scanner_disk_bucket_scan_states: Mutex<HashMap<ScannerDiskBucketScanKey, ScannerDiskBucketScanState>>,
scanner_bucket_drive_results: Mutex<ScannerBucketDriveResults>,
scanner_bucket_drive_result_clock: AtomicU64,
current_scan_cycle_bucket_drive_results_start: Mutex<HashMap<ScannerBucketDriveResultKey, u64>>,
last_scan_cycle_bucket_drive_results: Mutex<Vec<ScannerBucketDriveResultSnapshot>>,
scanner_disk_bucket_scan_states: Mutex<HashMap<String, ScannerDiskBucketScanState>>,
scanner_leader_lock_state: RwLock<String>,
scanner_leader_lock_held: AtomicBool,
scanner_leader_lock_last_error: RwLock<String>,
@@ -1035,14 +958,6 @@ pub struct ScannerSourceWorkSnapshot {
pub missed: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
pub struct ScannerBucketDriveResultSnapshot {
pub bucket: String,
pub drive: String,
pub result: String,
pub count: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
pub struct ScannerReplicationRepairSnapshot {
pub source: String,
@@ -1197,12 +1112,12 @@ pub struct ScannerLastMinute {
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
pub struct ScannerMetricsReport {
pub collected_at: Timestamp,
pub collected_at: DateTime<Utc>,
pub current_cycle: u64,
#[serde(default)]
pub current_cycle_active: bool,
pub current_started: Timestamp,
pub cycles_completed_at: Vec<Timestamp>,
pub current_started: DateTime<Utc>,
pub cycles_completed_at: Vec<DateTime<Utc>>,
pub ongoing_buckets: usize,
#[serde(default)]
pub active_scan_paths: usize,
@@ -1375,18 +1290,6 @@ pub struct ScannerMetricsReport {
pub partial_cycles: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
pub struct ScannerRuntimeDetailsReport {
#[serde(default)]
pub disk_bucket_scan_states: Vec<ScannerDiskBucketScanSnapshot>,
#[serde(default)]
pub bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
#[serde(default)]
pub current_cycle_bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
#[serde(default)]
pub last_cycle_bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
}
impl CurrentCycle {
pub fn unmarshal(&mut self, buf: &[u8]) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
*self = rmp_serde::from_slice(buf)?;
@@ -1754,7 +1657,6 @@ pub fn emit_scan_cycle_superseded(duration: Duration) {
pub fn emit_scan_bucket_drive_complete(success: bool, bucket: &str, disk: &str, duration: Duration) {
let result = if success { "success" } else { "error" };
global_metrics().record_scanner_bucket_drive_result(bucket, disk, result);
metrics::counter!(
OTEL_SCANNER_BUCKETS_SCANNED,
"result" => result,
@@ -1771,7 +1673,6 @@ pub fn emit_scan_bucket_drive_complete(success: bool, bucket: &str, disk: &str,
}
pub fn emit_scan_bucket_drive_partial(bucket: &str, disk: &str, duration: Duration) {
global_metrics().record_scanner_bucket_drive_result(bucket, disk, SCAN_CYCLE_RESULT_PARTIAL_LABEL);
metrics::counter!(
OTEL_SCANNER_BUCKETS_SCANNED,
"result" => SCAN_CYCLE_RESULT_PARTIAL_LABEL,
@@ -1822,10 +1723,6 @@ impl Metrics {
scanner_set_scans_queued: AtomicU64::new(0),
scanner_set_scans_active: AtomicU64::new(0),
scanner_disk_bucket_scan_states: Mutex::new(HashMap::new()),
scanner_bucket_drive_results: Mutex::new(ScannerBucketDriveResults::default()),
scanner_bucket_drive_result_clock: AtomicU64::new(0),
current_scan_cycle_bucket_drive_results_start: Mutex::new(HashMap::new()),
last_scan_cycle_bucket_drive_results: Mutex::new(Vec::new()),
scanner_leader_lock_state: RwLock::new("unknown".to_string()),
scanner_leader_lock_held: AtomicBool::new(false),
scanner_leader_lock_last_error: RwLock::new(String::new()),
@@ -2396,7 +2293,7 @@ impl Metrics {
queued: Option<usize>,
active: Option<usize>,
) {
let key = (pool.to_string(), set.to_string());
let key = format!("{pool}/{set}");
let mut states = self
.scanner_disk_bucket_scan_states
.lock()
@@ -2413,41 +2310,6 @@ impl Metrics {
}
}
pub fn record_scanner_bucket_drive_result(&self, bucket: &str, drive: &str, result: &str) {
if bucket.is_empty() || drive.is_empty() || result.is_empty() {
return;
}
let key = ScannerBucketDriveResultKey::new(bucket, drive, result);
let mut results = self
.scanner_bucket_drive_results
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
let last_seen = self.scanner_bucket_drive_result_clock.fetch_add(1, Ordering::Relaxed);
if let Some(previous_last_seen) = results.counts.get_mut(&key).map(|value| {
let previous_last_seen = value.last_seen;
value.count = value.count.saturating_add(1);
value.last_seen = last_seen;
previous_last_seen
}) {
results.eviction_index.remove(&(previous_last_seen, key.clone()));
results.eviction_index.insert((last_seen, key));
return;
}
if results.counts.len() >= MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS
&& let Some((_, stale_key)) = results.eviction_index.pop_first()
{
results.counts.remove(&stale_key);
}
if results.counts.len() < MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
results
.counts
.insert(key.clone(), ScannerBucketDriveResultValue { count: 1, last_seen });
results.eviction_index.insert((last_seen, key));
}
}
// -----------------------------------------------------------------------
// Read-side helpers
// -----------------------------------------------------------------------
@@ -2619,11 +2481,6 @@ impl Metrics {
&self.current_scan_cycle_replication_repair_work_start,
&replication_repair_snapshot,
);
let bucket_drive_results = self.scanner_bucket_drive_result_counts();
match self.current_scan_cycle_bucket_drive_results_start.lock() {
Ok(mut start) => *start = bucket_drive_results,
Err(poisoned) => *poisoned.into_inner() = bucket_drive_results,
}
self.current_scan_cycle_work_active.store(true, Ordering::Release);
snapshot
}
@@ -2636,11 +2493,6 @@ impl Metrics {
self.record_scan_cycle_work(work);
self.record_scan_cycle_source_work(&source_work);
self.record_scan_cycle_replication_repair_work(&replication_repair_work);
let bucket_drive_results = self.current_cycle_bucket_drive_result_snapshots();
match self.last_scan_cycle_bucket_drive_results.lock() {
Ok(mut last) => *last = bucket_drive_results,
Err(poisoned) => *poisoned.into_inner() = bucket_drive_results,
}
self.current_scan_cycle_work_active.store(false, Ordering::Release);
}
@@ -2724,105 +2576,6 @@ impl Metrics {
}
}
fn scanner_bucket_drive_result_counts(&self) -> HashMap<ScannerBucketDriveResultKey, u64> {
self.scanner_bucket_drive_results
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.counts
.iter()
.map(|(key, value)| (key.clone(), value.count))
.collect()
}
fn scanner_bucket_drive_result_snapshots(
counts: impl IntoIterator<Item = (ScannerBucketDriveResultKey, u64)>,
) -> Vec<ScannerBucketDriveResultSnapshot> {
let mut snapshots = counts
.into_iter()
.filter(|(_, count)| *count > 0)
.map(|(key, count)| ScannerBucketDriveResultSnapshot {
bucket: key.bucket,
drive: key.drive,
result: key.result,
count,
})
.collect::<Vec<_>>();
snapshots.sort_by(|left, right| {
left.bucket
.cmp(&right.bucket)
.then_with(|| left.drive.cmp(&right.drive))
.then_with(|| left.result.cmp(&right.result))
});
snapshots
}
fn scanner_bucket_drive_result_counter_snapshots(&self) -> Vec<ScannerBucketDriveResultSnapshot> {
Self::scanner_bucket_drive_result_snapshots(self.scanner_bucket_drive_result_counts())
}
fn current_cycle_bucket_drive_result_snapshots(&self) -> Vec<ScannerBucketDriveResultSnapshot> {
let current = self.scanner_bucket_drive_result_counts();
let start = self
.current_scan_cycle_bucket_drive_results_start
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.clone();
Self::scanner_bucket_drive_result_snapshots(current.into_iter().filter_map(|(key, count)| {
let delta = count.saturating_sub(start.get(&key).copied().unwrap_or_default());
(delta > 0).then_some((key, delta))
}))
}
pub fn scanner_runtime_details_report(&self) -> ScannerRuntimeDetailsReport {
self.scanner_runtime_details_report_for_active(self.current_scan_cycle_work_active.load(Ordering::Acquire))
}
fn scanner_runtime_details_report_for_active(&self, current_cycle_active: bool) -> ScannerRuntimeDetailsReport {
let current_cycle_bucket_drive_results = if current_cycle_active {
self.current_cycle_bucket_drive_result_snapshots()
} else {
Vec::new()
};
ScannerRuntimeDetailsReport {
disk_bucket_scan_states: self.scanner_disk_bucket_scan_state_snapshots(),
bucket_drive_results: self.scanner_bucket_drive_result_counter_snapshots(),
current_cycle_bucket_drive_results,
last_cycle_bucket_drive_results: self
.last_scan_cycle_bucket_drive_results
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.clone(),
}
}
fn scanner_disk_bucket_scan_state_snapshots(&self) -> Vec<ScannerDiskBucketScanSnapshot> {
let mut disk_bucket_scan_states = match self.scanner_disk_bucket_scan_states.lock() {
Ok(states) => states
.iter()
.map(|((pool, set), state)| ScannerDiskBucketScanSnapshot {
pool: pool.clone(),
set: set.clone(),
concurrency_limit: state.concurrency_limit,
queued: state.queued,
active: state.active,
})
.collect::<Vec<_>>(),
Err(poisoned) => poisoned
.into_inner()
.iter()
.map(|((pool, set), state)| ScannerDiskBucketScanSnapshot {
pool: pool.clone(),
set: set.clone(),
concurrency_limit: state.concurrency_limit,
queued: state.queued,
active: state.active,
})
.collect::<Vec<_>>(),
};
disk_bucket_scan_states.sort_by(|left, right| left.pool.cmp(&right.pool).then_with(|| left.set.cmp(&right.set)));
disk_bucket_scan_states
}
fn scanner_source_work_values(&self) -> Vec<ScannerSourceWorkValues> {
ScannerWorkSource::all()
.iter()
@@ -3008,26 +2761,20 @@ impl Metrics {
/// Build a full metrics report snapshot.
pub async fn report(&self) -> ScannerMetricsReport {
self.report_with_runtime_details().await.0
}
pub async fn report_with_runtime_details(&self) -> (ScannerMetricsReport, ScannerRuntimeDetailsReport) {
let mut m = ScannerMetricsReport::default();
let runtime_details;
let has_cycle = {
let cycle = self.cycle_info.read().await;
let has_cycle = if let Some(cycle) = cycle.as_ref() {
m.current_cycle = cycle.current;
m.cycles_completed_at = cycle.cycle_completed.iter().copied().map(chrono_to_jiff_timestamp).collect();
m.current_started = chrono_to_jiff_timestamp(cycle.started);
m.cycles_completed_at = cycle.cycle_completed.clone();
m.current_started = cycle.started;
true
} else {
false
};
m.current_cycle_active = self.current_scan_cycle_work_active.load(Ordering::Acquire);
if m.current_cycle_active {
// Keep cycle_info before cycle-baseline locks so active scrapes cannot mix two cycle identities.
let current_work = self.scan_cycle_work_since(self.current_scan_cycle_work_start());
let current_source_work = self.scanner_source_work_since(&self.current_scan_cycle_source_work_start_values());
let current_replication_repair_work =
@@ -3050,20 +2797,19 @@ impl Metrics {
m.current_cycle_replication_repair =
self.scanner_replication_repair_work_snapshots(&current_replication_repair_work);
}
runtime_details = self.scanner_runtime_details_report_for_active(m.current_cycle_active);
has_cycle
};
if !has_cycle && let Some(init_time) = crate::get_global_init_time().await {
m.current_started = chrono_to_jiff_timestamp(init_time);
m.current_started = init_time;
}
m.collected_at = Timestamp::now();
m.collected_at = Utc::now();
let current_path_snapshots = self.current_path_snapshots().await;
m.active_scan_paths = current_path_snapshots.len();
m.oldest_active_path_age_seconds = current_path_snapshots
.iter()
.map(|(_, state)| timestamp_elapsed_seconds_since(m.collected_at, state.updated_at))
.map(|(_, state)| m.collected_at.signed_duration_since(state.updated_at).num_seconds().max(0) as u64)
.max()
.unwrap_or_default();
m.active_paths = current_path_snapshots
@@ -3080,11 +2826,15 @@ impl Metrics {
m.current_set_scan_concurrency_limit = self.scanner_set_scan_concurrency_limit.load(Ordering::Relaxed);
m.current_set_scans_queued = self.scanner_set_scans_queued.load(Ordering::Relaxed);
m.current_set_scans_active = self.scanner_set_scans_active.load(Ordering::Relaxed);
let disk_bucket_scan_states = self.scanner_disk_bucket_scan_state_snapshots();
let (disk_scan_concurrency_limit, disk_bucket_scans_queued, disk_bucket_scans_active) =
disk_bucket_scan_states.iter().fold((0, 0, 0), |acc, state| {
(acc.0 + state.concurrency_limit, acc.1 + state.queued, acc.2 + state.active)
});
match self.scanner_disk_bucket_scan_states.lock() {
Ok(states) => states.values().fold((0, 0, 0), |acc, state| {
(acc.0 + state.concurrency_limit, acc.1 + state.queued, acc.2 + state.active)
}),
Err(poisoned) => poisoned.into_inner().values().fold((0, 0, 0), |acc, state| {
(acc.0 + state.concurrency_limit, acc.1 + state.queued, acc.2 + state.active)
}),
};
m.current_disk_scan_concurrency_limit = disk_scan_concurrency_limit;
m.current_disk_bucket_scans_queued = disk_bucket_scans_queued;
m.current_disk_bucket_scans_active = disk_bucket_scans_active;
@@ -3253,7 +3003,7 @@ impl Metrics {
m.pacing_pressure = scanner_pacing_pressure(&m);
m.maintenance_control = scanner_maintenance_control(&m);
(m, runtime_details)
m
}
}
@@ -3339,22 +3089,6 @@ impl Drop for CloseDiskGuard {
mod tests {
use super::*;
#[test]
fn scanner_metrics_report_timestamps_serialize_as_rfc3339_utc() {
let report = ScannerMetricsReport {
collected_at: Timestamp::constant(1_700_000_000, 123_456_000),
current_started: Timestamp::constant(1_699_999_940, 0),
cycles_completed_at: vec![Timestamp::constant(1_700_000_060, 987_654_000)],
..Default::default()
};
let value = serde_json::to_value(&report).expect("scanner metrics report should serialize");
assert_eq!(value["collected_at"].as_str(), Some("2023-11-14T22:13:20.123456Z"));
assert_eq!(value["current_started"].as_str(), Some("2023-11-14T22:12:20Z"));
assert_eq!(value["cycles_completed_at"][0].as_str(), Some("2023-11-14T22:14:20.987654Z"));
}
#[tokio::test]
async fn close_disk_guard_runs_cleanup_when_an_early_return_drops_it() {
let (closed_tx, closed_rx) = tokio::sync::oneshot::channel();
@@ -3413,7 +3147,7 @@ mod tests {
#[tokio::test]
async fn report_counts_active_scan_paths() {
let metrics = Metrics::new();
let updated_at = Timestamp::now() - jiff::SignedDuration::from_secs(12);
let updated_at = Utc::now() - chrono::Duration::seconds(12);
metrics.current_paths.write().await.insert(
"disk-a".to_string(),
Arc::new(CurrentPathTracker::new_at("bucket-a".to_string(), updated_at)),
@@ -3435,7 +3169,7 @@ mod tests {
let metrics = Metrics::new();
let tracker = Arc::new(CurrentPathTracker::new_at(
"bucket-a".to_string(),
Timestamp::now() - jiff::SignedDuration::from_secs(60 * 60),
Utc::now() - chrono::Duration::hours(1),
));
metrics
.current_paths
@@ -4208,7 +3942,7 @@ mod tests {
let report = metrics.report().await;
*crate::globals::GLOBAL_INIT_TIME.write().await = previous_init_time;
assert_eq!(report.current_started, chrono_to_jiff_timestamp(cycle_started));
assert_eq!(report.current_started, cycle_started);
}
#[tokio::test]
@@ -4366,137 +4100,6 @@ mod tests {
assert_eq!(report.life_time_ops.get("scan_bucket_drive_failure"), Some(&1));
}
#[tokio::test]
async fn report_includes_structured_bucket_drive_results() {
let metrics = Metrics::new();
metrics.record_scanner_bucket_drive_result("photos", "/data1", "success");
let cycle_start = metrics.start_scan_cycle_work();
metrics.record_scanner_bucket_drive_result("photos", "/data1", "partial");
let active_report = metrics.scanner_runtime_details_report();
assert_eq!(
active_report.current_cycle_bucket_drive_results,
vec![ScannerBucketDriveResultSnapshot {
bucket: "photos".to_string(),
drive: "/data1".to_string(),
result: "partial".to_string(),
count: 1,
}]
);
metrics.finish_scan_cycle_work(cycle_start);
let report = metrics.scanner_runtime_details_report();
assert_eq!(
report.bucket_drive_results,
vec![
ScannerBucketDriveResultSnapshot {
bucket: "photos".to_string(),
drive: "/data1".to_string(),
result: "partial".to_string(),
count: 1,
},
ScannerBucketDriveResultSnapshot {
bucket: "photos".to_string(),
drive: "/data1".to_string(),
result: "success".to_string(),
count: 1,
},
]
);
assert!(report.current_cycle_bucket_drive_results.is_empty());
assert_eq!(
report.last_cycle_bucket_drive_results,
vec![ScannerBucketDriveResultSnapshot {
bucket: "photos".to_string(),
drive: "/data1".to_string(),
result: "partial".to_string(),
count: 1,
}]
);
}
#[tokio::test]
async fn scanner_bucket_drive_results_are_bounded() {
let metrics = Metrics::new();
for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
}
metrics.record_scanner_bucket_drive_result("overflow", "/data1", "success");
let report = metrics.scanner_runtime_details_report();
assert_eq!(report.bucket_drive_results.len(), MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS);
assert!(
report
.bucket_drive_results
.iter()
.any(|snapshot| snapshot.bucket == "overflow")
);
assert!(
report
.bucket_drive_results
.iter()
.all(|snapshot| snapshot.bucket != "bucket-0")
);
}
#[tokio::test]
async fn scanner_bucket_drive_result_eviction_keeps_recent_keys() {
let metrics = Metrics::new();
for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
}
metrics.record_scanner_bucket_drive_result("bucket-0", "/data1", "success");
metrics.record_scanner_bucket_drive_result("overflow", "/data1", "success");
let report = metrics.scanner_runtime_details_report();
assert_eq!(report.bucket_drive_results.len(), MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS);
assert!(
report
.bucket_drive_results
.iter()
.any(|snapshot| snapshot.bucket == "bucket-0" && snapshot.count == 2)
);
assert!(
report
.bucket_drive_results
.iter()
.all(|snapshot| snapshot.bucket != "bucket-1")
);
}
#[tokio::test]
async fn scanner_bucket_drive_result_eviction_survives_full_refresh() {
let metrics = Metrics::new();
for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
}
for index in 0..MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS {
metrics.record_scanner_bucket_drive_result(&format!("bucket-{index}"), "/data1", "success");
}
metrics.record_scanner_bucket_drive_result("overflow", "/data1", "success");
let report = metrics.scanner_runtime_details_report();
assert_eq!(report.bucket_drive_results.len(), MAX_SCANNER_BUCKET_DRIVE_RESULT_KEYS);
assert!(
report
.bucket_drive_results
.iter()
.any(|snapshot| snapshot.bucket == "overflow")
);
assert!(
report
.bucket_drive_results
.iter()
.all(|snapshot| snapshot.bucket != "bucket-0")
);
}
#[tokio::test]
async fn report_includes_usage_freshness_status() {
let metrics = Metrics::new();
@@ -4631,7 +4234,7 @@ mod tests {
let active = metrics.report().await;
assert!(active.current_cycle_active);
assert_eq!(active.current_cycle, 12);
assert_eq!(active.current_started, chrono_to_jiff_timestamp(cycle_started));
assert_eq!(active.current_started, cycle_started);
let idle_cycle = CurrentCycle {
current: 0,
@@ -4662,10 +4265,9 @@ mod tests {
};
let cycle_ten_start = metrics.start_scan_cycle_work_with_cycle(cycle_ten.clone()).await;
metrics.operations[Metric::ScanObject as usize].store(1, Ordering::Relaxed);
metrics.record_scanner_bucket_drive_result("cycle-ten", "/data1", "partial");
let paths = metrics.current_paths.write().await;
let mut report = Box::pin(metrics.report_with_runtime_details());
let mut report = Box::pin(metrics.report());
let waker = std::task::Waker::noop();
let mut context = std::task::Context::from_waker(waker);
assert!(report.as_mut().poll(&mut context).is_pending());
@@ -4682,22 +4284,12 @@ mod tests {
})
.await;
metrics.operations[Metric::ScanObject as usize].store(101, Ordering::Relaxed);
metrics.record_scanner_bucket_drive_result("cycle-eleven", "/data1", "partial");
drop(paths);
let (snapshot, runtime_details) = report.await;
let snapshot = report.await;
assert_eq!(snapshot.current_cycle, 10);
assert_eq!(snapshot.current_cycle_objects_scanned, 1);
assert_eq!(
runtime_details.current_cycle_bucket_drive_results,
vec![ScannerBucketDriveResultSnapshot {
bucket: "cycle-ten".to_string(),
drive: "/data1".to_string(),
result: "partial".to_string(),
count: 1,
}]
);
metrics
.finish_scan_cycle_work_with_cycle(cycle_eleven_start, CurrentCycle::default())
-13
View File
@@ -230,19 +230,6 @@ pub const ENV_RUSTFS_KMS_ENABLE: &str = "RUSTFS_KMS_ENABLE";
/// Default value: false
pub const DEFAULT_KMS_ENABLE: bool = false;
/// Environment variable enabling per-key KMS authorization on the SSE-KMS data path.
///
/// When enabled, an SSE-KMS write additionally requires `kms:GenerateDataKey` and an
/// SSE-KMS read additionally requires `kms:Decrypt` on the resolved key, evaluated as
/// the requesting identity. SSE-S3 and SSE-C are unaffected.
pub const ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY: &str = "RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY";
/// Default per-key KMS authorization mode for the SSE-KMS data path.
///
/// Off for now so deployments whose identity policies only grant s3 actions keep
/// working; the default flips to on in a later release.
pub const DEFAULT_KMS_ENFORCE_SSE_KEY_POLICY: bool = false;
/// Environment variable for server KMS backend.
pub const ENV_RUSTFS_KMS_BACKEND: &str = "RUSTFS_KMS_BACKEND";
+4 -3
View File
@@ -177,9 +177,10 @@ const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
///
/// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
/// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
/// state holds roughly `authenticated RPC RPS x 601s` entries. This default is the minimum floor:
/// explicit operator values and resource-aware auto sizing both clamp upward to at least this
/// value. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
/// state holds roughly `authenticated RPC RPS x 601s` entries. The default sustains about 1,700
/// authenticated RPCs per second (about 120 MiB worst case, allocated only under sustained load);
/// operators must size it for the node's aggregate peak RPC rate before enabling strict replay
/// scope. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
/// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
/// the shared secret) — and increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow
+29 -446
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use serde::{Deserialize, Serialize, ser::SerializeMap as _};
use serde::{Deserialize, Serialize};
use std::{
collections::{HashMap, HashSet},
hash::{DefaultHasher, Hash, Hasher},
@@ -37,10 +37,6 @@ pub const USAGE_LAST_UPDATE_FUTURE_TOLERANCE: Duration = Duration::from_secs(5 *
/// Keeping the existing object name preserves rolling-upgrade and rollback
/// compatibility without allowing an ambiguous snapshot to become authoritative.
pub const DATA_USAGE_OBJECT_NAME: &str = ".usage.v2.json";
/// Latest structurally complete scanner observation. Unlike
/// [`DATA_USAGE_OBJECT_NAME`], this object is never authoritative for quota
/// admission because namespace activity may have raced the scan.
pub const DATA_USAGE_OBSERVED_OBJECT_NAME: &str = ".usage.observed.json";
/// Usage snapshot written by scanner implementations predating distributed
/// leadership fencing. It is read only when neither authoritative snapshot
@@ -55,36 +51,24 @@ pub fn usage_last_update_is_untrusted_future(existing_last_update: SystemTime, n
existing_last_update > now + USAGE_LAST_UPDATE_FUTURE_TOLERANCE
}
#[derive(Clone, Copy, Default, Debug, Serialize, Deserialize, PartialEq, Eq)]
#[derive(Clone, Copy, Default, Debug, Serialize, Deserialize, PartialEq)]
pub struct TierStats {
pub total_size: u64,
pub num_versions: u64,
pub num_objects: u64,
pub num_versions: i32,
pub num_objects: i32,
}
impl TierStats {
pub fn add(&self, u: &TierStats) -> TierStats {
TierStats {
total_size: self.total_size.saturating_add(u.total_size),
num_versions: self.num_versions.saturating_add(u.num_versions),
num_objects: self.num_objects.saturating_add(u.num_objects),
total_size: self.total_size + u.total_size,
num_versions: self.num_versions + u.num_versions,
num_objects: self.num_objects + u.num_objects,
}
}
/// True when [`TierStats::add`] would report the exact sum instead of saturating.
pub fn fits_add(&self, u: &TierStats) -> bool {
self.total_size.checked_add(u.total_size).is_some()
&& self.num_versions.checked_add(u.num_versions).is_some()
&& self.num_objects.checked_add(u.num_objects).is_some()
}
/// True when this tier contributed nothing, i.e. merging it is a no-op.
pub fn is_empty(&self) -> bool {
self.total_size == 0 && self.num_versions == 0 && self.num_objects == 0
}
}
#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq)]
pub struct AllTierStats {
pub tiers: HashMap<String, TierStats>,
}
@@ -94,35 +78,31 @@ impl AllTierStats {
Self { tiers: HashMap::new() }
}
pub fn is_empty(&self) -> bool {
self.tiers.is_empty()
}
/// Folds a scan summary's per-tier map in.
///
/// Scanners seed the map with a zeroed entry for every configured tier, so
/// empty contributions are skipped to keep the persisted cache from growing
/// one key per tier on every folder that never held tiered data.
pub fn add_sizes(&mut self, tiers: &HashMap<String, TierStats>) {
pub fn add_sizes(&mut self, tiers: HashMap<String, TierStats>) {
for (tier, st) in tiers {
if st.is_empty() {
continue;
}
let entry = self.tiers.entry(tier.clone()).or_default();
*entry = entry.add(st);
self.tiers
.insert(tier.clone(), self.tiers.get(&tier).copied().unwrap_or_default().add(&st));
}
}
pub fn merge(&mut self, other: &AllTierStats) {
self.add_sizes(&other.tiers);
pub fn merge(&mut self, other: AllTierStats) {
for (tier, st) in other.tiers {
self.tiers
.insert(tier.clone(), self.tiers.get(&tier).copied().unwrap_or_default().add(&st));
}
}
/// True when [`AllTierStats::merge`] would report exact sums for every tier.
pub fn fits_merge(&self, other: &AllTierStats) -> bool {
other
.tiers
.iter()
.all(|(tier, right)| self.tiers.get(tier).is_none_or(|left| left.fits_add(right)))
pub fn populate_stats(&self, stats: &mut HashMap<String, TierStats>) {
for (tier, st) in &self.tiers {
stats.insert(
tier.clone(),
TierStats {
total_size: st.total_size,
num_versions: st.num_versions,
num_objects: st.num_objects,
},
);
}
}
}
@@ -203,14 +183,6 @@ pub struct DataUsageInfo {
pub objects_total_size: u64,
/// Replication info across all buckets
pub replication_info: HashMap<String, BucketTargetUsageInfo>,
/// Usage per storage class and remote tier across all buckets.
///
/// Absent on snapshots written before per-tier accounting was published,
/// and on clusters with no remote tier configured: the scanner classifies
/// objects by tier (including `STANDARD`/`REDUCED_REDUNDANCY`) only once a
/// tier exists, so an absent value means "not accounted", never "zero".
#[serde(default, skip_serializing_if = "Option::is_none")]
pub tier_stats: Option<AllTierStats>,
/// Total number of buckets in this cluster
pub buckets_count: u64,
@@ -222,20 +194,6 @@ pub struct DataUsageInfo {
/// explicit entry for every bucket, including confirmed-empty buckets.
#[serde(default)]
pub usage_snapshot_complete: bool,
/// Whether no namespace activity or dirty-usage generation changed while
/// the coordinated snapshot was being produced.
///
/// `false` still describes a structurally complete, useful point-in-time
/// usage view, but follow-up scanner work remains pending. `None` is kept
/// for snapshots written before this status became observable.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub usage_snapshot_converged: Option<bool>,
/// Identity of the authoritative snapshot from which a nonconverged
/// observation started. Admin readers require an exact match before using
/// the observation, so bucket namespace mutations fence old observations
/// without relying on synchronized clocks.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub usage_snapshot_authoritative_baseline: Option<DataUsageSnapshotIdentity>,
/// Deprecated kept here for backward compatibility reasons
pub bucket_sizes: HashMap<String, u64>,
/// Per-disk snapshot information when available
@@ -243,59 +201,6 @@ pub struct DataUsageInfo {
pub disk_usage_status: Vec<DiskUsageStatus>,
}
/// Stable identity fields changed by both coordinated scanner publication and
/// backward-compatible bucket namespace cleanup.
#[derive(Debug, Default, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub struct DataUsageSnapshotIdentity {
pub last_update: Option<SystemTime>,
pub scanner_cycle: Option<u64>,
pub scanner_epoch: Option<u64>,
}
impl DataUsageInfo {
pub fn snapshot_identity(&self) -> DataUsageSnapshotIdentity {
DataUsageSnapshotIdentity {
last_update: self.last_update,
scanner_cycle: self.scanner_cycle,
scanner_epoch: self.scanner_epoch,
}
}
}
/// Return whether `candidate` was produced after `baseline`.
///
/// New coordinated snapshots are ordered by leadership epoch and scanner
/// cycle. The timestamp fallback preserves ordering for legacy snapshots that
/// predate those fields.
pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUsageInfo) -> bool {
match (
candidate.scanner_epoch.zip(candidate.scanner_cycle),
baseline.scanner_epoch.zip(baseline.scanner_cycle),
) {
(Some(candidate), Some(baseline)) => candidate > baseline,
(Some(_), None) => true,
(None, Some(_)) => false,
(None, None) => match (candidate.last_update, baseline.last_update) {
(Some(candidate), Some(baseline)) => candidate > baseline,
(Some(_), None) => true,
(None, Some(_) | None) => false,
},
}
}
/// Return whether a nonconverged observation may safely supersede the admin
/// view of `authoritative`.
///
/// The exact baseline identity is independent of clock ordering. Older binaries
/// already advance the authoritative timestamp when deleting a bucket, so a
/// rollback delete/recreate fences the previous bucket incarnation too.
pub fn observed_data_usage_is_newer(observed: &DataUsageInfo, authoritative: &DataUsageInfo) -> bool {
observed.usage_snapshot_converged == Some(false)
&& observed.is_complete_bucket_usage_snapshot()
&& observed.usage_snapshot_authoritative_baseline.as_ref() == Some(&authoritative.snapshot_identity())
&& data_usage_snapshot_is_newer(observed, authoritative)
}
/// Metadata describing the status of a disk-level data usage snapshot.
#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct DiskUsageStatus {
@@ -657,7 +562,7 @@ impl ReplicationAllStats {
}
/// Data usage cache entry
#[derive(Clone, Debug, Default, Deserialize)]
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
pub struct DataUsageEntry {
pub children: DataUsageHashMap,
// These fields do not include any children.
@@ -672,34 +577,6 @@ pub struct DataUsageEntry {
/// Number of objects that failed to scan (e.g., IO errors)
#[serde(default)]
pub failed_objects: usize,
/// Per-tier usage contributed by this entry, present only once a scan
/// observed tier-classified objects.
#[serde(default)]
pub all_tier_stats: Option<AllTierStats>,
}
impl Serialize for DataUsageEntry {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
// Keep entries map-encoded so older readers can ignore fields appended
// by newer scanner versions during rolling upgrades. The derived
// (array) encoding made any appended field a decode error for them.
let mut state = serializer.serialize_map(Some(11))?;
state.serialize_entry("children", &self.children)?;
state.serialize_entry("size", &self.size)?;
state.serialize_entry("objects", &self.objects)?;
state.serialize_entry("versions", &self.versions)?;
state.serialize_entry("delete_markers", &self.delete_markers)?;
state.serialize_entry("obj_sizes", &self.obj_sizes)?;
state.serialize_entry("obj_versions", &self.obj_versions)?;
state.serialize_entry("replication_stats", &self.replication_stats)?;
state.serialize_entry("compacted", &self.compacted)?;
state.serialize_entry("failed_objects", &self.failed_objects)?;
state.serialize_entry("all_tier_stats", &self.all_tier_stats)?;
state.end()
}
}
impl DataUsageEntry {
@@ -758,22 +635,10 @@ impl DataUsageEntry {
}
}
if let Some(o_tiers) = other.all_tier_stats.as_ref().filter(|tiers| !tiers.is_empty()) {
self.all_tier_stats.get_or_insert_with(AllTierStats::new).merge(o_tiers);
}
self.obj_sizes.merge_from(&other.obj_sizes);
self.obj_versions.merge_from(&other.obj_versions);
}
/// Folds a scan summary's per-tier map into this entry.
pub fn add_tier_sizes(&mut self, tiers: &HashMap<String, TierStats>) {
if tiers.values().all(TierStats::is_empty) {
return;
}
self.all_tier_stats.get_or_insert_with(AllTierStats::new).add_sizes(tiers);
}
pub fn checked_merge(&mut self, other: &DataUsageEntry) -> bool {
let scalar_counts_fit = self.objects.checked_add(other.objects).is_some()
&& self.versions.checked_add(other.versions).is_some()
@@ -833,12 +698,7 @@ impl DataUsageEntry {
}
};
let tier_stats_fit = match (&self.all_tier_stats, &other.all_tier_stats) {
(_, None) | (None, Some(_)) => true,
(Some(left), Some(right)) => left.fits_merge(right),
};
if !scalar_counts_fit || !histograms_fit || !replication_fits || !tier_stats_fit {
if !scalar_counts_fit || !histograms_fit || !replication_fits {
return false;
}
self.merge(other);
@@ -1178,7 +1038,6 @@ impl DataUsageCache {
versions_total_count: flat.versions as u64,
delete_markers_total_count: flat.delete_markers as u64,
objects_total_size: flat.size as u64,
tier_stats: flat.all_tier_stats.filter(|tiers| !tiers.is_empty()),
buckets_count: u64::try_from(buckets.len()).unwrap_or(u64::MAX),
buckets_usage,
usage_snapshot_complete: self.info.snapshot_complete,
@@ -1666,172 +1525,6 @@ mod tests {
buckets_count: u64,
}
fn tier_entry(tier: &str, stats: TierStats) -> DataUsageEntry {
let mut entry = DataUsageEntry::default();
entry.add_tier_sizes(&HashMap::from([(tier.to_string(), stats)]));
entry
}
#[test]
fn tier_stats_survive_entry_merge() {
let mut left = tier_entry(
"WARM",
TierStats {
total_size: 10,
num_versions: 2,
num_objects: 1,
},
);
let mut right = tier_entry(
"WARM",
TierStats {
total_size: 5,
num_versions: 1,
num_objects: 1,
},
);
right.add_tier_sizes(&HashMap::from([(
"COLD".to_string(),
TierStats {
total_size: 7,
num_versions: 1,
num_objects: 0,
},
)]));
assert!(left.checked_merge(&right), "merging exact tier totals must be accepted");
let tiers = &left.all_tier_stats.expect("merged entry keeps tier stats").tiers;
assert_eq!(
tiers.get("WARM"),
Some(&TierStats {
total_size: 15,
num_versions: 3,
num_objects: 2,
})
);
assert_eq!(
tiers.get("COLD"),
Some(&TierStats {
total_size: 7,
num_versions: 1,
num_objects: 0,
})
);
}
#[test]
fn tier_stats_merge_into_an_untiered_entry() {
let mut left = DataUsageEntry::default();
let right = tier_entry(
"WARM",
TierStats {
total_size: 10,
num_versions: 1,
num_objects: 1,
},
);
assert!(left.checked_merge(&right));
assert_eq!(
left.all_tier_stats.expect("tier stats adopted from the merged entry").tiers["WARM"],
TierStats {
total_size: 10,
num_versions: 1,
num_objects: 1,
}
);
}
#[test]
fn checked_merge_rejects_overflowing_tier_totals() {
let mut left = tier_entry(
"WARM",
TierStats {
total_size: u64::MAX,
num_versions: 1,
num_objects: 1,
},
);
let right = tier_entry(
"WARM",
TierStats {
total_size: 1,
num_versions: 1,
num_objects: 1,
},
);
assert!(!left.checked_merge(&right), "saturating tier totals must not be published");
assert_eq!(left.all_tier_stats.expect("left is untouched").tiers["WARM"].total_size, u64::MAX);
}
/// Entry shape released before per-tier accounting, using the derived
/// (array) encoding those writers produced.
#[derive(Serialize, Deserialize)]
struct LegacyEntry {
children: DataUsageHashMap,
size: usize,
objects: usize,
versions: usize,
delete_markers: usize,
obj_sizes: SizeHistogram,
obj_versions: VersionsHistogram,
replication_stats: Option<ReplicationAllStats>,
compacted: bool,
#[serde(default)]
failed_objects: usize,
}
#[test]
fn entries_are_map_encoded_so_appended_fields_stay_readable() {
// A derived (array) encoding turns every appended field into a decode
// error for readers built before it existed, which would cost a mixed
// -version cluster its whole scan cache. Entries must stay map-encoded.
let current = tier_entry(
"WARM",
TierStats {
total_size: 3,
num_versions: 1,
num_objects: 1,
},
);
let mut encoded = Vec::new();
current
.serialize(&mut rmp_serde::Serializer::new(&mut encoded))
.expect("encode current entry");
let legacy: LegacyEntry = rmp_serde::from_slice(&encoded).expect("legacy reader should ignore the appended field");
assert_eq!(legacy.objects, 0);
}
#[test]
fn legacy_array_encoded_entries_still_load() {
let legacy = LegacyEntry {
children: DataUsageHashMap::default(),
size: 12,
objects: 3,
versions: 4,
delete_markers: 1,
obj_sizes: SizeHistogram::default(),
obj_versions: VersionsHistogram::default(),
replication_stats: None,
compacted: false,
failed_objects: 2,
};
let mut encoded = Vec::new();
legacy
.serialize(&mut rmp_serde::Serializer::new(&mut encoded))
.expect("encode legacy entry");
let decoded: DataUsageEntry = rmp_serde::from_slice(&encoded).expect("current reader should default the missing field");
assert_eq!(decoded.size, 12);
assert_eq!(decoded.failed_objects, 2);
assert!(decoded.all_tier_stats.is_none());
}
#[test]
fn hash_path_uses_portable_slash_semantics() {
for (input, expected) in [
@@ -1854,8 +1547,6 @@ mod tests {
let current = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(DataUsageSnapshotIdentity::default()),
..Default::default()
};
let encoded = rmp_serde::to_vec_named(&current).expect("encode current data usage snapshot");
@@ -1863,76 +1554,6 @@ mod tests {
assert_eq!(legacy.buckets_count, 0);
assert!(current.is_complete_bucket_usage_snapshot());
assert_eq!(current.usage_snapshot_converged, Some(false));
}
#[test]
fn convergence_marker_defaults_to_unknown_for_older_snapshots() {
let encoded = rmp_serde::to_vec_named(&DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH),
usage_snapshot_complete: true,
..Default::default()
})
.expect("encode pre-convergence data usage snapshot");
let decoded: DataUsageInfo = rmp_serde::from_slice(&encoded).expect("decode older data usage snapshot");
assert!(decoded.is_complete_bucket_usage_snapshot());
assert_eq!(decoded.usage_snapshot_converged, None);
}
#[test]
fn observation_selection_is_clock_independent_and_baseline_fenced() {
let mut authoritative = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(600)),
scanner_epoch: Some(7),
scanner_cycle: Some(10),
usage_snapshot_complete: true,
..Default::default()
};
let observed = DataUsageInfo {
// A newer leader may have a slower wall clock.
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(300)),
scanner_epoch: Some(8),
scanner_cycle: Some(1),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(authoritative.snapshot_identity()),
..Default::default()
};
assert!(observed_data_usage_is_newer(&observed, &authoritative));
authoritative.last_update = Some(SystemTime::UNIX_EPOCH + Duration::from_secs(601));
assert!(
!observed_data_usage_is_newer(&observed, &authoritative),
"an old-binary namespace mutation must fence the prior bucket incarnation regardless of clock skew"
);
}
#[test]
fn observation_selection_requires_nonconverged_complete_newer_data() {
let authoritative = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH),
scanner_epoch: Some(2),
scanner_cycle: Some(10),
usage_snapshot_complete: true,
..Default::default()
};
let baseline = Some(authoritative.snapshot_identity());
let candidate = |epoch, cycle, converged, complete| DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
scanner_epoch: Some(epoch),
scanner_cycle: Some(cycle),
usage_snapshot_complete: complete,
usage_snapshot_converged: converged,
usage_snapshot_authoritative_baseline: baseline,
..Default::default()
};
assert!(observed_data_usage_is_newer(&candidate(2, 11, Some(false), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 9, Some(false), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(true), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(false), false), &authoritative));
}
#[test]
@@ -2280,44 +1901,6 @@ mod tests {
assert_eq!(info.buckets_count, 2);
assert!(info.buckets_usage.is_empty());
assert_eq!(info.objects_total_count, 3);
assert!(info.tier_stats.is_none());
}
#[test]
fn test_dui_reports_tier_usage_from_the_flattened_tree() {
let root_hash = hash_path("root");
let bucket_hash = hash_path("bucket-a");
let mut cache = DataUsageCache {
info: DataUsageCacheInfo {
name: "root".to_string(),
..Default::default()
},
..Default::default()
};
cache.replace_hashed(&root_hash, &None, &DataUsageEntry::default());
cache.replace_hashed(
&bucket_hash,
&Some(root_hash),
&tier_entry(
"WARM",
TierStats {
total_size: 40,
num_versions: 2,
num_objects: 2,
},
),
);
let info = cache.dui("root", &["bucket-a".to_string()]);
assert_eq!(
info.tier_stats.expect("child tier usage should roll up to the root").tiers["WARM"],
TierStats {
total_size: 40,
num_versions: 2,
num_objects: 2,
}
);
}
#[test]
+60 -1
View File
@@ -26,16 +26,75 @@
//! Later batches tracked on backlog#1154: config get/set, info, pools status,
//! group lifecycle, import/export IAM.
use crate::common::{RustFSTestEnvironment, admin_ok, admin_request, init_logging};
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::{Client, Config};
use http::header::{CONTENT_TYPE, HOST};
use reqwest::StatusCode;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::error::Error;
use tokio::time::{Duration, sleep};
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
type BoxError = Box<dyn Error + Send + Sync>;
/// Signs and sends an admin HTTP request with the given credential, returning
/// status and body. Native `/rustfs/admin/v3` requests and responses are plain
/// JSON (the MinIO-compat encryption applies only to `/minio/admin/v3` paths).
async fn admin_request(
base_url: &str,
method: http::Method,
path_and_query: &str,
body: Option<String>,
access_key: &str,
secret_key: &str,
) -> Result<(StatusCode, String), BoxError> {
let url = format!("{base_url}{path_and_query}");
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
let mut builder = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
if body.is_some() {
builder = builder.header(CONTENT_TYPE, "application/json");
}
let content_len = body.as_ref().map(|b| b.len() as i64).unwrap_or_default();
let signed = sign_v4(builder.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
let mut request = local_http_client().request(reqwest_method, &url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
if let Some(body) = body {
request = request.body(body);
}
let response = request.send().await?;
let status = response.status();
let text = response.text().await.unwrap_or_default();
Ok((status, text))
}
/// Root-credential admin request that must succeed; returns the response body.
async fn admin_ok(
env: &RustFSTestEnvironment,
method: http::Method,
path_and_query: &str,
body: Option<String>,
) -> Result<String, BoxError> {
let (status, text) = admin_request(&env.url, method.clone(), path_and_query, body, &env.access_key, &env.secret_key).await?;
if !status.is_success() {
return Err(format!("{method} {path_and_query} failed: {status} {text}").into());
}
Ok(text)
}
fn build_s3_client(url: &str, access_key: &str, secret_key: &str) -> Client {
let config = Config::builder()
+105 -83
View File
@@ -16,17 +16,91 @@
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging, signed_s3_request};
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::types::{
AccelerateConfiguration, BucketAccelerateStatus, BucketLoggingStatus, IndexDocument, LoggingEnabled, Payer,
RequestPaymentConfiguration, WebsiteConfiguration,
};
use http::Method;
use http::header::CONTENT_TYPE;
use serial_test::serial;
use std::path::PathBuf;
use std::process::Command;
use tracing::info;
fn awscurl_binary_path() -> PathBuf {
std::env::var_os("AWSCURL_PATH")
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from("awscurl"))
}
fn awscurl_available() -> bool {
Command::new(awscurl_binary_path()).arg("--version").output().is_ok()
}
fn execute_s3_awscurl(
method: &str,
url: &str,
access_key: &str,
secret_key: &str,
) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
let output = Command::new(awscurl_binary_path())
.args([
"--service",
"s3",
"--region",
"us-east-1",
"--access_key",
access_key,
"--secret_key",
secret_key,
"-i",
"-X",
method,
url,
])
.output()?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
let stdout = String::from_utf8_lossy(&output.stdout);
return Err(format!("awscurl failed: stderr='{stderr}', stdout='{stdout}'").into());
}
Ok(String::from_utf8_lossy(&output.stdout).to_string())
}
fn parse_status(raw: &str) -> Option<u16> {
raw.lines()
.filter_map(|line| {
if line.starts_with("HTTP/") {
line.split_whitespace().nth(1)?.parse::<u16>().ok()
} else {
None
}
})
.next_back()
}
fn parse_body(raw: &str) -> String {
if let Some(pos) = raw.rfind("\r\n\r\n") {
return raw[pos + 4..].to_string();
}
if let Some(pos) = raw.rfind("\n\n") {
return raw[pos + 2..].to_string();
}
String::new()
}
fn parse_headers(raw: &str) -> String {
let start = raw.rfind("HTTP/").unwrap_or(0);
let tail = &raw[start..];
if let Some(pos) = tail.find("\r\n\r\n") {
return tail[..pos].to_string();
}
if let Some(pos) = tail.find("\n\n") {
return tail[..pos].to_string();
}
tail.to_string()
}
#[tokio::test]
#[serial]
async fn test_dummy_bucket_compatibility_endpoints() {
@@ -396,6 +470,10 @@ mod tests {
async fn test_dummy_bucket_endpoints_http_contracts() {
init_logging();
info!("Starting test: dummy-compat bucket API HTTP contracts");
if !awscurl_available() {
info!("Skipping test_dummy_bucket_endpoints_http_contracts: awscurl binary not found");
return;
}
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
@@ -410,112 +488,56 @@ mod tests {
.await
.expect("Failed to create bucket");
let logging_response = signed_s3_request(
Method::GET,
&format!("{}/{bucket}?logging=", env.url),
None,
None,
&env.access_key,
&env.secret_key,
)
.await
.expect("GetBucketLogging HTTP request failed");
assert_eq!(logging_response.status(), 200, "GetBucketLogging should return 200");
let logging_body = logging_response
.text()
.await
.expect("Failed to read GetBucketLogging response body");
let logging_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?logging=", env.url), &env.access_key, &env.secret_key)
.expect("GetBucketLogging HTTP request failed");
assert_eq!(parse_status(&logging_raw), Some(200), "GetBucketLogging should return 200");
let logging_body = parse_body(&logging_raw);
assert!(
logging_body.contains("<BucketLoggingStatus"),
"GetBucketLogging response should contain BucketLoggingStatus XML, got: {logging_body}"
);
let accel_response = signed_s3_request(
Method::GET,
&format!("{}/{bucket}?accelerate=", env.url),
None,
None,
&env.access_key,
&env.secret_key,
)
.await
.expect("GetBucketAccelerateConfiguration HTTP request failed");
assert_eq!(accel_response.status(), 200, "GetBucketAccelerateConfiguration should return 200");
let accel_body = accel_response
.text()
.await
.expect("Failed to read GetBucketAccelerateConfiguration response body");
let accel_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?accelerate=", env.url), &env.access_key, &env.secret_key)
.expect("GetBucketAccelerateConfiguration HTTP request failed");
assert_eq!(parse_status(&accel_raw), Some(200), "GetBucketAccelerateConfiguration should return 200");
let accel_body = parse_body(&accel_raw);
assert!(
accel_body.contains("<AccelerateConfiguration"),
"GetBucketAccelerateConfiguration response should contain AccelerateConfiguration XML, got: {accel_body}"
);
let payment_response = signed_s3_request(
Method::GET,
&format!("{}/{bucket}?requestPayment=", env.url),
None,
None,
&env.access_key,
&env.secret_key,
)
.await
.expect("GetBucketRequestPayment HTTP request failed");
assert_eq!(payment_response.status(), 200, "GetBucketRequestPayment should return 200");
let payment_body = payment_response
.text()
.await
.expect("Failed to read GetBucketRequestPayment response body");
let payment_raw =
execute_s3_awscurl("GET", &format!("{}/{bucket}?requestPayment=", env.url), &env.access_key, &env.secret_key)
.expect("GetBucketRequestPayment HTTP request failed");
assert_eq!(parse_status(&payment_raw), Some(200), "GetBucketRequestPayment should return 200");
let payment_body = parse_body(&payment_raw);
assert!(
payment_body.contains("<Payer>BucketOwner</Payer>"),
"GetBucketRequestPayment should return BucketOwner payer, got: {payment_body}"
);
let website_response = signed_s3_request(
Method::GET,
&format!("{}/{bucket}?website=", env.url),
None,
None,
&env.access_key,
&env.secret_key,
)
.await
.expect("GetBucketWebsite HTTP request failed");
let website_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?website=", env.url), &env.access_key, &env.secret_key)
.expect("GetBucketWebsite HTTP request failed");
assert_eq!(
website_response.status(),
404,
parse_status(&website_raw),
Some(404),
"GetBucketWebsite should return 404 when website config is absent"
);
let website_content_type = website_response
.headers()
.get(CONTENT_TYPE)
.expect("GetBucketWebsite response should include Content-Type")
.to_str()
.expect("GetBucketWebsite Content-Type should be valid ASCII")
.to_ascii_lowercase();
let website_content_type = parse_headers(&website_raw).to_ascii_lowercase();
assert!(
website_content_type.contains("xml"),
website_content_type.contains("content-type:") && website_content_type.contains("xml"),
"GetBucketWebsite error response should be XML, got content-type: {website_content_type}"
);
let website_body = website_response
.text()
.await
.expect("Failed to read GetBucketWebsite response body");
let website_body = parse_body(&website_raw);
assert!(
website_body.contains("<Code>NoSuchWebsiteConfiguration</Code>"),
"GetBucketWebsite should return NoSuchWebsiteConfiguration code, got: {website_body}"
);
let delete_response = signed_s3_request(
Method::DELETE,
&format!("{}/{bucket}?website=", env.url),
None,
None,
&env.access_key,
&env.secret_key,
)
.await
.expect("DeleteBucketWebsite HTTP request failed");
assert_eq!(delete_response.status(), 204, "DeleteBucketWebsite should return 204");
let delete_raw =
execute_s3_awscurl("DELETE", &format!("{}/{bucket}?website=", env.url), &env.access_key, &env.secret_key)
.expect("DeleteBucketWebsite HTTP request failed");
assert_eq!(parse_status(&delete_raw), Some(204), "DeleteBucketWebsite should return 204");
env.stop_server();
}
@@ -1,295 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression tests for bucket statistics and data usage accuracy.
//!
//! Covers the recurring pattern where bucket statistics (object count, size)
//! show stale/incorrect values, remain at 0, or oscillate between complete,
//! partial, and zero. This has regressed 10+ times.
//!
//! ## Regression Issues
//!
//! - rustfs#5615: bucket statistics remain unchanged after data expiration
//! - rustfs#5008: Admin usage reports only one pool
//! - rustfs#5116: Admin usage reports stale 0/0 for non-empty bucket after upgrade
//! - rustfs#5055: console object count and size still loading
//! - rustfs#5010: Storage usage info changed abnormally
//! - rustfs#3662: Incorrect bucket, object count and size
//! - rustfs#3898: DataUsageInfo undercounts versioned bucket versions
//! - rustfs#1012: Object count in the console doesn't change
#[cfg(test)]
mod tests {
use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, RustFSTestEnvironment, awscurl_get, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use rustfs_data_usage::DataUsageInfo;
use serial_test::serial;
use std::error::Error;
use tokio::time::{Duration, sleep};
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
async fn get_data_usage(env: &RustFSTestEnvironment) -> Result<DataUsageInfo, Box<dyn Error + Send + Sync>> {
let url = format!("{}/rustfs/admin/v3/datausageinfo", env.url);
let resp = awscurl_get(&url, &env.access_key, &env.secret_key).await?;
Ok(serde_json::from_str(&resp)?)
}
/// RT-09: Verify bucket object count updates after PUT.
///
/// Regression pattern: bucket stats remain at 0 after objects are uploaded
/// (rustfs#5055, rustfs#1012).
///
/// Steps:
/// 1. Create a bucket
/// 2. Upload 10 objects
/// 3. Query admin data usage API
/// 4. Verify object count > 0
#[tokio::test]
#[serial]
async fn test_bucket_object_count_updates_after_put() -> TestResult {
init_logging();
info!("RT-09: bucket object count updates after PUT");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV)
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt09-stats-put";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Upload 10 objects
for i in 0..10 {
client
.put_object()
.bucket(bucket)
.key(format!("stat-obj-{i:04}.txt"))
.body(ByteStream::from_static(b"statistical data"))
.send()
.await
.expect("put object");
}
// Wait for scanner to process (up to 90 seconds)
let mut found_nonzero = false;
let mut last_query_error = None;
for attempt in 0..18 {
sleep(Duration::from_secs(5)).await;
let usage = match get_data_usage(&env).await {
Ok(usage) => {
last_query_error = None;
usage
}
Err(err) => {
last_query_error = Some(err.to_string());
continue;
}
};
if let Some(bucket_usage) = usage.buckets_usage.get(bucket) {
info!(" attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
if bucket_usage.objects_count >= 10 {
found_nonzero = true;
break;
}
}
}
assert!(
found_nonzero,
"RT-09 FAIL: bucket object count did not update after PUT 10 objects (regression: stats stuck at 0); last query error: {}",
last_query_error.as_deref().unwrap_or("none")
);
info!("RT-09 PASS: bucket object count updates after PUT");
Ok(())
}
/// RT-09b: Verify bucket stats update after DELETE.
///
/// Regression pattern: stats remain unchanged after objects are deleted
/// (rustfs#5615).
#[tokio::test]
#[serial]
async fn test_bucket_object_count_updates_after_delete() -> TestResult {
init_logging();
info!("RT-09b: bucket object count updates after DELETE");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV)
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt09b-stats-delete";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Upload 5 objects
for i in 0..5 {
client
.put_object()
.bucket(bucket)
.key(format!("del-stat-{i}.txt"))
.body(ByteStream::from_static(b"data"))
.send()
.await
.expect("put object");
}
let mut found_nonzero = false;
for attempt in 0..18 {
sleep(Duration::from_secs(5)).await;
if let Ok(usage) = get_data_usage(&env).await
&& let Some(bucket_usage) = usage.buckets_usage.get(bucket)
{
info!(" baseline attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
if bucket_usage.objects_count >= 5 {
found_nonzero = true;
break;
}
}
}
assert!(found_nonzero, "RT-09b setup failed: scanner did not observe the 5 uploaded objects");
// Delete all objects
for i in 0..5 {
client
.delete_object()
.bucket(bucket)
.key(format!("del-stat-{i}.txt"))
.send()
.await
.expect("delete object");
}
// Wait for scanner to update stats (up to 90 seconds)
let mut found_zero = false;
let mut last_query_error = None;
for attempt in 0..18 {
sleep(Duration::from_secs(5)).await;
let usage = match get_data_usage(&env).await {
Ok(usage) => {
last_query_error = None;
usage
}
Err(err) => {
last_query_error = Some(err.to_string());
continue;
}
};
if let Some(bucket_usage) = usage.buckets_usage.get(bucket) {
info!(" attempt {attempt}: objectsCount = {}", bucket_usage.objects_count);
if bucket_usage.objects_count == 0 {
found_zero = true;
break;
}
}
}
assert!(
found_zero,
"RT-09b FAIL: bucket object count did not update to 0 after deleting all objects (regression rustfs#5615); last query error: {}",
last_query_error.as_deref().unwrap_or("none")
);
info!("RT-09b PASS: bucket object count updates to 0 after DELETE");
Ok(())
}
/// RT-09c: Verify versioned bucket stats count all versions.
///
/// Regression pattern: DataUsageInfo undercounts versioned bucket versions
/// and delete markers (rustfs#3898).
#[tokio::test]
#[serial]
async fn test_versioned_bucket_stats_count_all_versions() -> TestResult {
init_logging();
info!("RT-09c: versioned bucket stats count all versions");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt09c-versioned-stats";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await
.expect("enable versioning");
// Create 3 versions of the same object
for i in 0..3 {
client
.put_object()
.bucket(bucket)
.key("multi-version.txt")
.body(ByteStream::from(format!("version-{i}").into_bytes()))
.send()
.await
.expect("put version");
}
// Create a delete marker
client
.delete_object()
.bucket(bucket)
.key("multi-version.txt")
.send()
.await
.expect("create delete marker");
// Verify versions via API (immediate, no scanner wait)
let versions = client
.list_object_versions()
.bucket(bucket)
.send()
.await
.expect("list versions");
assert_eq!(
versions.versions().len(),
3,
"RT-09c FAIL: expected 3 versions, found {}",
versions.versions().len()
);
assert_eq!(
versions.delete_markers().len(),
1,
"RT-09c FAIL: expected 1 delete marker, found {}",
versions.delete_markers().len()
);
info!("RT-09c PASS: versioned bucket correctly tracks all versions and delete markers");
Ok(())
}
}
+5 -140
View File
@@ -24,12 +24,7 @@
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::{Client, Config};
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use http::header::{CONTENT_TYPE, HOST};
use reqwest::Client as HttpClient;
use reqwest::StatusCode;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use std::ffi::OsStr;
use std::fs as stdfs;
use std::path::{Path, PathBuf};
@@ -47,34 +42,11 @@ use walkdir::WalkDir;
pub const DEFAULT_ACCESS_KEY: &str = "rustfsadmin";
pub const DEFAULT_SECRET_KEY: &str = "rustfsadmin";
pub const ENV_RUSTFS_BUILD_FEATURES: &str = "RUSTFS_BUILD_FEATURES";
pub(crate) const FAST_DATA_USAGE_SCANNER_ENV: &[(&str, &str)] =
&[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_SCANNER_START_DELAY_SECS", "0")];
pub const TEST_BUCKET: &str = "e2e-test-bucket";
const RUSTFS_FULL_FEATURE: &str = "full";
fn capture_log_path(log_dir: &Path, temp_dir: &str) -> Option<PathBuf> {
let temp_name = Path::new(temp_dir).file_name()?.to_string_lossy();
Some(log_dir.join(format!("{temp_name}.log")))
}
fn configured_capture_log_path(temp_dir: &str) -> Option<String> {
let log_dir = std::env::var_os("RUSTFS_E2E_LOG_DIR")?;
if stdfs::create_dir_all(&log_dir).is_err() {
warn!(?log_dir, "failed to create configured E2E server log directory");
return None;
}
capture_log_path(Path::new(&log_dir), temp_dir).map(|path| path.to_string_lossy().into_owned())
}
pub(crate) fn build_test_s3_config(
endpoint_url: &str,
access_key: &str,
secret_key: &str,
session_token: Option<&str>,
provider_name: &'static str,
) -> Config {
let credentials = Credentials::new(access_key, secret_key, session_token.map(str::to_owned), None, provider_name);
fn build_test_s3_config(endpoint_url: &str, access_key: &str, secret_key: &str, provider_name: &'static str) -> Config {
let credentials = Credentials::new(access_key, secret_key, None, None, provider_name);
let mut config = Config::builder()
.credentials_provider(credentials)
.region(Region::new("us-east-1"))
@@ -89,33 +61,6 @@ pub(crate) fn build_test_s3_config(
config.build()
}
pub(crate) fn build_test_sts_client(
endpoint_url: &str,
access_key: &str,
secret_key: &str,
session_token: Option<&str>,
provider_name: &'static str,
) -> aws_sdk_sts::Client {
let mut config = aws_sdk_sts::Config::builder()
.credentials_provider(aws_sdk_sts::config::Credentials::new(
access_key,
secret_key,
session_token.map(str::to_owned),
None,
provider_name,
))
.region(aws_sdk_sts::config::Region::new("us-east-1"))
.endpoint_url(endpoint_url)
.retry_config(aws_sdk_sts::config::retry::RetryConfig::standard().with_max_attempts(1))
.behavior_version_latest();
if endpoint_url.starts_with("http://") {
config = config.http_client(SmithyHttpClientBuilder::new().build_http());
}
aws_sdk_sts::Client::from_conf(config.build())
}
pub fn workspace_root() -> PathBuf {
let mut path = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
path.pop(); // e2e_test
@@ -130,70 +75,6 @@ pub fn local_http_client() -> HttpClient {
.expect("failed to build local reqwest client")
}
pub(crate) async fn signed_s3_request(
method: http::Method,
url: &str,
body: Option<String>,
content_type: Option<&str>,
access_key: &str,
secret_key: &str,
) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("S3 URL missing authority")?.to_string();
let mut request = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
if let Some(content_type) = content_type {
request = request.header(CONTENT_TYPE, content_type);
}
let content_length = i64::try_from(body.as_ref().map_or(0, String::len)).map_err(|_| "S3 request body is too large")?;
let signed = sign_v4(request.body(Body::empty())?, content_length, access_key, secret_key, "", "us-east-1");
let mut request = local_http_client().request(method, url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
if let Some(body) = body {
request = request.body(body);
}
Ok(request.send().await?)
}
/// Signs and sends an admin HTTP request with the given credentials.
pub(crate) async fn admin_request(
base_url: &str,
method: http::Method,
path_and_query: &str,
body: Option<String>,
access_key: &str,
secret_key: &str,
) -> Result<(StatusCode, String), Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{base_url}{path_and_query}");
let content_type = body.as_ref().map(|_| "application/json");
let response = signed_s3_request(method, &url, body, content_type, access_key, secret_key).await?;
let status = response.status();
let body = response.text().await?;
Ok((status, body))
}
/// Sends a root-credential admin request and returns its successful response body.
pub(crate) async fn admin_ok(
env: &RustFSTestEnvironment,
method: http::Method,
path_and_query: &str,
body: Option<String>,
) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
let (status, response_body) =
admin_request(&env.url, method.clone(), path_and_query, body, &env.access_key, &env.secret_key).await?;
if !status.is_success() {
return Err(format!("{method} {path_and_query} failed: {status} {response_body}").into());
}
Ok(response_body)
}
/// Resolve the RustFS binary relative to the workspace.
pub fn rustfs_binary_path() -> PathBuf {
rustfs_binary_path_with_features(requested_rustfs_build_features().as_deref())
@@ -423,7 +304,6 @@ impl RustFSTestEnvironment {
pub async fn new() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
let temp_dir = format!("/tmp/rustfs_e2e_test_{}", Uuid::new_v4());
fs::create_dir_all(&temp_dir).await?;
let capture_log_path = configured_capture_log_path(&temp_dir);
// Use a unique port for each test environment
let port = Self::find_available_port().await?;
@@ -437,7 +317,7 @@ impl RustFSTestEnvironment {
access_key: DEFAULT_ACCESS_KEY.to_string(),
secret_key: DEFAULT_SECRET_KEY.to_string(),
process: None,
capture_log_path,
capture_log_path: None,
})
}
@@ -445,7 +325,6 @@ impl RustFSTestEnvironment {
pub async fn with_address(address: &str) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
let temp_dir = format!("/tmp/rustfs_e2e_test_{}", Uuid::new_v4());
fs::create_dir_all(&temp_dir).await?;
let capture_log_path = configured_capture_log_path(&temp_dir);
let url = format!("http://{address}");
@@ -456,7 +335,7 @@ impl RustFSTestEnvironment {
access_key: DEFAULT_ACCESS_KEY.to_string(),
secret_key: DEFAULT_SECRET_KEY.to_string(),
process: None,
capture_log_path,
capture_log_path: None,
})
}
@@ -611,12 +490,7 @@ impl RustFSTestEnvironment {
/// Create an AWS S3 client configured for this RustFS instance
pub fn create_s3_client(&self) -> Client {
self.create_s3_client_with_credentials(&self.access_key, &self.secret_key)
}
/// Create an AWS S3 client with explicit credentials for this RustFS instance.
pub fn create_s3_client_with_credentials(&self, access_key: &str, secret_key: &str) -> Client {
Client::from_conf(build_test_s3_config(&self.url, access_key, secret_key, None, "e2e-test"))
Client::from_conf(build_test_s3_config(&self.url, &self.access_key, &self.secret_key, "e2e-test"))
}
/// Create test bucket
@@ -1348,7 +1222,6 @@ impl RustFSTestClusterEnvironment {
&self.nodes[node_idx].url,
&self.access_key,
&self.secret_key,
None,
"cluster-test",
)))
}
@@ -1462,14 +1335,6 @@ mod tests {
assert_eq!(normalize_rustfs_build_features(" , "), None);
}
#[test]
fn capture_log_path_uses_temp_directory_basename() {
assert_eq!(
capture_log_path(Path::new("/tmp/e2e-logs"), "/tmp/rustfs_e2e_test_abc"),
Some(PathBuf::from("/tmp/e2e-logs/rustfs_e2e_test_abc.log"))
);
}
#[test]
fn full_feature_enables_any_required_feature() {
assert!(rustfs_build_feature_enabled(Some("full"), "sftp"));
+12 -28
View File
@@ -18,7 +18,7 @@ use rustfs_data_usage::DataUsageInfo;
use serial_test::serial;
use tokio::time::{Duration, sleep};
use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, RustFSTestEnvironment, TEST_BUCKET, awscurl_get, init_logging};
use crate::common::{RustFSTestEnvironment, TEST_BUCKET, awscurl_get, init_logging};
async fn get_data_usage_info(env: &RustFSTestEnvironment) -> Result<DataUsageInfo, Box<dyn std::error::Error + Send + Sync>> {
let url = format!("{}/rustfs/admin/v3/datausageinfo", env.url);
@@ -35,26 +35,16 @@ where
F: FnMut(&DataUsageInfo) -> bool,
{
let mut last_usage = DataUsageInfo::default();
let mut last_query_error = None;
for _ in 0..45 {
match get_data_usage_info(env).await {
Ok(usage) => {
last_query_error = None;
if usage.buckets_usage.contains_key(bucket) && predicate(&usage) {
return Ok(usage);
}
last_usage = usage;
}
Err(err) => last_query_error = Some(err.to_string()),
let usage = get_data_usage_info(env).await?;
if usage.buckets_usage.contains_key(bucket) && predicate(&usage) {
return Ok(usage);
}
last_usage = usage;
sleep(Duration::from_secs(2)).await;
}
Err(format!(
"bucket usage did not converge for {bucket}; last usage: {last_usage:?}; last query error: {}",
last_query_error.as_deref().unwrap_or("none")
)
.into())
Err(format!("bucket usage did not converge for {bucket}; last usage: {last_usage:?}").into())
}
/// Regression test for data usage accuracy (issue #1012).
@@ -66,7 +56,7 @@ async fn data_usage_reports_all_objects() -> Result<(), Box<dyn std::error::Erro
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV).await?;
env.start_rustfs_server(vec![]).await?;
let client = env.create_s3_client();
@@ -84,14 +74,8 @@ async fn data_usage_reports_all_objects() -> Result<(), Box<dyn std::error::Erro
.await?;
}
let usage = wait_for_bucket_usage(&env, TEST_BUCKET, |usage| {
usage
.buckets_usage
.get(TEST_BUCKET)
.map(|bucket_usage| usage.objects_total_count >= 1000 && bucket_usage.objects_count >= 1000)
.unwrap_or(false)
})
.await?;
// Query admin data usage API
let usage = get_data_usage_info(&env).await?;
// Assert total object count and per-bucket count are not truncated
let bucket_usage = usage
@@ -124,7 +108,7 @@ async fn data_usage_reports_versioned_objects_and_delete_markers() -> Result<(),
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(vec![], FAST_DATA_USAGE_SCANNER_ENV).await?;
env.start_rustfs_server(vec![]).await?;
let client = env.create_s3_client();
let bucket = "data-usage-versioned";
@@ -200,8 +184,8 @@ async fn data_usage_reports_versioned_objects_and_delete_markers() -> Result<(),
assert_eq!(usage.versions_total_count, 3, "total version count should match bucket usage");
assert_eq!(usage.delete_markers_total_count, 1, "total delete marker count should match bucket usage");
env.restart_server_preserving_data(vec![], FAST_DATA_USAGE_SCANNER_ENV)
.await?;
env.stop_server();
env.start_rustfs_server(vec![]).await?;
let restarted_usage = wait_for_bucket_usage(&env, bucket, |usage| {
usage
@@ -1,445 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression tests for object delete operations.
//!
//! Covers the recurring pattern where DELETE succeeds at the API level but the
//! object remains visible in LIST, or deleted objects reappear after restart,
//! or versioned delete operations fail with FileAccessDenied.
//! This has regressed 15+ times across the entire release history.
//!
//! ## Regression Issues
//!
//! - rustfs#5375: delete object in a bucket list api also exist this object
//! - rustfs#5349: The deleted bucket was rebuilt after some time
//! - rustfs#5339: data not delete in Object Lock bucket
//! - rustfs#5029: Node Does Not Remove Files After Reconnect to Cluster
//! - rustfs#4978: DELETE fails with InternalError/FileAccessDenied on beta 10
//! - rustfs#760: Cannot delete a versioned bucket
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
use serial_test::serial;
use std::error::Error;
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
/// RT-05: Verify DELETE → LIST → HEAD consistency.
///
/// Regression pattern: DELETE returns 200 but the object remains in LIST.
/// Covers rustfs#5375.
///
/// Steps:
/// 1. Create a bucket and upload an object
/// 2. Verify the object is in LIST
/// 3. DELETE the object
/// 4. Verify the object is NOT in LIST
/// 5. Verify HEAD returns 404
#[tokio::test]
#[serial]
async fn test_delete_removes_object_from_list() -> TestResult {
init_logging();
info!("RT-05: delete removes object from list");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt05-delete-consistency";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Upload an object
client
.put_object()
.bucket(bucket)
.key("to-delete.txt")
.body(ByteStream::from_static(b"will be deleted"))
.send()
.await
.expect("put object");
// Verify it appears in LIST
let list = client
.list_objects_v2()
.bucket(bucket)
.send()
.await
.expect("list objects before delete");
assert!(
list.contents()
.iter()
.map(|o| o.key().unwrap_or(""))
.any(|key| key == "to-delete.txt"),
"RT-05 FAIL: object not in LIST before delete"
);
// DELETE
client
.delete_object()
.bucket(bucket)
.key("to-delete.txt")
.send()
.await
.expect("delete object");
// Verify NOT in LIST
let list = client
.list_objects_v2()
.bucket(bucket)
.send()
.await
.expect("list objects after delete");
assert!(
!list
.contents()
.iter()
.map(|o| o.key().unwrap_or(""))
.any(|key| key == "to-delete.txt"),
"RT-05 FAIL: deleted object still in LIST (regression rustfs#5375)"
);
// Verify HEAD returns 404
let head = client.head_object().bucket(bucket).key("to-delete.txt").send().await;
assert!(head.is_err(), "RT-05 FAIL: HEAD on deleted object should return error, got success");
info!("RT-05 PASS: delete correctly removes object from LIST and HEAD");
Ok(())
}
/// RT-05c: Verify batch delete (DeleteObjects) consistency.
///
/// Regression pattern: batch delete returns success but some objects
/// remain in LIST.
#[tokio::test]
#[serial]
async fn test_batch_delete_removes_all_objects() -> TestResult {
init_logging();
info!("RT-05c: batch delete removes all objects");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt05c-batch-delete";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Upload multiple objects
let keys: Vec<String> = (0..5).map(|i| format!("batch-{i:04}.txt")).collect();
for key in &keys {
client
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"batch-delete-me"))
.send()
.await
.expect("put object");
}
// Verify all in LIST
let list = client
.list_objects_v2()
.bucket(bucket)
.send()
.await
.expect("list before batch delete");
assert_eq!(
list.contents().len(),
5,
"RT-05c FAIL: expected 5 objects before batch delete, found {}",
list.contents().len()
);
// Batch delete
let objects: Vec<ObjectIdentifier> = keys
.iter()
.map(|k| ObjectIdentifier::builder().key(k).build().expect("build object id"))
.collect();
client
.delete_objects()
.bucket(bucket)
.delete(Delete::builder().set_objects(Some(objects)).build().expect("build delete"))
.send()
.await
.expect("batch delete");
// Verify all removed
let list = client
.list_objects_v2()
.bucket(bucket)
.send()
.await
.expect("list after batch delete");
assert!(
list.contents().is_empty(),
"RT-05c FAIL: {} objects remain after batch delete (regression: delete objects not fully applied)",
list.contents().len()
);
info!("RT-05c PASS: batch delete removes all objects");
Ok(())
}
/// RT-05d: Verify versioned delete → permanent delete → object gone.
///
/// Covers the pattern where permanent deletion of a specific version
/// fails with FileAccessDenied (rustfs#4978).
#[tokio::test]
#[serial]
async fn test_versioned_permanent_delete() -> TestResult {
init_logging();
info!("RT-05d: versioned permanent delete");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt05d-permanent-delete";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await
.expect("enable versioning");
// Upload a single object (single version)
let put_resp = client
.put_object()
.bucket(bucket)
.key("single-version.txt")
.body(ByteStream::from_static(b"to-be-permanently-deleted"))
.send()
.await
.expect("put object");
let version_id = put_resp.version_id().expect("version ID should be present").to_string();
// Permanently delete the specific version (rustfs#4978: FileAccessDenied)
client
.delete_object()
.bucket(bucket)
.key("single-version.txt")
.version_id(&version_id)
.send()
.await
.expect("permanent delete should succeed (regression rustfs#4978)");
// Verify the object is completely gone
let versions = client
.list_object_versions()
.bucket(bucket)
.send()
.await
.expect("list versions");
assert!(
versions.versions().is_empty(),
"RT-05d FAIL: version still present after permanent delete"
);
info!("RT-05d PASS: versioned permanent delete succeeds");
Ok(())
}
/// RT-05e: Verify delete marker + version history interaction.
///
/// Covers the pattern where creating a delete marker and then listing
/// versions shows incorrect state (rustfs#760).
#[tokio::test]
#[serial]
async fn test_versioned_delete_marker_and_list_consistency() -> TestResult {
init_logging();
info!("RT-05e: versioned delete marker and list consistency");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt05e-dm-consistency";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await
.expect("enable versioning");
// Create 3 versions
for i in 0..3 {
client
.put_object()
.bucket(bucket)
.key("history.txt")
.body(ByteStream::from(format!("v{i}").into_bytes()))
.send()
.await
.expect("put version");
}
// Create a delete marker
let del = client
.delete_object()
.bucket(bucket)
.key("history.txt")
.send()
.await
.expect("delete (create marker)");
assert!(del.delete_marker().unwrap_or(false), "RT-05e FAIL: should have created a delete marker");
// ListObjectVersions should show 3 versions + 1 delete marker
let versions = client
.list_object_versions()
.bucket(bucket)
.send()
.await
.expect("list versions");
assert_eq!(
versions.versions().len(),
3,
"RT-05e FAIL: expected 3 versions, found {}",
versions.versions().len()
);
assert_eq!(
versions.delete_markers().len(),
1,
"RT-05e FAIL: expected 1 delete marker, found {}",
versions.delete_markers().len()
);
// Now delete the delete marker (restore the object)
let dm_version = &versions.delete_markers()[0];
client
.delete_object()
.bucket(bucket)
.key("history.txt")
.version_id(dm_version.version_id().expect("dm version id"))
.send()
.await
.expect("delete delete-marker");
// HEAD should succeed now (latest version is accessible)
let head = client.head_object().bucket(bucket).key("history.txt").send().await;
assert!(head.is_ok(), "RT-05e FAIL: HEAD should succeed after removing delete marker");
info!("RT-05e PASS: versioned delete marker and list consistency");
Ok(())
}
/// RT-05f: Verify object deletion does not leave orphan data on disk.
///
/// Regression pattern: after delete, the object data files remain on disk
/// (rustfs#5029: Node Does Not Remove Files After Reconnect).
#[tokio::test]
#[serial]
async fn test_delete_removes_object_head_returns_404() -> TestResult {
init_logging();
info!("RT-05f: delete → HEAD 404 consistency");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt05f-delete-head";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Upload, delete, verify HEAD returns 404
let keys = vec!["small.txt", "medium.txt", "with-slash.txt", "special+chars.txt"];
for key in &keys {
client
.put_object()
.bucket(bucket)
.key(*key)
.body(ByteStream::from_static(b"delete-me"))
.send()
.await
.expect("put object");
}
for key in &keys {
client
.delete_object()
.bucket(bucket)
.key(*key)
.send()
.await
.expect("delete object");
}
// All HEAD requests should return 404
for key in &keys {
let head = client.head_object().bucket(bucket).key(*key).send().await;
assert!(head.is_err(), "RT-05f FAIL: HEAD on deleted key '{key}' should return error");
}
// LIST should be empty
let list = client
.list_objects_v2()
.bucket(bucket)
.send()
.await
.expect("list after all deletes");
assert!(
list.contents().is_empty(),
"RT-05f FAIL: {} objects remain after deleting all",
list.contents().len()
);
info!("RT-05f PASS: all deleted objects return 404 on HEAD");
Ok(())
}
}
@@ -1,202 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression tests for distributed cluster startup and quorum.
//!
//! Covers the recurring pattern where multi-node clusters fail to start due to
//! lock quorum issues, DNS resolution delays, or erasure quorum deadlocks.
//! This has regressed 7+ times.
//!
//! ## Regression Issues
//!
//! - rustfs#5416: RustFS cannot cold-start with 2/3 quorum when Pod DNS missing
//! - rustfs#2945: Distributed mode fails on K8s: erasure quorum deadlock
//! - rustfs#2794: distributed deployment does not become ready
//! - rustfs#2601: fresh pod immediately enters FaultyDisk state
//! - rustfs#4040: Distributed startup can fail lock quorum before AppContext initializes
//! - rustfs#5655: fix(ecstore): bootstrap fresh four-node clusters reliably
//! - rustfs#4954: S3/health endpoint unavailability after multi-pool scale-up
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestClusterEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use std::error::Error;
use tokio::time::{Duration, sleep};
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
/// RT-10: Verify 4-node cluster starts successfully and all nodes are ready.
///
/// Regression pattern: distributed startup fails with quorum deadlock or
/// lock acquisition timeout (rustfs#2945, rustfs#5655).
///
/// Steps:
/// 1. Create a 4-node cluster
/// 2. Start all nodes simultaneously
/// 3. Verify all nodes report healthy
/// 4. Verify S3 operations work through any node
#[tokio::test]
#[serial]
async fn test_four_node_cluster_startup_and_health() -> TestResult {
init_logging();
info!("RT-10: 4-node cluster startup and health");
let mut cluster = RustFSTestClusterEnvironment::new(4).await.expect("create 4-node cluster");
cluster.start().await.expect("start 4-node cluster");
// Create a bucket and verify it's accessible from all nodes
cluster
.create_test_bucket("rt10-startup")
.await
.expect("create bucket on cluster");
let clients = cluster.create_all_clients().expect("create per-node clients");
// Verify S3 operations work from every node
for (i, client) in clients.iter().enumerate() {
client
.put_object()
.bucket("rt10-startup")
.key(format!("from-node-{i}.txt"))
.body(ByteStream::from_static(b"hello from node"))
.send()
.await
.unwrap_or_else(|e| panic!("PUT from node {i} failed: {e}"));
}
// Verify all objects are visible from node 0
let list = clients[0]
.list_objects_v2()
.bucket("rt10-startup")
.send()
.await
.expect("list objects from node 0");
assert_eq!(
list.contents().len(),
4,
"RT-10 FAIL: expected 4 objects (one per node), found {}",
list.contents().len()
);
info!("RT-10 PASS: 4-node cluster starts and serves S3 from all nodes");
Ok(())
}
/// RT-10b: Verify cluster handles node restart gracefully.
///
/// Regression pattern: after a node restart, it cannot rejoin the cluster
/// or enters a faulty state (rustfs#2601).
#[tokio::test]
#[serial]
async fn test_cluster_survives_node_restart() -> TestResult {
init_logging();
info!("RT-10b: cluster survives node restart");
let mut cluster = RustFSTestClusterEnvironment::new(4).await.expect("create 4-node cluster");
cluster.start().await.expect("start cluster");
cluster.create_test_bucket("rt10b-restart").await.expect("create bucket");
// Write data
let clients = cluster.create_all_clients()?;
clients[0]
.put_object()
.bucket("rt10b-restart")
.key("before-restart.txt")
.body(ByteStream::from_static(b"persistent data"))
.send()
.await
.expect("put object before restart");
// Stop node 3
cluster.stop_node(3).expect("stop node 3");
sleep(Duration::from_secs(2)).await;
// Verify cluster still works with 3/4 nodes (quorum)
clients[0]
.put_object()
.bucket("rt10b-restart")
.key("during-offline.txt")
.body(ByteStream::from_static(b"written while node 3 down"))
.send()
.await
.expect("PUT should succeed with 3/4 nodes");
// Restart node 3
cluster.start_node(3).await.expect("restart node 3");
// Wait for node to rejoin
sleep(Duration::from_secs(3)).await;
// Verify the restarted node can serve reads
let list = clients[3]
.list_objects_v2()
.bucket("rt10b-restart")
.send()
.await
.expect("list from restarted node");
assert!(
list.contents().len() >= 2,
"RT-10b FAIL: restarted node sees {} objects, expected >= 2",
list.contents().len()
);
info!("RT-10b PASS: cluster survives and recovers from node restart");
Ok(())
}
/// RT-10c: Verify bucket creation persists across all nodes.
///
/// Regression pattern: bucket metadata is not replicated to all nodes,
/// causing NoSuchBucket errors on some nodes (rustfs#3191).
#[tokio::test]
#[serial]
async fn test_bucket_visible_from_all_nodes() -> TestResult {
init_logging();
info!("RT-10c: bucket visible from all nodes");
let mut cluster = RustFSTestClusterEnvironment::new(4).await.expect("create 4-node cluster");
cluster.start().await.expect("start cluster");
cluster
.create_test_bucket("rt10c-bucket-visibility")
.await
.expect("create bucket");
let clients = cluster.create_all_clients()?;
// Verify the bucket is visible from every node
for (i, client) in clients.iter().enumerate() {
let resp = client
.list_objects_v2()
.bucket("rt10c-bucket-visibility")
.send()
.await
.unwrap_or_else(|e| panic!("list from node {i} failed (NoSuchBucket?): {e}"));
assert!(resp.contents().is_empty(), "RT-10c: fresh bucket should be empty on node {i}");
}
info!("RT-10c PASS: bucket visible from all 4 nodes");
Ok(())
}
}
+1 -2
View File
@@ -574,8 +574,7 @@ fn parse_request(method: &Method, uri: &Uri) -> ParsedRequest {
(&Method::POST, true) if query.contains_key("uploads") => Operation::CreateMultipartUpload,
(&Method::POST, true) if upload_id.is_some() => Operation::CompleteMultipartUpload,
(&Method::DELETE, true) if upload_id.is_some() => Operation::AbortMultipartUpload,
// A replication PUT addresses the source version via `?versionId=`.
(&Method::PUT, true) if only_query_keys(&["versionId"]) => Operation::PutObject,
(&Method::PUT, true) if only_query_keys(&[]) => Operation::PutObject,
(&Method::GET, true) if only_query_keys(&["versionId"]) => Operation::GetObject,
(&Method::HEAD, true) if only_query_keys(&["versionId"]) => Operation::HeadObject,
(&Method::DELETE, true) if only_query_keys(&["versionId"]) => Operation::DeleteObject,
@@ -1687,44 +1687,6 @@ async fn four_node_inline_storage_and_get_boundaries() -> TestResult {
Ok(())
}
#[tokio::test]
#[serial]
async fn four_node_empty_legacy_volumes_start_as_fresh() -> TestResult {
init_logging();
let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
for data_dir in cluster.nodes.iter().flat_map(|node| &node.data_dirs) {
tokio::fs::create_dir_all(Path::new(data_dir).join(".minio.sys")).await?;
}
cluster.start().await?;
// Starting is not the assertion. The regression is that an empty legacy
// `.minio.sys` must be classified as a *fresh* volume, not as an existing
// MinIO deployment to adopt or migrate. Pin what that classification leaves
// on disk and in the namespace.
let buckets = cluster.create_s3_client(0)?.list_buckets().send().await?;
assert!(
buckets.buckets().is_empty(),
"a fresh classification must not adopt buckets from the pre-existing directories, got {:?}",
buckets.buckets().iter().filter_map(|b| b.name()).collect::<Vec<_>>()
);
for data_dir in cluster.nodes.iter().flat_map(|node| &node.data_dirs) {
assert!(
Path::new(data_dir).join(".rustfs.sys").join("format.json").is_file(),
"each drive must be formatted as fresh: {data_dir} has no .rustfs.sys/format.json"
);
let mut legacy = tokio::fs::read_dir(Path::new(data_dir).join(".minio.sys")).await?;
assert!(
legacy.next_entry().await?.is_none(),
"the empty legacy directory must be left untouched, not migrated into: {data_dir}"
);
}
Ok(())
}
#[tokio::test]
#[serial]
async fn four_node_inline_fallback_controls() -> TestResult {
@@ -2211,6 +2173,11 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
"queue_snapshot.{field} must be readable in terminal status: {terminal}"
);
}
assert!(
cold_tier_object_count(&cold_client).await? < 64,
"queue pressure should leave at least one object untransitioned"
);
Ok(())
}
@@ -21,12 +21,9 @@
use super::common::LocalKMSTestEnvironment;
use crate::common::{TEST_BUCKET, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
ChecksumAlgorithm, ChecksumMode, CompletedMultipartUpload, CompletedPart, ServerSideEncryption,
ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
};
use rustfs_rio::{Checksum, ChecksumType};
use serial_test::serial;
use tracing::{debug, info, warn};
@@ -276,7 +273,7 @@ async fn test_bucket_default_sse_kms_put_object() -> Result<(), Box<dyn std::err
/// Test 3: When bucket is configured with default encryption, create_multipart_upload should inherit the configuration
#[tokio::test]
#[serial]
async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
async fn test_bucket_default_encryption_multipart_upload() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Testing bucket default encryption impact on create_multipart_upload");
@@ -312,16 +309,15 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
.await
.expect("Failed to set bucket encryption");
// Step 2: Declare CRC32 without specifying encryption parameters. The AWS SDK
// calculates each UploadPart checksum and sends it as a flexible checksum.
info!("Creating CRC32 multipart upload that should use bucket default encryption");
let test_key = "test-multipart-bucket-default-crc32.bin";
// Step 2: Create multipart upload (without specifying encryption parameters)
info!("Creating multipart upload (without specifying encryption parameters, should use bucket default configuration)");
let test_key = "test-multipart-bucket-default.txt";
let create_multipart_response = s3_client
.create_multipart_upload()
.bucket(TEST_BUCKET)
.key(test_key)
.checksum_algorithm(ChecksumAlgorithm::Crc32)
// Note: No encryption parameters specified here, should use bucket default configuration
.send()
.await
.expect("Failed to create multipart upload");
@@ -347,61 +343,28 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
"create_multipart_upload response should contain correct KMS key ID"
);
// Step 3: Upload two parts. The first is exactly the S3 minimum size so this
// follows the same managed SSE-KMS multipart path as issue #5756.
const PART_SIZE: usize = 5 * 1024 * 1024;
let part1: Vec<u8> = (0..PART_SIZE).map(|i| (i % 251) as u8).collect();
let part2: Vec<u8> = (0..1024 * 1024).map(|i| ((i + 17) % 251) as u8).collect();
let expected_body: Vec<u8> = part1.iter().chain(&part2).copied().collect();
// Step 3: Upload a part and complete multipart upload
info!("Uploading part and completing multipart upload");
let test_data = b"test-multipart-bucket-default-encryption-data";
let upload_part = |part_number: i32, body: Vec<u8>| {
s3_client
.upload_part()
.bucket(TEST_BUCKET)
.key(test_key)
.upload_id(upload_id)
.part_number(part_number)
.checksum_algorithm(ChecksumAlgorithm::Crc32)
.body(ByteStream::from(body))
.send()
};
// Upload part 1
let upload_part_response = s3_client
.upload_part()
.bucket(TEST_BUCKET)
.key(test_key)
.upload_id(upload_id)
.part_number(1)
.body(test_data.to_vec().into())
.send()
.await
.expect("Failed to upload part");
let expected_part1_crc32 = Checksum::new_from_data(ChecksumType::CRC32, &part1)
.expect("calculate part 1 CRC32")
.encoded;
let upload1 = upload_part(1, part1).await.expect("Failed to upload part 1 with CRC32");
assert_eq!(
upload1.checksum_crc32(),
Some(expected_part1_crc32.as_str()),
"UploadPart must return the CRC32 calculated over plaintext"
);
let expected_part2_crc32 = Checksum::new_from_data(ChecksumType::CRC32, &part2)
.expect("calculate part 2 CRC32")
.encoded;
let upload2 = upload_part(2, part2).await.expect("Failed to upload part 2 with CRC32");
assert_eq!(
upload2.checksum_crc32(),
Some(expected_part2_crc32.as_str()),
"UploadPart must return the CRC32 calculated over plaintext"
);
let etag = upload_part_response.e_tag().unwrap().to_string();
// Complete multipart upload
let completed_upload = CompletedMultipartUpload::builder()
.parts(
CompletedPart::builder()
.part_number(1)
.e_tag(upload1.e_tag().expect("No ETag for part 1"))
.checksum_crc32(upload1.checksum_crc32().expect("No CRC32 for part 1"))
.build(),
)
.parts(
CompletedPart::builder()
.part_number(2)
.e_tag(upload2.e_tag().expect("No ETag for part 2"))
.checksum_crc32(upload2.checksum_crc32().expect("No CRC32 for part 2"))
.build(),
)
let completed_part = aws_sdk_s3::types::CompletedPart::builder()
.part_number(1)
.e_tag(&etag)
.build();
let complete_multipart_response = s3_client
@@ -409,7 +372,11 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
.bucket(TEST_BUCKET)
.key(test_key)
.upload_id(upload_id)
.multipart_upload(completed_upload)
.multipart_upload(
aws_sdk_s3::types::CompletedMultipartUpload::builder()
.parts(completed_part)
.build(),
)
.send()
.await
.expect("Failed to complete multipart upload");
@@ -433,7 +400,6 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
.get_object()
.bucket(TEST_BUCKET)
.key(test_key)
.checksum_mode(ChecksumMode::Enabled)
.send()
.await
.expect("Failed to get object");
@@ -444,13 +410,6 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
Some(&ServerSideEncryption::AwsKms),
"Final object should contain SSE-KMS encryption information"
);
if let Some(completed_crc32) = complete_multipart_response.checksum_crc32() {
assert_eq!(
get_response.checksum_crc32(),
Some(completed_crc32),
"GetObject should return the persisted composite CRC32 when completion reports it"
);
}
// Verify data integrity
let downloaded_data = get_response
@@ -459,11 +418,7 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
.await
.expect("Failed to collect body")
.into_bytes();
assert_eq!(
downloaded_data.as_ref(),
expected_body.as_slice(),
"Downloaded data should match the uploaded multipart body"
);
assert_eq!(&downloaded_data[..], test_data, "Downloaded data should match original data");
// Cleanup is handled automatically when the test environment is dropped
info!("Test passed: bucket default encryption correctly applied to multipart upload");
@@ -126,55 +126,7 @@ async fn assert_key_deletion_lifecycle(base_url: &str, access_key: &str, secret_
assert_eq!(cancelled["success"], true);
assert_eq!(cancelled["key_metadata"]["key_state"], "Enabled");
// A window outside 7-30 days is refused at the endpoint, whatever the
// backend: the bound is enforced once in the service, so no backend can
// stretch or skip it (rustfs/backlog#1585).
for days in [6, 31] {
let refused = kms_admin_request(
base_url,
http::Method::DELETE,
"/rustfs/admin/v3/kms/keys/delete",
Some(
&serde_json::json!({
"key_id": key_id,
"pending_window_in_days": days
})
.to_string(),
),
access_key,
secret_key,
)
.await
.err()
.ok_or_else(|| format!("a {days}-day deletion window must be refused"))?;
assert!(
refused.to_string().contains("400 Bad Request"),
"a {days}-day deletion window must report a client error: {refused}"
);
}
// Immediate deletion is no longer reachable through the query string, so it
// fails before the service gate is even consulted.
let refused = kms_admin_request(
base_url,
http::Method::DELETE,
&format!("/rustfs/admin/v3/kms/keys/delete?keyId={key_id}&force_immediate=true"),
None,
access_key,
secret_key,
)
.await
.err()
.ok_or("immediate KMS key deletion must not be reachable through the query string")?;
assert!(
refused.to_string().contains("400 Bad Request"),
"a query-string immediate deletion must report a client error: {refused}"
);
// A default server refuses to skip the waiting window (rustfs/backlog#1585):
// immediate deletion is unrecoverable and takes every object encrypted under
// the key with it, so the endpoint must reject it rather than honour it.
let refused = kms_admin_request(
let removed = kms_admin_request(
base_url,
http::Method::DELETE,
"/rustfs/admin/v3/kms/keys/delete",
@@ -188,49 +140,37 @@ async fn assert_key_deletion_lifecycle(base_url: &str, access_key: &str, secret_
access_key,
secret_key,
)
.await
.err()
.ok_or("immediate KMS key deletion must be refused on a default server")?;
assert!(
refused.to_string().contains("400 Bad Request"),
"refused immediate deletion must report a client error: {refused}"
);
// The refused requests left the key alone, so the window-bounded path still
// has something to schedule.
let described = kms_admin_request(
base_url,
http::Method::GET,
&format!("/rustfs/admin/v3/kms/keys/{key_id}"),
None,
access_key,
secret_key,
)
.await?;
let described: serde_json::Value = serde_json::from_str(&described)?;
assert_eq!(
described["key_metadata"]["key_state"], "Enabled",
"a refused immediate deletion must leave the key usable"
);
let removed: serde_json::Value = serde_json::from_str(&removed)?;
assert_eq!(removed["success"], true);
let rescheduled = kms_admin_request(
base_url,
http::Method::DELETE,
"/rustfs/admin/v3/kms/keys/delete",
Some(
&serde_json::json!({
"key_id": key_id,
"pending_window_in_days": 7
})
.to_string(),
),
access_key,
secret_key,
)
.await?;
let rescheduled: serde_json::Value = serde_json::from_str(&rescheduled)?;
assert_eq!(rescheduled["success"], true);
assert!(rescheduled["deletion_date"].is_string());
let listed =
kms_admin_request(base_url, http::Method::GET, "/rustfs/admin/v3/kms/keys", None, access_key, secret_key).await?;
let listed: serde_json::Value = serde_json::from_str(&listed)?;
assert_eq!(listed["success"], true);
let keys = listed["keys"]
.as_array()
.ok_or("list KMS keys response omitted keys after deletion")?;
if let Some(key) = keys.iter().find(|key| key["key_id"] == key_id) {
assert_eq!(key["status"], "PendingDeletion", "a retained force-deleted key must be pending deletion");
let removed = kms_admin_request(
base_url,
http::Method::DELETE,
"/rustfs/admin/v3/kms/keys/delete",
Some(
&serde_json::json!({
"key_id": key_id,
"force_immediate": true
})
.to_string(),
),
access_key,
secret_key,
)
.await?;
let removed: serde_json::Value = serde_json::from_str(&removed)?;
assert_eq!(removed["success"], true);
}
let listed =
kms_admin_request(base_url, http::Method::GET, "/rustfs/admin/v3/kms/keys", None, access_key, secret_key).await?;
@@ -239,11 +179,10 @@ async fn assert_key_deletion_lifecycle(base_url: &str, access_key: &str, secret_
let keys = listed["keys"]
.as_array()
.ok_or("final list KMS keys response omitted keys after deletion")?;
let key = keys
.iter()
.find(|key| key["key_id"] == key_id)
.ok_or("a key awaiting its deletion window must still be listed")?;
assert_eq!(key["status"], "PendingDeletion", "a scheduled key must be pending deletion");
assert!(
keys.iter().all(|key| key["key_id"] != key_id),
"force-deleted KMS key must no longer appear in list"
);
Ok(())
}
@@ -1,351 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression test: a same-key CopyObject that only rewrites metadata must never re-key a
//! managed-SSE (SSE-S3 / SSE-KMS) object.
//!
//! On an **unversioned** bucket the handler marks a same-name copy `metadata_only`, and the
//! store layer then updates `xl.meta` in place without touching the data blocks. The handler
//! nevertheless strips the source encryption metadata and generates a *fresh* DEK for the
//! destination. Combining the two writes "new DEK + old ciphertext": the object is permanently
//! undecryptable. The fix forces a full data rewrite whenever the copy re-derives managed
//! encryption material, so the stored bytes always match the key metadata beside them.
//!
//! Companion to `copy_object_version_restore_sse_test` (issue #4238), which pins the same
//! invariant for the versioned historical-restore path.
use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
use crate::common::init_logging;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
MetadataDirective, ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration,
ServerSideEncryptionRule,
};
use serial_test::serial;
use tracing::info;
#[tokio::test]
#[serial]
async fn test_metadata_replace_self_copy_of_sse_object_stays_decryptable() {
init_logging();
info!("same-key CopyObject with REPLACE metadata must not re-key an SSE-S3 object");
let mut kms_env = LocalKMSTestEnvironment::new().await.expect("failed to create local KMS env");
let default_key_id = "rustfs-e2e-test-default-key";
let keys_dir = kms_env.kms_keys_dir.clone();
create_key_with_specific_id(&keys_dir, default_key_id)
.await
.expect("failed to create local KMS key");
kms_env
.base_env
.start_rustfs_server_with_env(
vec![
"--kms-enable",
"--kms-backend",
"local",
"--kms-key-dir",
&keys_dir,
"--kms-default-key-id",
default_key_id,
],
&[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")],
)
.await
.expect("failed to start RustFS with local KMS");
tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
let client = kms_env.base_env.create_s3_client();
// Deliberately an UNVERSIONED bucket: that is the branch where the store layer can service
// the self-copy as a pure metadata update.
let bucket = "copy-object-self-copy-sse-test";
let key = "secrets/report.txt";
client
.create_bucket()
.bucket(bucket)
.send()
.await
.expect("failed to create bucket");
// Content long enough that a truncated/garbled decrypt cannot coincidentally match.
let content = b"encrypted payload that must survive a metadata-only self copy -- 0123456789";
let put = client
.put_object()
.bucket(bucket)
.key(key)
.content_type("text/plain; charset=utf-8")
.metadata("stage", "before")
.server_side_encryption(ServerSideEncryption::Aes256)
.body(ByteStream::from_static(content))
.send()
.await
.expect("PUT failed");
assert_eq!(put.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
// Copy the object onto itself, replacing user metadata. This is the `mc cp --attr` /
// "edit metadata in place" shape that AWS supports on an existing object.
let copy_out = client
.copy_object()
.bucket(bucket)
.key(key)
.copy_source(format!("{bucket}/{key}"))
.metadata_directive(MetadataDirective::Replace)
.content_type("text/plain; charset=utf-8")
.metadata("stage", "after")
.server_side_encryption(ServerSideEncryption::Aes256)
.send()
.await
.expect("same-key CopyObject with REPLACE metadata must succeed");
assert_eq!(copy_out.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
// The object must still decrypt to the original plaintext. Before the fix the stored
// ciphertext was left untouched while the metadata carried a brand-new DEK, so this GET
// either failed outright or returned garbage.
let get = client
.get_object()
.bucket(bucket)
.key(key)
.send()
.await
.expect("GET after self-copy failed: the object was re-keyed without rewriting the ciphertext");
assert_eq!(get.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
assert_eq!(
get.metadata().and_then(|m| m.get("stage")),
Some(&"after".to_string()),
"REPLACE metadata must take effect"
);
let body = get.body.collect().await.expect("collect body").into_bytes();
assert_eq!(
body.as_ref(),
content,
"object must still decrypt to the original plaintext after a metadata-only self copy"
);
kms_env.base_env.stop_server();
}
#[tokio::test]
#[serial]
async fn test_metadata_replace_self_copy_dropping_sse_rewrites_plaintext() {
init_logging();
info!("same-key CopyObject that drops SSE must rewrite the data, not orphan the ciphertext");
let mut kms_env = LocalKMSTestEnvironment::new().await.expect("failed to create local KMS env");
let default_key_id = "rustfs-e2e-test-default-key";
let keys_dir = kms_env.kms_keys_dir.clone();
create_key_with_specific_id(&keys_dir, default_key_id)
.await
.expect("failed to create local KMS key");
kms_env
.base_env
.start_rustfs_server_with_env(
vec![
"--kms-enable",
"--kms-backend",
"local",
"--kms-key-dir",
&keys_dir,
"--kms-default-key-id",
default_key_id,
],
&[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")],
)
.await
.expect("failed to start RustFS with local KMS");
tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
let client = kms_env.base_env.create_s3_client();
// Unversioned, and deliberately WITHOUT a bucket default-encryption rule, so the copy below
// resolves to "no destination encryption".
let bucket = "copy-object-self-copy-drop-sse-test";
let key = "secrets/report.txt";
client
.create_bucket()
.bucket(bucket)
.send()
.await
.expect("failed to create bucket");
let content = b"encrypted payload whose ciphertext must not survive as bogus plaintext -- 0123456789";
client
.put_object()
.bucket(bucket)
.key(key)
.metadata("stage", "before")
.server_side_encryption(ServerSideEncryption::Aes256)
.body(ByteStream::from_static(content))
.send()
.await
.expect("PUT failed");
// Self-copy with REPLACE and no SSE header. Per AWS semantics the destination ends up
// unencrypted. The dangerous outcome is the silent one: the handler strips the source key
// metadata while a metadata-only copy leaves the ciphertext in place, so a later GET would
// hand back raw ciphertext as if it were plaintext — corruption with no error anywhere.
client
.copy_object()
.bucket(bucket)
.key(key)
.copy_source(format!("{bucket}/{key}"))
.metadata_directive(MetadataDirective::Replace)
.metadata("stage", "after")
.send()
.await
.expect("same-key CopyObject dropping SSE must succeed");
let get = client
.get_object()
.bucket(bucket)
.key(key)
.send()
.await
.expect("GET after self-copy failed");
assert_eq!(
get.server_side_encryption(),
None,
"destination must be unencrypted once the copy drops SSE"
);
assert_eq!(
get.metadata().and_then(|m| m.get("stage")),
Some(&"after".to_string()),
"REPLACE metadata must take effect"
);
let body = get.body.collect().await.expect("collect body").into_bytes();
assert_eq!(
body.as_ref(),
content,
"object must read back as the original plaintext, not the orphaned ciphertext"
);
kms_env.base_env.stop_server();
}
#[tokio::test]
#[serial]
async fn test_metadata_replace_self_copy_under_bucket_default_sse_stays_decryptable() {
init_logging();
info!("bucket default encryption must also keep a same-key copy off the metadata-only path");
let mut kms_env = LocalKMSTestEnvironment::new().await.expect("failed to create local KMS env");
let default_key_id = "rustfs-e2e-test-default-key";
let keys_dir = kms_env.kms_keys_dir.clone();
create_key_with_specific_id(&keys_dir, default_key_id)
.await
.expect("failed to create local KMS key");
kms_env
.base_env
.start_rustfs_server_with_env(
vec![
"--kms-enable",
"--kms-backend",
"local",
"--kms-key-dir",
&keys_dir,
"--kms-default-key-id",
default_key_id,
],
&[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")],
)
.await
.expect("failed to start RustFS with local KMS");
tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;
let client = kms_env.base_env.create_s3_client();
let bucket = "copy-object-self-copy-bucket-default-sse-test";
let key = "secrets/report.txt";
client
.create_bucket()
.bucket(bucket)
.send()
.await
.expect("failed to create bucket");
// Store the object as PLAINTEXT first: no SSE header and no bucket default rule yet. This is
// what makes the case sharp — at copy time the source metadata carries no encryption markers,
// so the source-side half of the guard cannot fire.
let content = b"plaintext payload that must not be orphaned under a new DEK -- 0123456789";
let put = client
.put_object()
.bucket(bucket)
.key(key)
.metadata("stage", "before")
.body(ByteStream::from_static(content))
.send()
.await
.expect("PUT failed");
assert_eq!(put.server_side_encryption(), None, "the object must start out unencrypted");
// Only NOW enable bucket default encryption. The destination's encryption therefore comes
// from the bucket rule and from nowhere else: the source is unencrypted and the copy request
// carries no SSE header. A guard that only inspects request headers (MinIO decides
// `isTargetEncrypted` from `crypto.S3.IsRequested(r.Header)`) would let this through, yet
// `sse_encryption` still mints a fresh DEK from the resolved bucket default — which is why
// the guard keys off the *effective* encryption rather than the requested one.
let encryption_config = ServerSideEncryptionConfiguration::builder()
.rules(
ServerSideEncryptionRule::builder()
.apply_server_side_encryption_by_default(
ServerSideEncryptionByDefault::builder()
.sse_algorithm(ServerSideEncryption::Aes256)
.build()
.unwrap(),
)
.build(),
)
.build()
.unwrap();
client
.put_bucket_encryption()
.bucket(bucket)
.server_side_encryption_configuration(encryption_config)
.send()
.await
.expect("failed to set bucket default encryption");
// No SSE header on the copy — the bucket default alone drives the destination encryption.
client
.copy_object()
.bucket(bucket)
.key(key)
.copy_source(format!("{bucket}/{key}"))
.metadata_directive(MetadataDirective::Replace)
.metadata("stage", "after")
.send()
.await
.expect("same-key CopyObject under bucket default encryption must succeed");
let get = client
.get_object()
.bucket(bucket)
.key(key)
.send()
.await
.expect("GET after self-copy failed: the object was re-keyed without rewriting the ciphertext");
assert_eq!(get.server_side_encryption(), Some(&ServerSideEncryption::Aes256));
assert_eq!(
get.metadata().and_then(|m| m.get("stage")),
Some(&"after".to_string()),
"REPLACE metadata must take effect"
);
let body = get.body.collect().await.expect("collect body").into_bytes();
assert_eq!(
body.as_ref(),
content,
"object must still decrypt to the original plaintext after a metadata-only self copy"
);
kms_env.base_env.stop_server();
}
@@ -1,483 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Negative authorization matrix for per-key KMS access control.
//!
//! Every case here is an end-to-end denial that the pre-`kms` resource server
//! allowed, so a regression that reopens one of them fails this file rather than
//! only a unit test. The matrix varies one dimension at a time:
//!
//! - **wrong identity**: a caller holding S3 rights but no `kms` grant at all
//! - **wrong key**: a caller scoped to key A naming key B
//! - **wrong action**: a caller holding `kms:GenerateDataKey` but not `kms:Decrypt`
//! (and, on the admin plane, `kms:DisableKey` but not `kms:RotateKey`)
//! - **wrong context**: an explicit `Deny` beating a wildcard `Allow`, and SSE-S3
//! staying exempt from `kms` authorization
//!
//! Each matrix opens with a positive control. Without it a denial proves nothing:
//! an identity whose policy has not propagated yet is denied everything.
use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
use crate::common::{admin_ok, admin_request, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::config::{Config, Credentials, Region};
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::ServerSideEncryption;
use serial_test::serial;
use std::time::Duration;
use tracing::info;
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
const ALLOWED_KEY: &str = "kms-matrix-allowed-key";
const OTHER_KEY: &str = "kms-matrix-other-key";
const BUCKET: &str = "kms-authz-matrix";
const SECRET: &str = "kms-matrix-secret";
const PAYLOAD: &[u8] = b"kms authorization matrix payload";
/// How long an identity change may take to reach the request path.
const IAM_PROPAGATION: Duration = Duration::from_secs(20);
fn s3_client(url: &str, access_key: &str, secret_key: &str) -> Client {
let config = Config::builder()
.credentials_provider(Credentials::new(access_key, secret_key, None, None, "kms-authz-matrix"))
.region(Region::new("us-east-1"))
.endpoint_url(url)
.force_path_style(true)
.behavior_version_latest()
.build();
Client::from_conf(config)
}
/// Start a server whose SSE-KMS data path authorizes against the named key.
///
/// The enforcement switch defaults to off for compatibility, so it has to be set
/// explicitly; without it every negative case below would silently pass as an allow.
async fn start_enforcing_server(env: &mut LocalKMSTestEnvironment, extra_env: &[(&str, &str)]) -> TestResult {
create_key_with_specific_id(&env.kms_keys_dir, ALLOWED_KEY).await?;
create_key_with_specific_id(&env.kms_keys_dir, OTHER_KEY).await?;
let key_dir = env.kms_keys_dir.clone();
let args = vec![
"--kms-enable",
"--kms-backend",
"local",
"--kms-key-dir",
key_dir.as_str(),
"--kms-default-key-id",
ALLOWED_KEY,
];
let mut envs = vec![("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")];
envs.extend_from_slice(extra_env);
env.base_env.start_rustfs_server_with_env(args, &envs).await?;
Ok(())
}
/// Create `user` with `policy_document` attached under a canned policy of the same name.
async fn provision_user(env: &LocalKMSTestEnvironment, user: &str, policy_document: &str) -> TestResult {
admin_ok(
&env.base_env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-canned-policy?name={user}"),
Some(policy_document.to_string()),
)
.await?;
provision_user_with_policy(env, user, user).await
}
/// Create `user` and attach an existing policy (built-in or canned) by name.
async fn provision_user_with_policy(env: &LocalKMSTestEnvironment, user: &str, policy_name: &str) -> TestResult {
admin_ok(
&env.base_env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-user?accessKey={user}"),
Some(serde_json::json!({ "secretKey": SECRET, "status": "enabled" }).to_string()),
)
.await?;
admin_ok(
&env.base_env,
http::Method::PUT,
&format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={user}&isGroup=false"),
None,
)
.await?;
Ok(())
}
/// The S3 half of every data-path policy below: full object access, no KMS grant.
fn s3_full_access_statement() -> serde_json::Value {
serde_json::json!({
"Effect": "Allow",
"Action": ["s3:*"],
"Resource": ["arn:aws:s3:::*"]
})
}
fn policy_document(statements: Vec<serde_json::Value>) -> String {
serde_json::json!({ "Version": "2012-10-17", "Statement": statements }).to_string()
}
async fn put_sse_kms(client: &Client, key: &str, kms_key_id: &str) -> Result<(), aws_sdk_s3::Error> {
client
.put_object()
.bucket(BUCKET)
.key(key)
.body(ByteStream::from_static(PAYLOAD))
.server_side_encryption(ServerSideEncryption::AwsKms)
.ssekms_key_id(kms_key_id)
.send()
.await
.map(|_| ())
.map_err(aws_sdk_s3::Error::from)
}
/// Assert the operation failed with `AccessDenied` rather than any other error.
///
/// A bare `is_err` would also accept `KMSKeyDisabled` or an internal error, which
/// would hide both a leak of key state and an outage masquerading as a denial.
fn assert_access_denied<T: std::fmt::Debug>(result: Result<T, aws_sdk_s3::Error>, what: &str) {
let error = result.expect_err(&format!("{what} must be denied"));
assert_eq!(error.code(), Some("AccessDenied"), "{what} must fail with AccessDenied: {error:?}");
}
/// Retry an SSE-KMS write until the identity's policy has reached the request path.
async fn wait_for_sse_kms_write(client: &Client, key: &str, kms_key_id: &str) -> TestResult {
let deadline = tokio::time::Instant::now() + IAM_PROPAGATION;
loop {
match put_sse_kms(client, key, kms_key_id).await {
Ok(()) => return Ok(()),
Err(error) if tokio::time::Instant::now() >= deadline => {
return Err(format!("positive control never became authorized: {error:?}").into());
}
Err(_) => tokio::time::sleep(Duration::from_millis(500)).await,
}
}
}
/// Retry an admin call until it stops returning 403, i.e. the policy is live.
async fn wait_for_admin_success(
env: &LocalKMSTestEnvironment,
user: &str,
method: http::Method,
path: &str,
body: Option<String>,
) -> TestResult {
let deadline = tokio::time::Instant::now() + IAM_PROPAGATION;
loop {
let (status, response) = admin_request(&env.base_env.url, method.clone(), path, body.clone(), user, SECRET).await?;
if status.is_success() {
return Ok(());
}
if tokio::time::Instant::now() >= deadline {
return Err(format!("positive control never became authorized: {method} {path} -> {status} {response}").into());
}
tokio::time::sleep(Duration::from_millis(500)).await;
}
}
async fn assert_admin_denied(
env: &LocalKMSTestEnvironment,
user: &str,
method: http::Method,
path: &str,
body: Option<String>,
what: &str,
) -> TestResult {
let (status, response) = admin_request(&env.base_env.url, method, path, body, user, SECRET).await?;
assert_eq!(status.as_u16(), 403, "{what} must be denied, got {status}: {response}");
assert!(response.contains("AccessDenied"), "{what} must carry AccessDenied: {response}");
Ok(())
}
fn disable_body(key_id: &str) -> String {
serde_json::json!({ "key_id": key_id }).to_string()
}
/// Data-path matrix: SSE-KMS writes and reads are authorized against the resolved key.
#[tokio::test]
#[serial]
async fn sse_kms_per_key_authorization_negative_matrix() -> TestResult {
init_logging();
let mut env = LocalKMSTestEnvironment::new().await?;
start_enforcing_server(&mut env, &[("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "true")]).await?;
env.base_env.create_test_bucket(BUCKET).await?;
// Scoped to ALLOWED_KEY only.
provision_user(
&env,
"kmsmatrixscoped",
&policy_document(vec![
s3_full_access_statement(),
serde_json::json!({
"Effect": "Allow",
"Action": ["kms:GenerateDataKey", "kms:Decrypt"],
"Resource": [format!("arn:aws:kms:::key/{ALLOWED_KEY}")]
}),
]),
)
.await?;
// S3 rights only: the identity shape that existed before per-key authorization.
provision_user(&env, "kmsmatrixs3only", &policy_document(vec![s3_full_access_statement()])).await?;
// May wrap a data key but may never unwrap one.
provision_user(
&env,
"kmsmatrixwriter",
&policy_document(vec![
s3_full_access_statement(),
serde_json::json!({
"Effect": "Allow",
"Action": ["kms:GenerateDataKey"],
"Resource": ["arn:aws:kms:::*"]
}),
]),
)
.await?;
// Wildcard allow, explicit deny on one key.
provision_user(
&env,
"kmsmatrixdenied",
&policy_document(vec![
s3_full_access_statement(),
serde_json::json!({
"Effect": "Allow",
"Action": ["kms:*"],
"Resource": ["arn:aws:kms:::*"]
}),
serde_json::json!({
"Effect": "Deny",
"Action": ["kms:*"],
"Resource": [format!("arn:aws:kms:::key/{OTHER_KEY}")]
}),
]),
)
.await?;
let scoped = s3_client(&env.base_env.url, "kmsmatrixscoped", SECRET);
let s3_only = s3_client(&env.base_env.url, "kmsmatrixs3only", SECRET);
let writer = s3_client(&env.base_env.url, "kmsmatrixwriter", SECRET);
let denied = s3_client(&env.base_env.url, "kmsmatrixdenied", SECRET);
// --- positive control -----------------------------------------------------
wait_for_sse_kms_write(&scoped, "scoped/allowed", ALLOWED_KEY).await?;
let read = scoped.get_object().bucket(BUCKET).key("scoped/allowed").send().await?;
assert_eq!(read.body.collect().await?.into_bytes().as_ref(), PAYLOAD);
info!("positive control: scoped identity may write and read under its own key");
// --- wrong key ------------------------------------------------------------
assert_access_denied(
put_sse_kms(&scoped, "scoped/other", OTHER_KEY).await,
"SSE-KMS write under a key outside the identity's scope",
);
// --- wrong identity -------------------------------------------------------
assert_access_denied(
put_sse_kms(&s3_only, "s3only/allowed", ALLOWED_KEY).await,
"SSE-KMS write by an identity holding no kms grant",
);
// The object the scoped identity wrote is readable by its owner only.
assert_access_denied(
s3_only
.get_object()
.bucket(BUCKET)
.key("scoped/allowed")
.send()
.await
.map(|_| ())
.map_err(aws_sdk_s3::Error::from),
"SSE-KMS read by an identity holding no kms grant",
);
// --- wrong action ---------------------------------------------------------
wait_for_sse_kms_write(&writer, "writer/allowed", ALLOWED_KEY).await?;
assert_access_denied(
writer
.get_object()
.bucket(BUCKET)
.key("writer/allowed")
.send()
.await
.map(|_| ())
.map_err(aws_sdk_s3::Error::from),
"SSE-KMS read by an identity holding kms:GenerateDataKey but not kms:Decrypt",
);
// --- wrong context: explicit Deny beats a wildcard Allow -------------------
wait_for_sse_kms_write(&denied, "denied/allowed", ALLOWED_KEY).await?;
assert_access_denied(
put_sse_kms(&denied, "denied/other", OTHER_KEY).await,
"SSE-KMS write under a key covered by an explicit Deny",
);
// --- wrong context: SSE-S3 is out of scope --------------------------------
// SSE-S3 wraps its data key with a server-owned key the caller never names, so
// it must stay reachable for an identity with no kms grant at all.
s3_only
.put_object()
.bucket(BUCKET)
.key("s3only/sse-s3")
.body(ByteStream::from_static(PAYLOAD))
.server_side_encryption(ServerSideEncryption::Aes256)
.send()
.await?;
let sse_s3_read = s3_only.get_object().bucket(BUCKET).key("s3only/sse-s3").send().await?;
assert_eq!(sse_s3_read.body.collect().await?.into_bytes().as_ref(), PAYLOAD);
// ... and so must an unencrypted object.
s3_only
.put_object()
.bucket(BUCKET)
.key("s3only/plain")
.body(ByteStream::from_static(PAYLOAD))
.send()
.await?;
s3_only.get_object().bucket(BUCKET).key("s3only/plain").send().await?;
Ok(())
}
/// Admin-plane matrix: KMS key endpoints are authorized against the key they name.
///
/// Runs without the SSE enforcement switch: admin scoping is unconditional, and
/// leaving the switch off proves the two planes are independent.
#[tokio::test]
#[serial]
async fn kms_admin_per_key_authorization_negative_matrix() -> TestResult {
init_logging();
let mut env = LocalKMSTestEnvironment::new().await?;
start_enforcing_server(&mut env, &[]).await?;
// Built-in role templates, attached by name.
provision_user_with_policy(&env, "kmsmatrixkeyadmin", "KMSKeyAdministrator").await?;
provision_user_with_policy(&env, "kmsmatrixauditor", "KMSAuditor").await?;
// A narrowed copy of the administrator template, scoped to one key.
provision_user(
&env,
"kmsmatrixscopedadmin",
&policy_document(vec![serde_json::json!({
"Effect": "Allow",
"Action": ["kms:DisableKey", "kms:EnableKey"],
"Resource": [format!("arn:aws:kms:::key/{ALLOWED_KEY}")]
})]),
)
.await?;
// --- positive control -----------------------------------------------------
wait_for_admin_success(
&env,
"kmsmatrixkeyadmin",
http::Method::POST,
"/rustfs/admin/v3/kms/keys/disable",
Some(disable_body(OTHER_KEY)),
)
.await?;
admin_request(
&env.base_env.url,
http::Method::POST,
"/rustfs/admin/v3/kms/keys/enable",
Some(disable_body(OTHER_KEY)),
"kmsmatrixkeyadmin",
SECRET,
)
.await?;
// --- wrong action: the administrator template withholds service-wide powers -
assert_admin_denied(
&env,
"kmsmatrixkeyadmin",
http::Method::GET,
"/rustfs/admin/v3/kms/config",
None,
"KMSKeyAdministrator reading the KMS backend configuration (kms:Configure)",
)
.await?;
assert_admin_denied(
&env,
"kmsmatrixkeyadmin",
http::Method::GET,
"/rustfs/admin/v3/kms/backup",
None,
"KMSKeyAdministrator exporting a backup bundle (kms:Backup)",
)
.await?;
// Separation of duties: managing a key never implies using it.
assert_admin_denied(
&env,
"kmsmatrixkeyadmin",
http::Method::POST,
"/rustfs/admin/v3/kms/generate-data-key",
Some(serde_json::json!({ "key_id": ALLOWED_KEY }).to_string()),
"KMSKeyAdministrator generating a data key (kms:GenerateDataKey)",
)
.await?;
// --- wrong action: the auditor template is read-only ----------------------
wait_for_admin_success(&env, "kmsmatrixauditor", http::Method::GET, "/rustfs/admin/v3/kms/keys", None).await?;
assert_admin_denied(
&env,
"kmsmatrixauditor",
http::Method::POST,
"/rustfs/admin/v3/kms/keys/disable",
Some(disable_body(ALLOWED_KEY)),
"KMSAuditor disabling a key (kms:DisableKey)",
)
.await?;
// --- wrong key ------------------------------------------------------------
wait_for_admin_success(
&env,
"kmsmatrixscopedadmin",
http::Method::POST,
"/rustfs/admin/v3/kms/keys/disable",
Some(disable_body(ALLOWED_KEY)),
)
.await?;
assert_admin_denied(
&env,
"kmsmatrixscopedadmin",
http::Method::POST,
"/rustfs/admin/v3/kms/keys/disable",
Some(disable_body(OTHER_KEY)),
"key-scoped administrator disabling a key outside its scope",
)
.await?;
// --- wrong action, same key ----------------------------------------------
assert_admin_denied(
&env,
"kmsmatrixscopedadmin",
http::Method::POST,
"/rustfs/admin/v3/kms/keys/rotate",
Some(disable_body(ALLOWED_KEY)),
"key-scoped administrator rotating a key it may only enable and disable",
)
.await?;
// --- wrong identity -------------------------------------------------------
provision_user(&env, "kmsmatrixnokms", &policy_document(vec![s3_full_access_statement()])).await?;
assert_admin_denied(
&env,
"kmsmatrixnokms",
http::Method::POST,
"/rustfs/admin/v3/kms/keys/disable",
Some(disable_body(ALLOWED_KEY)),
"identity holding no kms grant disabling a key",
)
.await?;
Ok(())
}
+15 -34
View File
@@ -417,22 +417,6 @@ async fn test_vault_kms_key_crud(
info!("✅ Read: Successfully listed keys, found test key");
// A waiting window outside 7-30 days is refused at the endpoint for this
// backend too: the bound is enforced once in the service (rustfs/backlog#1585).
for days in [6, 31] {
let window_error = crate::common::execute_awscurl(
&format!("{base_url}/rustfs/admin/v3/kms/keys/delete?keyId={key_id}&pending_window_in_days={days}"),
"DELETE",
None,
access_key,
secret_key,
)
.await
.err()
.ok_or_else(|| format!("A {days}-day deletion window must be refused"))?;
info!("✅ Delete window {} correctly refused: {}", days, window_error);
}
// Delete
let delete_response = crate::common::execute_awscurl(
&format!("{base_url}/rustfs/admin/v3/kms/keys/delete?keyId={key_id}"),
@@ -465,32 +449,29 @@ async fn test_vault_kms_key_crud(
info!("✅ Delete verification: Key state correctly changed to: {}", key_state);
// Force Delete - the query string can no longer ask for immediate deletion,
// and a default server refuses it in any case (rustfs/backlog#1585):
// destroying the key material immediately would take every object encrypted
// under the key with it.
let force_delete_error = crate::common::execute_awscurl(
// Force Delete - Force immediate deletion for PendingDeletion key
let force_delete_response = crate::common::execute_awscurl(
&format!("{base_url}/rustfs/admin/v3/kms/keys/delete?keyId={key_id}&force_immediate=true"),
"DELETE",
None,
access_key,
secret_key,
)
.await
.expect_err("Immediate KMS key deletion must be refused on a default server");
info!("✅ Force Delete: correctly refused for key {}: {}", key_id, force_delete_error);
.await?;
// The refused request must leave the key exactly as it was: still present,
// still pending deletion, still recoverable through cancel-deletion.
let describe_after_refusal =
crate::common::awscurl_get(&format!("{base_url}/rustfs/admin/v3/kms/keys/{key_id}"), access_key, secret_key).await?;
let describe_after_refusal: serde_json::Value = serde_json::from_str(&describe_after_refusal)?;
assert_eq!(
describe_after_refusal["key_metadata"]["key_state"], "PendingDeletion",
"A refused immediate deletion must leave the key pending deletion"
);
// Parse and validate the force delete response
let force_delete_result: serde_json::Value = serde_json::from_str(&force_delete_response)?;
assert_eq!(force_delete_result["success"], true, "Force delete operation must return success=true");
info!("✅ Force Delete: Successfully force deleted key: {}", key_id);
info!("✅ Force Delete verification: Key survived the refused immediate deletion");
// Verify key no longer exists after force deletion (should return error)
let describe_force_deleted_result =
crate::common::awscurl_get(&format!("{base_url}/rustfs/admin/v3/kms/keys/{key_id}"), access_key, secret_key).await;
// After force deletion, key should not be found (GET should fail)
assert!(describe_force_deleted_result.is_err(), "Force deleted key should not be found");
info!("✅ Force Delete verification: Key was permanently deleted and is no longer accessible");
info!("Vault KMS key CRUD operations completed successfully");
Ok(())
-6
View File
@@ -48,14 +48,8 @@ mod bucket_default_encryption_test;
#[cfg(test)]
mod encryption_metadata_test;
#[cfg(test)]
mod copy_object_self_copy_sse_test;
#[cfg(test)]
mod copy_object_version_restore_sse_test;
#[cfg(test)]
mod configured_roundtrip_test;
#[cfg(test)]
mod kms_authorization_negative_matrix_test;
-36
View File
@@ -290,14 +290,6 @@ mod overwrite_cleanup_regression_test;
#[cfg(test)]
mod list_buckets_double_slash_test;
// Regression coverage for bucket-scoped ListBuckets authorization fallback.
#[cfg(test)]
mod list_buckets_auth_test;
// ListBuckets visibility follows IAM authorization, not bucket policy.
#[cfg(test)]
mod list_buckets_iam_filter_test;
// Regression test for backlog#629(b): region-aware CreateBucket SigV4.
#[cfg(test)]
mod create_bucket_region_test;
@@ -306,32 +298,4 @@ mod create_bucket_region_test;
#[cfg(test)]
mod copy_source_invalid_date_test;
// P0 regression: event notification startup race (rustfs#5387, #5681, #5401, #5183, #5115, #4796)
#[cfg(test)]
mod notification_startup_regression_test;
// P0 regression: lifecycle/ILM object expiration (rustfs#5407, #5167, #4963, #5615, #4879)
#[cfg(test)]
mod lifecycle_regression_test;
// P0 regression: delete operations consistency (rustfs#5375, #5349, #5339, #5029, #4978, #760)
#[cfg(test)]
mod delete_regression_test;
// P1 regression: listing/metacache completeness (rustfs#5166, #5156, #5051, #4810, #4648, #3191)
#[cfg(test)]
mod listing_regression_test;
// P1 regression: bucket statistics accuracy (rustfs#5615, #5008, #5116, #5055, #3898, #1012)
#[cfg(test)]
mod bucket_stats_regression_test;
// P1 regression: distributed startup/quorum (rustfs#5416, #2945, #2794, #2601, #4040, #5655)
#[cfg(test)]
mod distributed_startup_regression_test;
// P1 regression: tier/ILM transition (rustfs#5218, #5130, #5011, #4826, #5024)
#[cfg(test)]
mod tier_transition_regression_test;
pub mod tls_gen;
@@ -1,360 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression tests for lifecycle/ILM object expiration and transition.
//!
//! Covers the recurring pattern where ILM expiration rules do not actually
//! delete objects, or lifecycle rule parameters are silently corrupted.
//! This has regressed 6+ times.
//!
//! ## Regression Issues
//!
//! - rustfs#5407: lifecycle not delete any bucket object
//! - rustfs#5167: lifecycle not delete object
//! - rustfs#4963: lifecycle rule 3 days → effective value 0 days
//! - rustfs#5615: bucket statistics remain unchanged after data expiration
//! - rustfs#4879: ILM serial lane: restore transition never completes
//! - rustfs#5442: Uncheck of Replicate Delete still deletes the file
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
BucketLifecycleConfiguration, BucketVersioningStatus, ExpirationStatus, LifecycleExpiration, LifecycleRule,
LifecycleRuleFilter, NoncurrentVersionExpiration, VersioningConfiguration,
};
use serial_test::serial;
use std::error::Error;
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
async fn setup_versioned_bucket(client: &Client, bucket: &str) -> TestResult {
client
.create_bucket()
.bucket(bucket)
.send()
.await
.map_err(|e| format!("create bucket: {e}"))?;
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await
.map_err(|e| format!("enable versioning: {e}"))?;
Ok(())
}
/// RT-03: Verify that a lifecycle expiration rule actually deletes objects.
///
/// Regression pattern: lifecycle rules are accepted but the scanner never
/// processes them, leaving expired objects in place.
///
/// Steps:
/// 1. Create a versioned bucket
/// 2. Upload several objects
/// 3. Apply a lifecycle rule with 1-day expiration
/// 4. Wait for the scanner to process
/// 5. Verify objects are still present (they shouldn't expire yet — 1 day)
/// 6. Verify the lifecycle rule was persisted correctly (not corrupted to 0 days)
///
/// This tests the rule persistence path (rustfs#4963: 3 days → 0 days).
#[tokio::test]
#[serial]
async fn test_lifecycle_expiration_rule_persists_correctly() -> TestResult {
init_logging();
info!("RT-03: lifecycle expiration rule persists correctly");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt03-lifecycle-persist";
setup_versioned_bucket(&client, bucket).await?;
// Apply a lifecycle rule with 1-day expiration on a prefix
let rule = LifecycleRule::builder()
.id("expire-after-1-day")
.status(ExpirationStatus::Enabled)
.filter(LifecycleRuleFilter::builder().prefix("logs/").build())
.expiration(LifecycleExpiration::builder().days(1).build())
.build()
.expect("build lifecycle rule");
client
.put_bucket_lifecycle_configuration()
.bucket(bucket)
.lifecycle_configuration(
BucketLifecycleConfiguration::builder()
.rules(rule)
.build()
.expect("build lifecycle config"),
)
.send()
.await
.expect("put lifecycle configuration");
// Read back and verify the rule was not corrupted (rustfs#4963: days → 0)
let resp = client
.get_bucket_lifecycle_configuration()
.bucket(bucket)
.send()
.await
.expect("get lifecycle configuration");
let rules = resp.rules();
assert_eq!(rules.len(), 1, "RT-03 FAIL: expected exactly 1 lifecycle rule");
let retrieved = &rules[0];
assert_eq!(retrieved.id(), Some("expire-after-1-day"), "RT-03 FAIL: rule ID mismatch");
assert_eq!(retrieved.status(), &ExpirationStatus::Enabled, "RT-03 FAIL: rule should be Enabled");
let exp = retrieved.expiration().expect("expiration should be set");
assert_eq!(
exp.days(),
Some(1),
"RT-03 FAIL: expiration days corrupted (regression rustfs#4963: expected 1, got {:?})",
exp.days()
);
info!("RT-03 PASS: lifecycle expiration rule persists correctly");
Ok(())
}
/// RT-03b: Verify lifecycle rule with noncurrent version expiration.
///
/// Covers the pattern where noncurrent version expiration rules are
/// accepted but old versions are never cleaned up.
#[tokio::test]
#[serial]
async fn test_lifecycle_noncurrent_version_expiration_rule_persists() -> TestResult {
init_logging();
info!("RT-03b: noncurrent version expiration rule persists");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt03b-noncurrent-expire";
setup_versioned_bucket(&client, bucket).await?;
// Create multiple versions of the same object
for i in 0..3 {
client
.put_object()
.bucket(bucket)
.key("versioned-obj.txt")
.body(ByteStream::from(format!("version-{i}").into_bytes()))
.send()
.await
.expect("put object version");
}
// Verify we have 3 versions
let versions = client
.list_object_versions()
.bucket(bucket)
.send()
.await
.expect("list versions");
let count = versions.versions().len();
assert_eq!(count, 3, "RT-03b FAIL: expected 3 versions, found {count}");
// Apply noncurrent version expiration rule
let rule = LifecycleRule::builder()
.id("expire-noncurrent-after-1-day")
.status(ExpirationStatus::Enabled)
.filter(LifecycleRuleFilter::builder().prefix("").build())
.noncurrent_version_expiration(NoncurrentVersionExpiration::builder().noncurrent_days(1).build())
.build()
.expect("build lifecycle rule");
client
.put_bucket_lifecycle_configuration()
.bucket(bucket)
.lifecycle_configuration(
BucketLifecycleConfiguration::builder()
.rules(rule)
.build()
.expect("build lifecycle config"),
)
.send()
.await
.expect("put lifecycle configuration");
// Read back and verify
let resp = client
.get_bucket_lifecycle_configuration()
.bucket(bucket)
.send()
.await
.expect("get lifecycle configuration");
let rules = resp.rules();
assert_eq!(rules.len(), 1, "RT-03b FAIL: expected 1 rule");
let nc_exp = rules[0]
.noncurrent_version_expiration()
.expect("noncurrent expiration should be set");
assert_eq!(nc_exp.noncurrent_days(), Some(1), "RT-03b FAIL: noncurrent days corrupted");
info!("RT-03b PASS: noncurrent version expiration rule persists correctly");
Ok(())
}
/// RT-04: Verify lifecycle rule with prefix filter persists after restart.
///
/// Covers the pattern where lifecycle rules are accepted but silently lost
/// after restart. Transition rules require a configured remote tier
/// (tested in reliant/tiering.rs), so this test uses expiration only.
#[tokio::test]
#[serial]
async fn test_lifecycle_prefix_rule_persists() -> TestResult {
init_logging();
info!("RT-04: lifecycle prefix rule persists");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt04-lifecycle-prefix";
setup_versioned_bucket(&client, bucket).await?;
let rule = LifecycleRule::builder()
.id("expire-archive-after-7-days")
.status(ExpirationStatus::Enabled)
.filter(LifecycleRuleFilter::builder().prefix("archive/").build())
.expiration(LifecycleExpiration::builder().days(7).build())
.build()
.expect("build lifecycle rule");
client
.put_bucket_lifecycle_configuration()
.bucket(bucket)
.lifecycle_configuration(
BucketLifecycleConfiguration::builder()
.rules(rule)
.build()
.expect("build lifecycle config"),
)
.send()
.await
.expect("put lifecycle configuration");
// Restart server
env.restart_server_preserving_data(vec![], &[]).await.expect("restart RustFS");
// Verify the rule survived restart
let resp = client
.get_bucket_lifecycle_configuration()
.bucket(bucket)
.send()
.await
.expect("get lifecycle after restart");
let rules = resp.rules();
assert_eq!(rules.len(), 1, "RT-04 FAIL: expected 1 rule after restart");
let exp = rules[0].expiration().expect("expiration should be set");
assert_eq!(exp.days(), Some(7), "RT-04 FAIL: expiration days corrupted after restart");
info!("RT-04 PASS: lifecycle prefix rule persists after restart");
Ok(())
}
/// RT-05b: Verify delete marker creation in versioned bucket.
///
/// Regression pattern: DELETE on a versioned object fails or does not
/// create a delete marker, or the delete marker is not visible in LIST.
#[tokio::test]
#[serial]
async fn test_delete_marker_creation_and_visibility() -> TestResult {
init_logging();
info!("RT-05b: delete marker creation and visibility");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt05b-delete-marker";
setup_versioned_bucket(&client, bucket).await?;
// Put an object
client
.put_object()
.bucket(bucket)
.key("marker-test.txt")
.body(ByteStream::from_static(b"to-be-deleted"))
.send()
.await
.expect("put object");
// Delete without specifying versionId → should create a delete marker
let del_resp = client
.delete_object()
.bucket(bucket)
.key("marker-test.txt")
.send()
.await
.expect("delete object");
// The response should indicate a delete marker was created
assert!(
del_resp.delete_marker().unwrap_or(false),
"RT-05b FAIL: DELETE on versioned object did not create a delete marker"
);
// ListObjectVersions should show both the original version and the delete marker
let versions = client
.list_object_versions()
.bucket(bucket)
.send()
.await
.expect("list versions");
let delete_markers: Vec<_> = versions
.delete_markers()
.iter()
.filter(|dm| dm.key() == Some("marker-test.txt"))
.collect();
assert_eq!(
delete_markers.len(),
1,
"RT-05b FAIL: expected 1 delete marker, found {}",
delete_markers.len()
);
info!("RT-05b PASS: delete marker created and visible");
Ok(())
}
}
@@ -1,88 +0,0 @@
// Copyright 2026 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression coverage for the MinIO-compatible filtered ListBuckets fallback.
use crate::common::{RustFSTestEnvironment, admin_ok, init_logging};
use std::error::Error;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
#[tokio::test]
async fn bucket_scoped_policy_returns_only_authorized_bucket() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let root_client = env.create_s3_client();
let allowed_bucket = "list-buckets-authorized";
let hidden_bucket = "list-buckets-hidden";
let user = "listbucketsuser";
let secret = "listbucketssecret";
let policy = "list-buckets-scoped";
root_client.create_bucket().bucket(allowed_bucket).send().await?;
root_client.create_bucket().bucket(hidden_bucket).send().await?;
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-canned-policy?name={policy}"),
Some(
serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:*"],
"Resource": [
format!("arn:aws:s3:::{allowed_bucket}"),
format!("arn:aws:s3:::{allowed_bucket}/*")
]
}]
})
.to_string(),
),
)
.await?;
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-user?accessKey={user}"),
Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
)
.await?;
admin_ok(
&env,
http::Method::POST,
"/rustfs/admin/v3/idp/builtin/policy/attach",
Some(serde_json::json!({ "policies": [policy], "user": user }).to_string()),
)
.await?;
let client = env.create_s3_client_with_credentials(user, secret);
// Capture ListBuckets first so the direct-access control cannot warm bucket metadata and mask the regression.
let listed = client.list_buckets().send().await;
client.list_objects_v2().bucket(allowed_bucket).send().await?;
let listed = listed?;
let names = listed
.buckets()
.iter()
.filter_map(|bucket| bucket.name().map(ToOwned::to_owned))
.collect::<Vec<_>>();
assert_eq!(names, vec![allowed_bucket]);
Ok(())
}
@@ -1,459 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::common::{RustFSTestEnvironment, admin_ok, build_test_s3_config, build_test_sts_client, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use serial_test::serial;
use tokio::time::{Duration, Instant};
fn user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
Client::from_conf(build_test_s3_config(
&env.url,
access_key,
secret_key,
session_token,
"list-buckets-iam-filter",
))
}
fn bucket_names(buckets: &[aws_sdk_s3::types::Bucket]) -> Vec<String> {
let mut names = buckets
.iter()
.filter_map(|bucket| bucket.name().map(str::to_owned))
.collect::<Vec<_>>();
names.sort();
names
}
async fn create_user(
env: &RustFSTestEnvironment,
access_key: &str,
secret_key: &str,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let body = serde_json::json!({ "secretKey": secret_key, "status": "enabled" }).to_string();
admin_ok(
env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-user?accessKey={access_key}"),
Some(body),
)
.await?;
Ok(())
}
async fn create_service_account(
env: &RustFSTestEnvironment,
target_user: &str,
policy: Option<&serde_json::Value>,
) -> Result<(String, String), Box<dyn std::error::Error + Send + Sync>> {
let request = match policy {
Some(policy) => serde_json::json!({ "targetUser": target_user, "policy": policy }),
None => serde_json::json!({ "targetUser": target_user }),
};
let response = admin_ok(env, http::Method::PUT, "/rustfs/admin/v3/add-service-accounts", Some(request.to_string())).await?;
let response: serde_json::Value = serde_json::from_str(&response)?;
let access_key = response["credentials"]["accessKey"]
.as_str()
.ok_or("service account response should contain credentials.accessKey")?
.to_owned();
let secret_key = response["credentials"]["secretKey"]
.as_str()
.ok_or("service account response should contain credentials.secretKey")?
.to_owned();
Ok((access_key, secret_key))
}
#[tokio::test]
#[serial]
async fn list_buckets_filters_with_iam_bucket_resources() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.capture_log_path = Some(format!("{}/server.log", env.temp_dir));
env.start_rustfs_server_with_env(vec![], &[("RUST_LOG", "rustfs=debug,rustfs_notify=debug")])
.await?;
let admin_client = env.create_s3_client();
for bucket in [
"benchmark-artifacts",
"benchmark-denied",
"benchmark-location-only",
"benchmark-test1",
"testuser1-artifacts",
] {
admin_client.create_bucket().bucket(bucket).send().await?;
}
assert_eq!(
bucket_names(admin_client.list_buckets().send().await?.buckets()),
vec![
"benchmark-artifacts",
"benchmark-denied",
"benchmark-location-only",
"benchmark-test1",
"testuser1-artifacts"
]
);
let access_key = "benchmark";
let secret_key = "benchmark-secret-1234567890";
create_user(&env, access_key, secret_key).await?;
let policy_name = "benchmark-bucket-prefix";
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:*"],
"Resource": ["arn:aws:s3:::benchmark-*", "arn:aws:s3:::benchmark-*/*"],
"Condition": {
"StringEquals": {
"s3:prefix": [""],
"s3:delimiter": ["/"]
}
}
},
{
"Effect": "Deny",
"Action": ["s3:ListBucket", "s3:GetBucketLocation"],
"Resource": ["arn:aws:s3:::benchmark-denied"]
},
{
"Effect": "Deny",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::benchmark-location-only"]
},
{
"Effect": "Allow",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"]
}
]
})
.to_string();
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-canned-policy?name={policy_name}"),
Some(policy),
)
.await?;
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={access_key}&isGroup=false"),
Some(String::new()),
)
.await?;
let bucket_policy_allow = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": { "AWS": [access_key] },
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::testuser1-artifacts"]
}]
})
.to_string();
admin_client
.put_bucket_policy()
.bucket("testuser1-artifacts")
.policy(bucket_policy_allow)
.send()
.await?;
let bucket_policy_deny = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Deny",
"Principal": { "AWS": [access_key] },
"Action": ["s3:ListBucket", "s3:GetBucketLocation"],
"Resource": ["arn:aws:s3:::benchmark-artifacts"]
}]
})
.to_string();
admin_client
.put_bucket_policy()
.bucket("benchmark-artifacts")
.policy(bucket_policy_deny)
.send()
.await?;
let benchmark_client = user_client(&env, access_key, secret_key, None);
benchmark_client
.list_objects_v2()
.bucket("testuser1-artifacts")
.send()
.await?;
assert_eq!(
bucket_names(benchmark_client.list_buckets().send().await?.buckets()),
vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
);
let log_path = env.capture_log_path.as_deref().expect("server log path should be configured");
let deadline = Instant::now() + Duration::from_secs(5);
let audit_log = loop {
let audit_log = tokio::fs::read_to_string(log_path).await?;
if [
"iam_implicit_deny",
"s3_authorization_denied",
"ListAllMyBucketsAction",
"benchmark",
"DEBUG",
]
.iter()
.all(|field| audit_log.contains(field))
|| Instant::now() >= deadline
{
break audit_log;
}
tokio::time::sleep(Duration::from_millis(50)).await;
};
assert_eq!(audit_log.matches("iam_implicit_deny").count(), 1, "{audit_log}");
for field in ["s3_authorization_denied", "ListAllMyBucketsAction", "benchmark", "DEBUG"] {
assert!(audit_log.contains(field), "missing {field} in {audit_log}");
}
let denied_access_key = "no-bucket-access";
let denied_secret_key = "no-bucket-access-secret-1234567890";
create_user(&env, denied_access_key, denied_secret_key).await?;
let denied = user_client(&env, denied_access_key, denied_secret_key, None)
.list_buckets()
.send()
.await
.expect_err("a user without IAM bucket permissions must be denied");
assert_eq!(denied.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
let put_only_policy_name = "put-only-no-bucket-discovery";
let put_only_policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:PutObject"],
"Resource": ["arn:aws:s3:::benchmark-*/*"]
}]
})
.to_string();
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-canned-policy?name={put_only_policy_name}"),
Some(put_only_policy),
)
.await?;
admin_ok(
&env,
http::Method::PUT,
&format!(
"/rustfs/admin/v3/set-user-or-group-policy?policyName={put_only_policy_name}&userOrGroup={denied_access_key}&isGroup=false"
),
Some(String::new()),
)
.await?;
let denied = user_client(&env, denied_access_key, denied_secret_key, None)
.list_buckets()
.send()
.await
.expect_err("an unrelated IAM action must not reveal bucket names");
assert_eq!(denied.as_service_error().and_then(ProvideErrorMetadata::code), Some("AccessDenied"));
let list_all_policy_name = "list-all-buckets";
let list_all_policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:ListAllMyBuckets"],
"Resource": ["arn:aws:s3:::*"]
}]
})
.to_string();
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-canned-policy?name={list_all_policy_name}"),
Some(list_all_policy),
)
.await?;
admin_ok(
&env,
http::Method::PUT,
&format!(
"/rustfs/admin/v3/set-user-or-group-policy?policyName={list_all_policy_name}&userOrGroup={denied_access_key}&isGroup=false"
),
Some(String::new()),
)
.await?;
assert_eq!(
bucket_names(
user_client(&env, denied_access_key, denied_secret_key, None)
.list_buckets()
.send()
.await?
.buckets()
),
vec![
"benchmark-artifacts",
"benchmark-denied",
"benchmark-location-only",
"benchmark-test1",
"testuser1-artifacts"
]
);
let group_user = "benchmark-group-user";
let group_secret = "benchmark-group-secret-1234567890";
let group_name = "benchmark-group";
create_user(&env, group_user, group_secret).await?;
admin_ok(
&env,
http::Method::PUT,
"/rustfs/admin/v3/update-group-members",
Some(
serde_json::json!({
"group": group_name,
"members": [group_user],
"isRemove": false,
"groupStatus": "enabled"
})
.to_string(),
),
)
.await?;
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/set-user-or-group-policy?policyName={policy_name}&userOrGroup={group_name}&isGroup=true"),
Some(String::new()),
)
.await?;
assert_eq!(
bucket_names(
user_client(&env, group_user, group_secret, None)
.list_buckets()
.send()
.await?
.buckets()
),
vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
);
let (service_access_key, service_secret_key) = create_service_account(&env, group_user, None).await?;
assert_eq!(
bucket_names(
user_client(&env, &service_access_key, &service_secret_key, None)
.list_buckets()
.send()
.await?
.buckets()
),
vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
);
let service_account_policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:ListBucket", "s3:GetBucketLocation"],
"Resource": ["arn:aws:s3:::benchmark-test1"],
"Condition": {
"StringEquals": {
"s3:prefix": [""],
"s3:delimiter": ["/"]
}
}
}]
});
let (restricted_service_access_key, restricted_service_secret_key) =
create_service_account(&env, group_user, Some(&service_account_policy)).await?;
assert_eq!(
bucket_names(
user_client(&env, &restricted_service_access_key, &restricted_service_secret_key, None,)
.list_buckets()
.send()
.await?
.buckets()
),
vec!["benchmark-test1"]
);
let sts_client = build_test_sts_client(&env.url, group_user, group_secret, None, "list-buckets-iam-filter-sts");
let inherited = sts_client
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/list-buckets")
.role_session_name("list-buckets-iam-filter-inherited")
.send()
.await?;
let inherited = inherited
.credentials()
.ok_or("AssumeRole response should contain inherited temporary credentials")?;
assert_eq!(
bucket_names(
user_client(
&env,
inherited.access_key_id(),
inherited.secret_access_key(),
Some(inherited.session_token()),
)
.list_buckets()
.send()
.await?
.buckets()
),
vec!["benchmark-artifacts", "benchmark-location-only", "benchmark-test1"]
);
let session_policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:ListBucket", "s3:GetBucketLocation"],
"Resource": ["arn:aws:s3:::benchmark-test1"],
"Condition": {
"StringEquals": {
"s3:prefix": [""],
"s3:delimiter": ["/"]
}
}
}]
})
.to_string();
let assumed = sts_client
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/list-buckets")
.role_session_name("list-buckets-iam-filter")
.policy(session_policy)
.send()
.await?;
let temporary = assumed
.credentials()
.ok_or("AssumeRole response should contain temporary credentials")?;
assert_eq!(
bucket_names(
user_client(
&env,
temporary.access_key_id(),
temporary.secret_access_key(),
Some(temporary.session_token()),
)
.list_buckets()
.send()
.await?
.buckets()
),
vec!["benchmark-test1"]
);
Ok(())
}
@@ -1,357 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression tests for object listing and metacache consistency.
//!
//! Covers the recurring pattern where ListObjectsV2 returns incomplete results,
//! silently truncates with IsTruncated=false, or corrupts the metadata cache.
//! This has regressed 8+ times.
//!
//! ## Regression Issues
//!
//! - rustfs#5166: Metacache listing quorum failed timeout after cluster startup
//! - rustfs#5156: Metacache producer failed
//! - rustfs#5051: ListObjectsV2 returns empty results for shallow prefixes
//! - rustfs#4810: walk_dir timeout silently truncates listings (200, IsTruncated=false)
//! - rustfs#4648: Object listing oscillates between complete, partial, and zero
//! - rustfs#3191: ListObjectsV2 timeout corrupts metadata cache → NoSuchBucket
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use std::collections::HashSet;
use std::error::Error;
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
/// RT-06: Verify ListObjectsV2 pagination completeness for medium-sized bucket.
///
/// Regression pattern: listing returns 200 with IsTruncated=false but
/// misses objects (rustfs#4810: walk_dir timeout truncation).
///
/// Steps:
/// 1. Upload 100 objects with known keys
/// 2. List all objects via pagination (max_keys=10)
/// 3. Verify all 100 keys are returned exactly once
/// 4. Verify no duplicates or skipped keys
#[tokio::test]
#[serial]
async fn test_list_objects_v2_completeness_100_objects() -> TestResult {
init_logging();
info!("RT-06: listing completeness with 100 objects");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt06-list-completeness";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Upload 100 objects
let expected_keys: Vec<String> = (0..100).map(|i| format!("obj-{i:04}.txt")).collect();
for key in &expected_keys {
client
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"data"))
.send()
.await
.expect("put object");
}
// Paginate through all objects (small page size to force multiple pages)
let mut all_keys: Vec<String> = Vec::new();
let mut continuation_token: Option<String> = None;
loop {
let mut req = client.list_objects_v2().bucket(bucket).max_keys(10);
if let Some(ref token) = continuation_token {
req = req.continuation_token(token);
}
let resp = req.send().await.expect("list objects page");
for obj in resp.contents() {
all_keys.push(obj.key().unwrap_or("").to_string());
}
if !resp.is_truncated().unwrap_or(false) {
break;
}
continuation_token = resp.next_continuation_token().map(|s| s.to_string());
}
// Verify completeness and uniqueness
let unique_keys: HashSet<&str> = all_keys.iter().map(|s| s.as_str()).collect();
assert_eq!(
all_keys.len(),
100,
"RT-06 FAIL: expected 100 objects, listed {} (regression: walk_dir truncation)",
all_keys.len()
);
assert_eq!(
unique_keys.len(),
100,
"RT-06 FAIL: found {} unique keys but listed {} total (duplicates!)",
unique_keys.len(),
all_keys.len()
);
for key in &expected_keys {
assert!(
unique_keys.contains(key.as_str()),
"RT-06 FAIL: key '{key}' missing from listing (regression rustfs#4810)"
);
}
info!("RT-06 PASS: all 100 objects listed completely and uniquely");
Ok(())
}
/// RT-06b: Verify listing with prefix filter returns correct subset.
///
/// Regression pattern: prefix filter returns empty or includes wrong keys
/// (rustfs#5051: empty results for shallow prefixes).
#[tokio::test]
#[serial]
async fn test_list_objects_v2_prefix_filter_correctness() -> TestResult {
init_logging();
info!("RT-06b: prefix filter correctness");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt06b-prefix-filter";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Upload objects with different prefixes
for i in 0..5 {
client
.put_object()
.bucket(bucket)
.key(format!("logs/app-{i:04}.log"))
.body(ByteStream::from_static(b"log data"))
.send()
.await
.expect("put log object");
client
.put_object()
.bucket(bucket)
.key(format!("data/file-{i:04}.csv"))
.body(ByteStream::from_static(b"csv data"))
.send()
.await
.expect("put data object");
}
// List with prefix "logs/" — should return exactly 5
let resp = client
.list_objects_v2()
.bucket(bucket)
.prefix("logs/")
.send()
.await
.expect("list with prefix");
assert_eq!(
resp.contents().len(),
5,
"RT-06b FAIL: expected 5 objects with prefix 'logs/', found {} (regression rustfs#5051)",
resp.contents().len()
);
for obj in resp.contents() {
assert!(
obj.key().unwrap_or("").starts_with("logs/"),
"RT-06b FAIL: object '{}' does not match prefix 'logs/'",
obj.key().unwrap_or("?")
);
}
// List with prefix "data/" — should return exactly 5
let resp = client
.list_objects_v2()
.bucket(bucket)
.prefix("data/")
.send()
.await
.expect("list with data/ prefix");
assert_eq!(
resp.contents().len(),
5,
"RT-06b FAIL: expected 5 objects with prefix 'data/', found {}",
resp.contents().len()
);
// List with prefix "nonexistent/" — should return 0
let resp = client
.list_objects_v2()
.bucket(bucket)
.prefix("nonexistent/")
.send()
.await
.expect("list with nonexistent prefix");
assert!(
resp.contents().is_empty(),
"RT-06b FAIL: expected 0 objects with prefix 'nonexistent/', found {}",
resp.contents().len()
);
info!("RT-06b PASS: prefix filter returns correct subset");
Ok(())
}
/// RT-06c: Verify listing with delimiter and CommonPrefixes.
///
/// Regression pattern: delimiter handling produces incorrect CommonPrefixes
/// or misses objects at the delimiter boundary.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_delimiter_common_prefixes() -> TestResult {
init_logging();
info!("RT-06c: delimiter and CommonPrefixes");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt06c-delimiter";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Create a hierarchical structure
let keys = vec!["a.txt", "dir1/b.txt", "dir1/sub1/c.txt", "dir1/sub2/d.txt", "dir2/e.txt"];
for key in &keys {
client
.put_object()
.bucket(bucket)
.key(*key)
.body(ByteStream::from_static(b"content"))
.send()
.await
.expect("put object");
}
// List with delimiter "/" at root level
let resp = client
.list_objects_v2()
.bucket(bucket)
.delimiter("/")
.send()
.await
.expect("list with delimiter");
// Should have 1 object (a.txt) and 2 common prefixes (dir1/, dir2/)
let contents: Vec<_> = resp.contents().iter().map(|o| o.key().unwrap_or("")).collect();
let prefixes: Vec<_> = resp.common_prefixes().iter().map(|p| p.prefix().unwrap_or("")).collect();
assert!(contents.contains(&"a.txt"), "RT-06c FAIL: root object 'a.txt' missing from listing");
assert_eq!(contents.len(), 1, "RT-06c FAIL: expected 1 root-level object, found {}", contents.len());
assert_eq!(prefixes.len(), 2, "RT-06c FAIL: expected 2 common prefixes, found {:?}", prefixes);
assert!(prefixes.contains(&"dir1/"), "RT-06c FAIL: 'dir1/' missing from CommonPrefixes");
assert!(prefixes.contains(&"dir2/"), "RT-06c FAIL: 'dir2/' missing from CommonPrefixes");
info!("RT-06c PASS: delimiter and CommonPrefixes correct");
Ok(())
}
/// RT-06d: Verify listing returns correct IsTruncated flag.
///
/// Regression pattern: IsTruncated=false when there are more objects
/// (rustfs#4810: walk_dir timeout truncation with false IsTruncated).
#[tokio::test]
#[serial]
async fn test_list_objects_v2_is_truncated_correctness() -> TestResult {
init_logging();
info!("RT-06d: IsTruncated correctness");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt06d-truncated";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Upload 15 objects
for i in 0..15 {
client
.put_object()
.bucket(bucket)
.key(format!("item-{i:04}.txt"))
.body(ByteStream::from_static(b"data"))
.send()
.await
.expect("put object");
}
// List with max_keys=5 — should be truncated
let resp = client
.list_objects_v2()
.bucket(bucket)
.max_keys(5)
.send()
.await
.expect("list with max_keys=5");
assert!(
resp.is_truncated().unwrap_or(false),
"RT-06d FAIL: IsTruncated should be true with 15 objects and max_keys=5"
);
assert_eq!(resp.contents().len(), 5, "RT-06d FAIL: expected 5 objects in first page");
assert!(
resp.next_continuation_token().is_some(),
"RT-06d FAIL: NextContinuationToken should be present when truncated"
);
// List with max_keys=100 — should NOT be truncated
let resp = client
.list_objects_v2()
.bucket(bucket)
.max_keys(100)
.send()
.await
.expect("list with max_keys=100");
assert!(
!resp.is_truncated().unwrap_or(false),
"RT-06d FAIL: IsTruncated should be false with 15 objects and max_keys=100"
);
assert_eq!(resp.contents().len(), 15, "RT-06d FAIL: expected 15 objects with max_keys=100");
info!("RT-06d PASS: IsTruncated flag is correct");
Ok(())
}
}
+50 -411
View File
@@ -62,33 +62,6 @@ fn md5_hex(input: impl AsRef<[u8]>) -> String {
hex::encode(hasher.finalize())
}
async fn create_restricted_user(
env: &RustFSTestEnvironment,
username: &str,
secret_key: &str,
) -> Result<(), Box<dyn Error + Send + Sync>> {
let url = format!("{}/rustfs/admin/v3/add-user?accessKey={username}", env.url);
let body = serde_json::json!({
"secretKey": secret_key,
"status": "enabled"
})
.to_string();
crate::common::awscurl_put(&url, &body, &env.access_key, &env.secret_key).await?;
Ok(())
}
fn restricted_user_client(env: &RustFSTestEnvironment, username: &str, secret_key: &str) -> aws_sdk_s3::Client {
let credentials = aws_sdk_s3::config::Credentials::new(username, secret_key, None, None, "snowball-pax-auth-test");
let config = aws_sdk_s3::Config::builder()
.credentials_provider(credentials)
.region(aws_sdk_s3::config::Region::new("us-east-1"))
.endpoint_url(&env.url)
.force_path_style(true)
.behavior_version_latest()
.build();
aws_sdk_s3::Client::from_conf(config)
}
/// Env var consumed by the local SSE-S3 DEK provider when KMS is not configured.
///
/// Since rustfs#3564 the server fails closed on managed SSE (SSE-S3 or
@@ -3584,8 +3557,8 @@ async fn test_anonymous_post_object_rejects_expires_field_missing_from_policy_co
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_retention_without_permission()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
async fn test_anonymous_post_object_accepts_object_lock_retention_fields() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -3594,6 +3567,8 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_without_permis
let bucket = "anon-post-policy-object-lock-retention";
let object_key = "uploads/object-lock-retention.txt";
let retain_until = "2037-10-21T07:28:00Z";
let expected_body = b"post-policy-object-lock-retention-body".to_vec();
let admin_client = env.create_s3_client();
admin_client
.create_bucket()
@@ -3618,7 +3593,7 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_without_permis
.text("x-amz-object-lock-retain-until-date", retain_until)
.part(
"file",
reqwest::multipart::Part::bytes(b"post-policy-object-lock-retention-body".to_vec())
reqwest::multipart::Part::bytes(expected_body.clone())
.file_name("upload.txt")
.mime_str("text/plain")?,
);
@@ -3632,8 +3607,26 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_without_permis
let status = post_resp.status();
let response_body = post_resp.text().await?;
assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
assert!(response_body.contains("AccessDenied"));
assert_eq!(status, reqwest::StatusCode::NO_CONTENT);
assert!(response_body.is_empty(), "204 response should not contain a body, got: {response_body}");
let retention = admin_client
.get_object_retention()
.bucket(bucket)
.key(object_key)
.send()
.await?;
let retention = retention.retention().expect("retention should be present");
assert_eq!(retention.mode().map(|value| value.as_str()), Some("GOVERNANCE"));
let retain_until_out = retention
.retain_until_date()
.expect("retain_until_date should be present")
.fmt(aws_sdk_s3::primitives::DateTimeFormat::DateTime)?;
assert_eq!(retain_until_out, retain_until);
let get_out = admin_client.get_object().bucket(bucket).key(object_key).send().await?;
let uploaded = get_out.body.collect().await?.into_bytes();
assert_eq!(uploaded.as_ref(), expected_body.as_slice());
Ok(())
}
@@ -3822,8 +3815,8 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_missing_from_p
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permission()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
async fn test_anonymous_post_object_accepts_object_lock_legal_hold_field() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -3831,6 +3824,8 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permi
let bucket = "anon-post-policy-object-lock-legal-hold";
let object_key = "uploads/object-lock-legal-hold.txt";
let expected_body = b"post-policy-object-lock-legal-hold-body".to_vec();
let admin_client = env.create_s3_client();
admin_client
.create_bucket()
@@ -3853,7 +3848,7 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permi
.text("x-amz-object-lock-legal-hold", "ON")
.part(
"file",
reqwest::multipart::Part::bytes(b"post-policy-object-lock-legal-hold-body".to_vec())
reqwest::multipart::Part::bytes(expected_body.clone())
.file_name("upload.txt")
.mime_str("text/plain")?,
);
@@ -3867,8 +3862,26 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permi
let status = post_resp.status();
let response_body = post_resp.text().await?;
assert_eq!(status, reqwest::StatusCode::FORBIDDEN);
assert!(response_body.contains("AccessDenied"));
assert_eq!(status, reqwest::StatusCode::NO_CONTENT);
assert!(response_body.is_empty(), "204 response should not contain a body, got: {response_body}");
let legal_hold = admin_client
.get_object_legal_hold()
.bucket(bucket)
.key(object_key)
.send()
.await?;
assert_eq!(
legal_hold
.legal_hold()
.and_then(|value| value.status())
.map(|value| value.as_str()),
Some("ON")
);
let get_out = admin_client.get_object().bucket(bucket).key(object_key).send().await?;
let uploaded = get_out.body.collect().await?.into_bytes();
assert_eq!(uploaded.as_ref(), expected_body.as_slice());
Ok(())
}
@@ -5645,70 +5658,6 @@ async fn test_signed_put_object_extract_preserves_object_lock_retention() -> Res
Ok(())
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_pax_retention_overrides_request_retention()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let bucket = "signed-extract-pax-retention-precedence";
let archive_key = "retention.tar";
let extracted_key = "alpha.txt";
let request_retain_until = aws_sdk_s3::primitives::DateTime::from_secs(2_114_380_800);
let pax_retain_until = "2040-01-01T00:00:00Z";
let client = env.create_s3_client();
client
.create_bucket()
.bucket(bucket)
.object_lock_enabled_for_bucket(true)
.send()
.await?;
let pax = HashMap::from([
("minio.metadata.x-amz-object-lock-mode", "COMPLIANCE".to_string()),
("minio.metadata.x-amz-object-lock-retain-until-date", pax_retain_until.to_string()),
]);
let archive = make_tar_with_pax_entry(extracted_key, b"alpha-body", None, &pax).await;
client
.put_object()
.bucket(bucket)
.key(archive_key)
.object_lock_mode(aws_sdk_s3::types::ObjectLockMode::Governance)
.object_lock_retain_until_date(request_retain_until)
.body(ByteStream::from(archive))
.customize()
.mutate_request(|req| {
req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
})
.send()
.await?;
let retention = client
.get_object_retention()
.bucket(bucket)
.key(extracted_key)
.send()
.await?
.retention()
.expect("retention should be present")
.clone();
assert_eq!(retention.mode().map(|value| value.as_str()), Some("COMPLIANCE"));
assert_eq!(
retention
.retain_until_date()
.expect("retain_until_date should be present")
.fmt(aws_sdk_s3::primitives::DateTimeFormat::DateTime)?,
pax_retain_until
);
Ok(())
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_returns_archive_etag() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
@@ -5833,316 +5782,6 @@ async fn test_signed_put_object_extract_preserves_pax_metadata_and_version_id()
Ok(())
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_authorizes_each_pax_privilege_and_retention_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if !crate::common::awscurl_available() {
return Ok(());
}
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let bucket = "signed-extract-pax-auth";
let put_only_user = "snowball-put-only";
let put_only_secret = "snowball-put-only-secret";
let conditional_user = "snowball-retention-condition";
let conditional_secret = "snowball-retention-condition-secret";
let wrong_action_user = "snowball-wrong-action";
let wrong_action_secret = "snowball-wrong-action-secret";
let version_condition_user = "snowball-version-condition";
let version_condition_secret = "snowball-version-condition-secret";
let pax_context_user = "snowball-pax-context";
let pax_context_secret = "snowball-pax-context-secret";
let conditional_version_id = Uuid::new_v4().to_string();
let admin_client = env.create_s3_client();
admin_client
.create_bucket()
.bucket(bucket)
.object_lock_enabled_for_bucket(true)
.send()
.await?;
create_restricted_user(&env, put_only_user, put_only_secret).await?;
create_restricted_user(&env, conditional_user, conditional_secret).await?;
create_restricted_user(&env, wrong_action_user, wrong_action_secret).await?;
create_restricted_user(&env, version_condition_user, version_condition_secret).await?;
create_restricted_user(&env, pax_context_user, pax_context_secret).await?;
let object_resource = format!("arn:aws:s3:::{bucket}/*");
let context_archive_resources = [
format!("arn:aws:s3:::{bucket}/tag-context.tar"),
format!("arn:aws:s3:::{bucket}/lock-context.tar"),
];
let tag_entry_resource = format!("arn:aws:s3:::{bucket}/tag-context-entry.txt");
let lock_entry_resource = format!("arn:aws:s3:::{bucket}/lock-context-entry.txt");
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PutOnly",
"Effect": "Allow",
"Principal": { "AWS": [put_only_user] },
"Action": ["s3:PutObject"],
"Resource": [object_resource.clone()]
},
{
"Sid": "RetentionWithLimit",
"Effect": "Allow",
"Principal": { "AWS": [conditional_user] },
"Action": ["s3:PutObject", "s3:PutObjectRetention"],
"Resource": [object_resource.clone()]
},
{
"Sid": "DenyRetentionBeyondCutoff",
"Effect": "Deny",
"Principal": { "AWS": [conditional_user] },
"Action": ["s3:PutObject"],
"Resource": [object_resource.clone()],
"Condition": {
"DateGreaterThan": {
"s3:object-lock-retain-until-date": "2030-01-01T00:00:00Z"
}
}
},
{
"Sid": "WrongAdditionalAction",
"Effect": "Allow",
"Principal": { "AWS": [wrong_action_user] },
"Action": ["s3:PutObject", "s3:PutObjectLegalHold"],
"Resource": [object_resource.clone()]
},
{
"Sid": "VersionConditionPut",
"Effect": "Allow",
"Principal": { "AWS": [version_condition_user] },
"Action": ["s3:PutObject"],
"Resource": [object_resource.clone()]
},
{
"Sid": "VersionConditionReplicate",
"Effect": "Allow",
"Principal": { "AWS": [version_condition_user] },
"Action": ["s3:ReplicateObject"],
"Resource": [object_resource],
"Condition": {
"StringEquals": {
"s3:VersionId": conditional_version_id.clone()
}
}
},
{
"Sid": "PaxContextArchives",
"Effect": "Allow",
"Principal": { "AWS": [pax_context_user] },
"Action": ["s3:PutObject", "s3:PutObjectRetention", "s3:PutObjectTagging"],
"Resource": context_archive_resources
},
{
"Sid": "PaxTagContextPut",
"Effect": "Allow",
"Principal": { "AWS": [pax_context_user] },
"Action": ["s3:PutObject"],
"Resource": [tag_entry_resource.clone()],
"Condition": {
"StringEquals": {
"s3:RequestObjectTag/classification": "public"
}
}
},
{
"Sid": "PaxTagContextAction",
"Effect": "Allow",
"Principal": { "AWS": [pax_context_user] },
"Action": ["s3:PutObjectTagging"],
"Resource": [tag_entry_resource]
},
{
"Sid": "PaxLockContextPut",
"Effect": "Allow",
"Principal": { "AWS": [pax_context_user] },
"Action": ["s3:PutObject"],
"Resource": [lock_entry_resource.clone()],
"Condition": {
"StringEquals": {
"s3:object-lock-mode": "COMPLIANCE"
}
}
},
{
"Sid": "PaxLockContextAction",
"Effect": "Allow",
"Principal": { "AWS": [pax_context_user] },
"Action": ["s3:PutObjectRetention"],
"Resource": [lock_entry_resource]
}
]
})
.to_string();
admin_client.put_bucket_policy().bucket(bucket).policy(policy).send().await?;
let put_only_client = restricted_user_client(&env, put_only_user, put_only_secret);
let conditional_client = restricted_user_client(&env, conditional_user, conditional_secret);
let wrong_action_client = restricted_user_client(&env, wrong_action_user, wrong_action_secret);
let cases = [
(
"legal-hold.tar",
put_only_client,
HashMap::from([("minio.metadata.x-amz-object-lock-legal-hold", "ON".to_string())]),
),
(
"retention-condition.tar",
conditional_client,
HashMap::from([
("minio.metadata.x-amz-object-lock-mode", "COMPLIANCE".to_string()),
("minio.metadata.x-amz-object-lock-retain-until-date", "2099-01-01T00:00:00Z".to_string()),
]),
),
(
"version-id.tar",
wrong_action_client,
HashMap::from([("minio.versionId", Uuid::new_v4().to_string())]),
),
];
for (archive_key, client, pax) in cases {
let archive = make_tar_with_pax_entry("entry.txt", b"must-not-write", None, &pax).await;
let err = client
.put_object()
.bucket(bucket)
.key(archive_key)
.body(ByteStream::from(archive))
.customize()
.mutate_request(|req| {
req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
})
.send()
.await
.expect_err("missing, conditional, or wrong PAX privilege must be rejected");
assert_eq!(
err.as_service_error().and_then(|error| error.meta().code()),
Some("AccessDenied"),
"{archive_key}"
);
}
let version_condition_client = restricted_user_client(&env, version_condition_user, version_condition_secret);
let mismatching_version_pax = HashMap::from([("minio.versionId", Uuid::new_v4().to_string())]);
let archive = make_tar_with_pax_entry("version-mismatch-entry.txt", b"must-not-write", None, &mismatching_version_pax).await;
let err = version_condition_client
.put_object()
.bucket(bucket)
.key("version-mismatch.tar")
.body(ByteStream::from(archive))
.customize()
.mutate_request(|req| {
req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
})
.send()
.await
.expect_err("a mismatching PAX version ID must fail the replication condition");
assert_eq!(err.as_service_error().and_then(|error| error.meta().code()), Some("AccessDenied"));
let err = admin_client
.head_object()
.bucket(bucket)
.key("version-mismatch-entry.txt")
.send()
.await
.expect_err("a denied PAX entry must not be written");
assert!(matches!(
err.as_service_error().and_then(|error| error.meta().code()),
Some("NoSuchKey" | "NotFound")
));
let matching_version_pax = HashMap::from([("minio.versionId", conditional_version_id)]);
let archive = make_tar_with_pax_entry("condition-entry.txt", b"condition-body", None, &matching_version_pax).await;
version_condition_client
.put_object()
.bucket(bucket)
.key("version-condition.tar")
.body(ByteStream::from(archive))
.customize()
.mutate_request(|req| {
req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
})
.send()
.await?;
let stored = admin_client
.get_object()
.bucket(bucket)
.key("condition-entry.txt")
.send()
.await?;
assert_eq!(stored.body.collect().await?.into_bytes().as_ref(), b"condition-body");
let pax_context_client = restricted_user_client(&env, pax_context_user, pax_context_secret);
let tag_pax = HashMap::from([("minio.metadata.x-amz-tagging", "classification=public".to_string())]);
let archive = make_tar_with_pax_entry("tag-context-entry.txt", b"tag-context-body", None, &tag_pax).await;
pax_context_client
.put_object()
.bucket(bucket)
.key("tag-context.tar")
.tagging("classification=restricted")
.body(ByteStream::from(archive))
.customize()
.mutate_request(|req| {
req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
})
.send()
.await?;
let tags = admin_client
.get_object_tagging()
.bucket(bucket)
.key("tag-context-entry.txt")
.send()
.await?;
assert!(
tags.tag_set()
.iter()
.any(|tag| tag.key() == "classification" && tag.value() == "public")
);
let pax_retain_until = "2040-01-01T00:00:00Z";
let lock_pax = HashMap::from([
("minio.metadata.x-amz-object-lock-mode", "COMPLIANCE".to_string()),
("minio.metadata.x-amz-object-lock-retain-until-date", pax_retain_until.to_string()),
]);
let archive = make_tar_with_pax_entry("lock-context-entry.txt", b"lock-context-body", None, &lock_pax).await;
pax_context_client
.put_object()
.bucket(bucket)
.key("lock-context.tar")
.object_lock_mode(aws_sdk_s3::types::ObjectLockMode::Governance)
.object_lock_retain_until_date(aws_sdk_s3::primitives::DateTime::from_secs(2_114_380_800))
.body(ByteStream::from(archive))
.customize()
.mutate_request(|req| {
req.headers_mut().insert("x-amz-meta-snowball-auto-extract", "true");
})
.send()
.await?;
let retention = admin_client
.get_object_retention()
.bucket(bucket)
.key("lock-context-entry.txt")
.send()
.await?
.retention()
.expect("PAX retention should be present")
.clone();
assert_eq!(retention.mode().map(|mode| mode.as_str()), Some("COMPLIANCE"));
assert_eq!(
retention
.retain_until_date()
.expect("PAX retain-until should be present")
.fmt(aws_sdk_s3::primitives::DateTimeFormat::DateTime)?,
pax_retain_until
);
Ok(())
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_accepts_compat_header() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
@@ -1,153 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression tests for the event notification startup race.
//!
//! Covers the recurring pattern where webhook/audit targets fail to load at boot
//! due to startup ordering (notification runtime starts before server config is
//! loaded). This has regressed 9+ times across beta.3 ~ beta.12.
//!
//! ## Regression Issues
//!
//! - rustfs#5387: webhook notifications broken again in beta.9+
//! - rustfs#5681: Audit webhook targets are not loaded at boot
//! - rustfs#5401: Event Destinations broken again
//! - rustfs#5183: Audit webhooks stay offline after restart
//! - rustfs#5115: init_event_notifier loses startup race against server config load
//! - rustfs#4796: Pulsar event destinations offline after restart
//! - rustfs#5428: MQTT bucket notifications stop on restarted cluster node
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use serial_test::serial;
use std::error::Error;
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
/// RT-01: Verify that the notification runtime initializes correctly at boot.
///
/// Regression pattern: notification runtime initializes before server config
/// is fully loaded, causing webhook targets to never come online.
///
/// This test verifies the startup ordering by checking that the server
/// starts successfully with notification enabled and can serve S3 requests.
/// A full webhook delivery test is in notification_webhook_test.rs.
#[tokio::test]
#[serial]
async fn test_notification_enabled_server_starts_cleanly() -> TestResult {
init_logging();
info!("RT-01: notification enabled server starts cleanly");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false"), ("RUSTFS_NOTIFY_ENABLE", "true")])
.await
.expect("start RustFS with notifications enabled");
let client = env.create_s3_client();
let bucket = "rt01-notify-startup";
// Server should be healthy and able to serve S3 requests
client
.create_bucket()
.bucket(bucket)
.send()
.await
.expect("create bucket with notifications enabled");
client
.put_object()
.bucket(bucket)
.key("test.txt")
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"test"))
.send()
.await
.expect("put object with notifications enabled");
info!("RT-01 PASS: notification enabled server starts and serves S3");
Ok(())
}
/// RT-02: Verify notification config persists after server restart.
///
/// Regression pattern: after a node restart, notification targets stay
/// offline permanently because the config is not re-loaded.
///
/// Steps:
/// 1. Start server with notification enabled
/// 2. Create bucket and configure notification
/// 3. Restart server
/// 4. Verify notification config still exists
#[tokio::test]
#[serial]
async fn test_notification_config_survives_restart() -> TestResult {
init_logging();
info!("RT-02: notification config survives restart");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false"), ("RUSTFS_NOTIFY_ENABLE", "true")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt02-notify-restart";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Enable versioning (required for notification configuration)
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await
.expect("enable versioning");
// Note: We can't fully test notification config persistence without a
// configured target. But we verify the server restarts cleanly with
// notification enabled, which is the core regression scenario.
env.restart_server_preserving_data(vec![], &[])
.await
.expect("restart RustFS with notifications enabled");
// Verify bucket still exists and is accessible after restart
let list = client
.list_objects_v2()
.bucket(bucket)
.send()
.await
.expect("list objects after restart");
assert!(list.contents().is_empty(), "RT-02: bucket should be empty after restart");
// Verify we can still write objects (notification runtime initialized)
client
.put_object()
.bucket(bucket)
.key("after-restart.txt")
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"post-restart"))
.send()
.await
.expect("put object after restart — notification runtime must be initialized");
info!("RT-02 PASS: server with notifications survives restart");
Ok(())
}
}
@@ -24,7 +24,7 @@
//! * PUT / multipart-complete / DeleteObject / DeleteObjects each deliver one event with the correct
//! eventName, bucket, key, versionId and eTag.
//! * prefix/suffix filters drop non-matching keys (rule-engine gate).
//! * an event queued while the target endpoint rejects delivery is redelivered
//! * an event queued while the target endpoint is unreachable is redelivered
//! from the on-disk store once the endpoint recovers (store-and-forward).
//! * responseElements and the S3 response use the canonical request ID while
//! requestParameters preserve a conflicting client-supplied value.
@@ -897,10 +897,11 @@ async fn test_webhook_event_delivery_and_filtering() -> TestResult {
Ok(())
}
/// An event queued while the target endpoint rejects delivery survives on the
/// An event queued while the target endpoint is unreachable survives on the
/// durable store and is redelivered once the endpoint comes back.
#[tokio::test]
#[serial]
#[ignore = "FAILING deterministically on main since it landed (#4821): the target is created but never appears in /rustfs/admin/v3/target/arns, so wait_for_target_registered times out. Quarantined per the flake policy; remove with the fix for rustfs#4852"]
async fn test_webhook_redelivers_event_after_target_recovers() -> TestResult {
init_logging();
@@ -931,55 +932,28 @@ async fn test_webhook_redelivers_event_after_target_recovers() -> TestResult {
wait_for_target_registered(&env, target).await?;
put_notification_config(&client, bucket, target, "uploads/", ".dat").await?;
// Replace the healthy setup listener with one that rejects the first POST.
// Waiting for that response below proves the queued event reached a failed
// delivery attempt before the endpoint recovers.
// Take the endpoint down (drops the listener, so connections are refused —
// a retryable NotConnected), then PUT: the event cannot be delivered and
// must survive on the durable queue store.
setup_handle.abort();
let _ = setup_handle.await;
let listener = TcpListener::bind(("0.0.0.0", port)).await?;
let key = "uploads/redeliver.dat";
client
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"queued while target rejects"))
.body(ByteStream::from_static(b"queued while target down"))
.send()
.await?;
let mut failure_handle = tokio::spawn(async move {
loop {
let (mut stream, _) = listener.accept().await?;
let (method, _) = timeout(Duration::from_secs(5), read_http_message(&mut stream)).await??;
if method == "HEAD" {
stream
.write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
.await?;
stream.shutdown().await?;
continue;
}
if method == "POST" {
stream
.write_all(b"HTTP/1.1 503 Service Unavailable\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
.await?;
stream.shutdown().await?;
return Ok::<(), BoxError>(());
}
}
});
// Hold the endpoint down long enough for at least one replay attempt to
// fail (the replay worker scans the store every 500ms), so recovery below
// exercises real redelivery rather than a first-attempt success.
tokio::time::sleep(Duration::from_secs(2)).await;
let rejected = match timeout(Duration::from_secs(20), &mut failure_handle).await {
Ok(rejected) => rejected,
Err(_) => {
failure_handle.abort();
let _ = failure_handle.await;
return Err("webhook replay did not reach the rejecting endpoint".into());
}
};
rejected??;
// Bring the endpoint back on the same port; the replay worker rescans the
// durable queue and delivers the retained event.
// Bring the endpoint back on the same port; the replay worker retries with
// exponential backoff and delivers the queued event.
let listener = TcpListener::bind(("0.0.0.0", port)).await?;
let (tx, mut rx) = mpsc::unbounded_channel();
let handle = serve_event_collector(listener, tx);
+14 -29
View File
@@ -12,11 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::common::{RustFSTestEnvironment, admin_request, awscurl_delete, awscurl_get, awscurl_post, awscurl_put, init_logging};
use crate::common::{RustFSTestEnvironment, awscurl_delete, awscurl_get, awscurl_post, awscurl_put, init_logging};
use aws_sdk_s3::Client;
use http::{Method, StatusCode};
use serial_test::serial;
use tokio::time::{Duration, sleep, timeout};
use tracing::{debug, info};
fn skip_without_awscurl() -> bool {
@@ -39,8 +37,7 @@ impl QuotaTestEnv {
pub async fn new() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
let bucket_name = format!("quota-test-{}", uuid::Uuid::new_v4());
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_SCANNER_START_DELAY_SECS", "0")])
.await?;
env.start_rustfs_server(vec![]).await?;
let client = env.create_s3_client();
Ok(Self {
@@ -70,7 +67,18 @@ impl QuotaTestEnv {
}
pub async fn set_bucket_quota(&self, quota_bytes: u64) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
self.set_bucket_quota_for(&self.bucket_name, quota_bytes).await
let url = format!("{}/rustfs/admin/v3/quota/{}", self.env.url, self.bucket_name);
let quota_config = serde_json::json!({
"quota": quota_bytes,
"quota_type": "HARD"
});
let response = awscurl_put(&url, &quota_config.to_string(), &self.env.access_key, &self.env.secret_key).await?;
if response.contains("error") {
Err(format!("Failed to set quota: {}", response).into())
} else {
Ok(())
}
}
pub async fn get_bucket_quota(&self) -> Result<Option<u64>, Box<dyn std::error::Error + Send + Sync>> {
@@ -170,29 +178,6 @@ impl QuotaTestEnv {
bucket: &str,
quota_bytes: u64,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let stats_path = format!("/rustfs/admin/v3/quota-stats/{bucket}");
let readiness = async {
loop {
let (status, response) =
admin_request(&self.env.url, Method::GET, &stats_path, None, &self.env.access_key, &self.env.secret_key)
.await?;
if status.is_success() {
return Ok::<(), Box<dyn std::error::Error + Send + Sync>>(());
}
if status != StatusCode::SERVICE_UNAVAILABLE {
return Err(format!("quota usage readiness failed for {bucket}: {status} {response}").into());
}
sleep(Duration::from_secs(1)).await;
}
};
match timeout(Duration::from_secs(30), readiness).await {
Ok(result) => result?,
Err(_) => {
return Err(format!("quota usage did not become authoritative for {bucket} within 30 seconds").into());
}
}
let url = format!("{}/rustfs/admin/v3/quota/{}", self.env.url, bucket);
let quota_config = serde_json::json!({
"quota": quota_bytes,
+4 -6
View File
@@ -15,7 +15,9 @@
use super::{grpc_lock_client::GrpcLockClient, grpc_lock_server::spawn_lock_server};
use rustfs_lock::client::{LockClient, local::LocalClient};
use rustfs_lock::{GlobalLockManager, LockInfo, LockRequest, LockResponse, LockStats, LockType, NamespaceLock, ObjectKey};
use rustfs_lock::{
GlobalLockManager, LockError, LockInfo, LockRequest, LockResponse, LockStats, LockType, NamespaceLock, ObjectKey,
};
use std::sync::Arc;
use std::time::Duration;
@@ -33,11 +35,7 @@ struct FailingClient;
#[async_trait::async_trait]
impl rustfs_lock::LockClient for FailingClient {
async fn acquire_lock(&self, _request: &rustfs_lock::LockRequest) -> rustfs_lock::Result<LockResponse> {
// Match RemoteClient's transport-failure response so the coordinator can count this node toward quorum loss.
Ok(LockResponse::failure(
"Remote lock RPC failed: simulated gRPC node failure",
Duration::ZERO,
))
Err(LockError::internal("simulated gRPC node failure"))
}
async fn release(&self, _lock_id: &rustfs_lock::LockId) -> rustfs_lock::Result<bool> {
+46 -18
View File
@@ -382,6 +382,7 @@ struct ManualTransitionRunReport {
skipped_delete_marker: u64,
skipped_directory: u64,
skipped_replication: u64,
skipped_already_transitioned: u64,
skipped_already_in_flight: u64,
skipped_queue_full: u64,
skipped_queue_closed: u64,
@@ -407,6 +408,41 @@ fn assert_completed_or_in_flight_partial(state: &str, report: &ManualTransitionR
}
}
fn assert_conflict_winner_report(state: &str, report: &ManualTransitionRunReport, expected_objects: u64, context: &str) {
assert_completed_or_in_flight_partial(state, report, context);
if report.skipped_already_in_flight > 0 {
assert!(
report.scanned <= expected_objects,
"{context}: scanned more objects than the conflict scope contains: {report:#?}"
);
assert!(
report.eligible <= expected_objects,
"{context}: marked more objects eligible than the conflict scope contains: {report:#?}"
);
assert_eq!(
report.enqueued + report.skipped_already_in_flight,
report.eligible,
"{context}: partial in-flight accounting must cover every eligible object: {report:#?}"
);
} else {
assert_eq!(report.scanned, expected_objects, "{context}: {report:#?}");
assert_eq!(
report.eligible + report.skipped_already_transitioned,
expected_objects,
"{context}: {report:#?}"
);
assert_eq!(
report.enqueued + report.skipped_already_in_flight,
expected_objects,
"{context}: {report:#?}"
);
}
assert_eq!(
report.transition_completed, report.enqueued,
"{context}: winner must wait for all queued transitions: {report:#?}"
);
}
#[derive(Debug, Deserialize)]
struct ManualTransitionQueueSnapshot {
queue_capacity: u64,
@@ -1240,15 +1276,8 @@ async fn test_manual_transition_async_scope_conflicts_report_active_job() -> Tes
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
let mut hot = RustFSTestEnvironment::new().await?;
hot.start_rustfs_server_with_env(
vec![],
&[
("RUSTFS_SCANNER_ENABLED", "false"),
("RUSTFS_SCANNER_CYCLE", "3600"),
(MANUAL_TRANSITION_CANCEL_BARRIER_ENV, "1"),
],
)
.await?;
hot.start_rustfs_server_with_env(vec![], &[("RUSTFS_SCANNER_ENABLED", "false"), ("RUSTFS_SCANNER_CYCLE", "3600")])
.await?;
let hot_client = hot.create_s3_client();
add_rustfs_tier(&hot, &cold).await?;
@@ -1318,21 +1347,20 @@ async fn test_manual_transition_async_scope_conflicts_report_active_job() -> Tes
assert_eq!(conflict.cancel_endpoint, status_endpoint);
assert!(!conflict.scope_key.is_empty());
manual_transition_job_cancel(&hot, cancel_endpoint).await?;
let terminal = wait_for_manual_transition_job_terminal(&hot, status_endpoint, MANUAL_ASYNC_CONFLICT_TERMINAL_TIMEOUT).await?;
assert_eq!(terminal.job_id, job_id);
assert_eq!(terminal.status, "cancelled", "terminal conflict winner response: {terminal:#?}");
assert!(!terminal.report.dry_run);
assert_eq!(terminal.report.bucket, MANUAL_ASYNC_CONFLICT_BUCKET);
assert_eq!(terminal.report.prefix, accepted.report.prefix);
assert!(terminal.report.cancelled, "terminal conflict winner response: {terminal:#?}");
assert_eq!(terminal.report.scanned, 0, "terminal conflict winner response: {terminal:#?}");
assert_eq!(terminal.report.enqueued, 0, "terminal conflict winner response: {terminal:#?}");
assert_eq!(
terminal.report.transition_completed, 0,
"terminal conflict winner response: {terminal:#?}"
assert_conflict_winner_report(
&terminal.status,
&terminal.report,
MANUAL_ASYNC_CONFLICT_OBJECTS as u64,
"terminal conflict winner response",
);
assert_eq!(terminal.report.dry_run_eligible, 0, "terminal conflict winner response: {terminal:#?}");
assert_eq!(terminal.report.transition_failed, 0, "terminal conflict winner response: {terminal:#?}");
assert_eq!(terminal.report.tier_failure, 0, "terminal conflict winner response: {terminal:#?}");
let after_remote_count = cold_tier_object_count(&cold_client).await?;
assert!(after_remote_count >= before_remote_count);
assert!(after_remote_count <= before_remote_count + MANUAL_ASYNC_CONFLICT_OBJECTS);
File diff suppressed because it is too large Load Diff
+57 -562
View File
@@ -12,45 +12,69 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::common::{RustFSTestEnvironment, admin_ok, build_test_s3_config, build_test_sts_client, init_logging};
use aws_sdk_sts::Client;
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_sts::config::retry::RetryConfig;
use aws_sdk_sts::config::{Credentials, Region};
use aws_sdk_sts::error::ProvideErrorMetadata;
use aws_sdk_sts::operation::RequestId;
use bytes::Bytes;
use http::header::{AUTHORIZATION, CONTENT_TYPE};
use http::{Request, Response};
use http_body_util::{BodyExt, Full};
use hyper::body::Incoming;
use hyper::server::conn::http1;
use hyper::service::service_fn;
use hyper_util::rt::TokioIo;
use serde_json::Value;
use aws_sdk_sts::{Client, Config};
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use http::header::{CONTENT_TYPE, HOST};
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::collections::BTreeSet;
use std::convert::Infallible;
use std::error::Error;
use tokio::net::TcpListener;
use tokio::sync::mpsc;
use tokio::task::{JoinHandle, JoinSet};
use tokio::time::{Duration, timeout};
type BoxError = Box<dyn Error + Send + Sync>;
type TestResult = Result<(), BoxError>;
const OPA_AUTH_TOKEN: &str = "sts-opa-token";
fn sts_client(url: &str, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
build_test_sts_client(url, access_key, secret_key, session_token, "e2e-sts-query-compat")
let mut config = Config::builder()
.credentials_provider(Credentials::new(
access_key,
secret_key,
session_token.map(str::to_owned),
None,
"e2e-sts-query-compat",
))
.region(Region::new("us-east-1"))
.endpoint_url(url)
.retry_config(RetryConfig::standard().with_max_attempts(1))
.behavior_version_latest();
if url.starts_with("http://") {
config = config.http_client(SmithyHttpClientBuilder::new().build_http());
}
Client::from_conf(config.build())
}
async fn create_root_service_account(env: &RustFSTestEnvironment) -> Result<(String, String), BoxError> {
let body = admin_ok(
env,
http::Method::PUT,
"/rustfs/admin/v3/add-service-accounts",
Some(serde_json::json!({ "targetUser": env.access_key.clone() }).to_string()),
)
.await?;
let response: Value = serde_json::from_str(&body)?;
let path = "/rustfs/admin/v3/add-service-accounts";
let url = format!("{}{path}", env.url);
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
let body = serde_json::json!({ "targetUser": env.access_key.clone() }).to_string();
let request = http::Request::builder()
.method(http::Method::PUT)
.uri(uri)
.header(HOST, authority)
.header(CONTENT_TYPE, "application/json")
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
.body(Body::empty())?;
let content_length = i64::try_from(body.len()).map_err(|_| "service account request body is too large")?;
let signed = sign_v4(request, content_length, &env.access_key, &env.secret_key, "", "us-east-1");
let mut request = local_http_client().put(&url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
let response = request.body(body).send().await?;
let status = response.status();
let body = response.text().await?;
if !status.is_success() {
return Err(format!("create service account failed: {status} {body}").into());
}
let response: serde_json::Value = serde_json::from_str(&body)?;
let access_key = response["credentials"]["accessKey"]
.as_str()
.ok_or("service account response should contain credentials.accessKey")?
@@ -62,293 +86,28 @@ async fn create_root_service_account(env: &RustFSTestEnvironment) -> Result<(Str
Ok((access_key, secret_key))
}
async fn create_user_with_policy(
env: &RustFSTestEnvironment,
user: &str,
secret: &str,
policy_name: &str,
statements: Value,
) -> TestResult {
create_user(env, user, secret).await?;
admin_ok(
env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-canned-policy?name={policy_name}"),
Some(
serde_json::json!({
"Version": "2012-10-17",
"Statement": statements,
})
.to_string(),
),
)
.await?;
admin_ok(
env,
http::Method::POST,
"/rustfs/admin/v3/idp/builtin/policy/attach",
Some(serde_json::json!({ "policies": [policy_name], "user": user }).to_string()),
)
.await?;
Ok(())
}
async fn create_user(env: &RustFSTestEnvironment, user: &str, secret: &str) -> TestResult {
admin_ok(
env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-user?accessKey={user}"),
Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
)
.await?;
Ok(())
}
async fn assert_access_denied(client: &Client, context: &str) -> TestResult {
async fn assert_chaining_denied(client: &Client, credential_kind: &str) -> TestResult {
let error = client
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/test")
.role_session_name("sts-query-compat-e2e")
.send()
.await
.expect_err("AssumeRole must be denied");
.expect_err("credential chaining must be denied");
let service_error = error
.as_service_error()
.ok_or_else(|| format!("{context} should deserialize as an STS service error: {error:?}"))?;
.ok_or_else(|| format!("{credential_kind} denial should deserialize as an STS service error: {error:?}"))?;
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(403));
assert_eq!(service_error.code(), Some("AccessDenied"));
assert_eq!(service_error.message(), Some("Access Denied"));
assert!(
error.request_id().is_some_and(|request_id| !request_id.is_empty()),
"{context} should include a request ID"
"{credential_kind} denial should include a request ID"
);
Ok(())
}
async fn assert_list_buckets_access_denied(
env: &RustFSTestEnvironment,
access_key: &str,
secret_key: &str,
context: &str,
) -> TestResult {
let error = aws_sdk_s3::Client::from_conf(build_test_s3_config(
&env.url,
access_key,
secret_key,
None,
"e2e-list-buckets-opa-unavailable",
))
.list_buckets()
.send()
.await
.expect_err("ListBuckets must be denied while OPA is unavailable");
let service_error = error
.as_service_error()
.ok_or_else(|| format!("{context} should deserialize as an S3 service error: {error:?}"))?;
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(403));
assert_eq!(service_error.code(), Some("AccessDenied"));
Ok(())
}
async fn assert_opa_unavailable_denies_sts_and_list_buckets(env: &RustFSTestEnvironment, context: &str) -> TestResult {
let user = "opaunavailable";
let secret = "stsOpaUnavailableSecret123";
create_user_with_policy(
env,
user,
secret,
"sts-opa-unavailable-local-policy",
serde_json::json!([{
"Effect": "Allow",
"Action": ["s3:ListAllMyBuckets"],
"Resource": ["arn:aws:s3:::*"],
}]),
)
.await?;
assert_access_denied(&sts_client(&env.url, user, secret, None), context).await?;
assert_list_buckets_access_denied(env, user, secret, context).await
}
async fn handle_opa_request(
request: Request<Incoming>,
requests: mpsc::UnboundedSender<Value>,
validation_started: mpsc::UnboundedSender<()>,
validation_mode: OpaValidationMode,
expected_authorization: Option<String>,
) -> Result<Response<Full<Bytes>>, Infallible> {
if let Some(expected_authorization) = expected_authorization
&& request.headers().get(AUTHORIZATION).and_then(|value| value.to_str().ok()) != Some(expected_authorization.as_str())
{
return Ok(Response::builder()
.status(401)
.body(Full::new(Bytes::new()))
.expect("static OPA unauthorized response must be valid"));
}
let body = match request.into_body().collect().await {
Ok(body) => body.to_bytes(),
Err(error) => {
return Ok(Response::builder()
.status(400)
.body(Full::new(Bytes::from(error.to_string())))
.expect("static OPA error response must be valid"));
}
};
let payload = if body.is_empty() {
None
} else {
match serde_json::from_slice::<Value>(&body) {
Ok(payload) => Some(payload),
Err(error) => {
return Ok(Response::builder()
.status(400)
.body(Full::new(Bytes::from(error.to_string())))
.expect("static OPA error response must be valid"));
}
}
};
if payload.is_none() {
let _ = validation_started.send(());
match validation_mode {
OpaValidationMode::Blocked => std::future::pending::<()>().await,
OpaValidationMode::Unavailable => {
return Ok(Response::builder()
.status(503)
.body(Full::new(Bytes::new()))
.expect("static OPA unavailable response must be valid"));
}
OpaValidationMode::Ready => {}
}
}
let allow = match payload.as_ref().and_then(|value| value.pointer("/input/identity/account")) {
Some(Value::String(account)) if account == "opaallow" => payload
.as_ref()
.and_then(|value| value.pointer("/input/context/deny_only"))
.and_then(Value::as_bool)
.unwrap_or(false),
Some(Value::String(account)) if account == "opadeny" => false,
Some(Value::String(account))
if account == "opaunavailable" && matches!(validation_mode, OpaValidationMode::Unavailable) =>
{
true
}
Some(Value::String(account)) if account == "opalistbuckets" => {
let action = payload
.as_ref()
.and_then(|value| value.pointer("/input/action"))
.and_then(Value::as_str);
let bucket = payload
.as_ref()
.and_then(|value| value.pointer("/input/resource/bucket"))
.and_then(Value::as_str);
matches!(
(action, bucket),
(Some("s3:ListBucket"), Some("opa-list-visible")) | (Some("s3:GetBucketLocation"), Some("opa-list-location"))
)
}
None => true,
_ => false,
};
if let Some(payload) = payload {
let _ = requests.send(payload);
}
let body =
serde_json::to_vec(&serde_json::json!({ "result": { "allow": allow } })).expect("static OPA response must serialize");
Ok(Response::builder()
.header(CONTENT_TYPE, "application/json")
.body(Full::new(Bytes::from(body)))
.expect("static OPA response must be valid"))
}
#[derive(Clone, Copy)]
enum OpaValidationMode {
Ready,
Blocked,
Unavailable,
}
struct OpaMock {
url: String,
requests: mpsc::UnboundedReceiver<Value>,
validation_started: mpsc::UnboundedReceiver<()>,
task: JoinHandle<()>,
}
impl OpaMock {
async fn start() -> Result<Self, BoxError> {
Self::start_with_mode(OpaValidationMode::Ready, Some(OPA_AUTH_TOKEN)).await
}
async fn start_blocked() -> Result<Self, BoxError> {
Self::start_with_mode(OpaValidationMode::Blocked, None).await
}
async fn start_with_mode(validation_mode: OpaValidationMode, auth_token: Option<&str>) -> Result<Self, BoxError> {
let listener = TcpListener::bind("127.0.0.1:0").await?;
let url = format!("http://{}/v1/data/rustfs/authz/allow", listener.local_addr()?);
let (requests_tx, requests) = mpsc::unbounded_channel();
let (validation_started_tx, validation_started) = mpsc::unbounded_channel();
let expected_authorization = auth_token.map(|token| format!("Bearer {token}"));
let task = tokio::spawn(async move {
let mut connections = JoinSet::new();
loop {
tokio::select! {
accepted = listener.accept() => {
let Ok((stream, _)) = accepted else { break };
let requests = requests_tx.clone();
let validation_started = validation_started_tx.clone();
let validation_mode = validation_mode;
let expected_authorization = expected_authorization.clone();
connections.spawn(async move {
let handler = service_fn(move |request| {
handle_opa_request(
request,
requests.clone(),
validation_started.clone(),
validation_mode,
expected_authorization.clone(),
)
});
let _ = http1::Builder::new()
.serve_connection(TokioIo::new(stream), handler)
.await;
});
}
_ = connections.join_next(), if !connections.is_empty() => {}
}
}
});
Ok(Self {
url,
requests,
validation_started,
task,
})
}
async fn next_request(&mut self) -> Result<Value, BoxError> {
timeout(Duration::from_secs(5), self.requests.recv())
.await?
.ok_or_else(|| "OPA request channel closed".into())
}
async fn wait_for_validation(&mut self) -> TestResult {
timeout(Duration::from_secs(5), self.validation_started.recv())
.await?
.ok_or_else(|| "OPA validation channel closed".into())
}
}
impl Drop for OpaMock {
fn drop(&mut self) {
self.task.abort();
}
}
#[tokio::test]
#[serial]
async fn test_sts_query_responses_are_aws_sdk_compatible() -> TestResult {
@@ -396,7 +155,7 @@ async fn test_sts_query_responses_are_aws_sdk_compatible() -> TestResult {
"signature rejection should include a request ID"
);
assert_access_denied(
assert_chaining_denied(
&sts_client(
&env.url,
temporary.access_key_id(),
@@ -408,271 +167,7 @@ async fn test_sts_query_responses_are_aws_sdk_compatible() -> TestResult {
.await?;
let (service_access_key, service_secret_key) = create_root_service_account(&env).await?;
assert_access_denied(
&sts_client(&env.url, &service_access_key, &service_secret_key, None),
"service-account denial",
)
.await?;
let implicit_user = "stsimplicit";
let explicit_allow_user = "stsallow";
let explicit_deny_user = "stsdeny";
let policyless_user = "stspolicyless";
let secret = "stsAuthzSecret123";
create_user(&env, policyless_user, secret).await?;
assert_access_denied(&sts_client(&env.url, policyless_user, secret, None), "policyless user").await?;
create_user_with_policy(
&env,
implicit_user,
secret,
"sts-implicit-policy",
serde_json::json!([{
"Effect": "Allow",
"Action": ["s3:ListAllMyBuckets"],
"Resource": ["arn:aws:s3:::*"],
}]),
)
.await?;
create_user_with_policy(
&env,
explicit_allow_user,
secret,
"sts-allow-policy",
serde_json::json!([{
"Effect": "Allow",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"],
}]),
)
.await?;
create_user_with_policy(
&env,
explicit_deny_user,
secret,
"sts-deny-policy",
serde_json::json!([
{
"Effect": "Allow",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"],
},
{
"Effect": "Deny",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"],
}
]),
)
.await?;
for user in [implicit_user, explicit_allow_user] {
let output = sts_client(&env.url, user, secret, None)
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/test")
.role_session_name("sts-authz-e2e")
.send()
.await
.map_err(|error| format!("{user} should be allowed to call AssumeRole: {error:?}"))?;
let credentials = output
.credentials()
.ok_or_else(|| format!("{user} AssumeRole response should contain credentials"))?;
assert!(!credentials.access_key_id().is_empty());
assert!(!credentials.secret_access_key().is_empty());
assert!(!credentials.session_token().is_empty());
}
assert_access_denied(&sts_client(&env.url, explicit_deny_user, secret, None), "explicit sts:AssumeRole Deny").await?;
env.stop_server();
Ok(())
}
#[tokio::test]
#[serial]
async fn test_sts_assume_role_opa_contract() -> TestResult {
init_logging();
let mut opa = OpaMock::start().await?;
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(
vec![],
&[
("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
],
)
.await?;
let secret = "stsOpaSecret123";
create_user_with_policy(
&env,
"opaallow",
secret,
"sts-opa-local-deny-policy",
serde_json::json!([{
"Effect": "Deny",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"],
}]),
)
.await?;
create_user_with_policy(
&env,
"opadeny",
secret,
"sts-opa-local-allow-policy",
serde_json::json!([{
"Effect": "Allow",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"],
}]),
)
.await?;
sts_client(&env.url, "opaallow", secret, None)
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/test")
.role_session_name("sts-opa-contract")
.send()
.await
.map_err(|error| format!("OPA allow should override the local explicit Deny: {error:?}"))?;
assert_access_denied(
&sts_client(&env.url, "opadeny", secret, None),
"OPA denial despite local sts:AssumeRole Allow",
)
.await?;
let mut accounts = BTreeSet::new();
for _ in 0..2 {
let request = opa.next_request().await?;
assert_eq!(request.pointer("/input/action").and_then(Value::as_str), Some("sts:AssumeRole"));
assert_eq!(request.pointer("/input/context/deny_only").and_then(Value::as_bool), Some(true));
let account = request
.pointer("/input/identity/account")
.and_then(Value::as_str)
.ok_or("OPA input should include identity.account")?;
accounts.insert(account.to_owned());
}
assert_eq!(accounts, BTreeSet::from(["opaallow".to_owned(), "opadeny".to_owned()]));
env.stop_server();
Ok(())
}
#[tokio::test]
#[serial]
async fn test_list_buckets_opa_contract() -> TestResult {
init_logging();
let mut opa = OpaMock::start().await?;
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(
vec![],
&[
("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
],
)
.await?;
let admin_client = env.create_s3_client();
for bucket in ["opa-list-hidden", "opa-list-location", "opa-list-visible"] {
admin_client.create_bucket().bucket(bucket).send().await?;
}
let user = "opalistbuckets";
let secret = "opaListBucketsSecret123";
create_user(&env, user, secret).await?;
let output = aws_sdk_s3::Client::from_conf(build_test_s3_config(&env.url, user, secret, None, "e2e-list-buckets-opa"))
.list_buckets()
.send()
.await?;
let mut names = output
.buckets()
.iter()
.filter_map(|bucket| bucket.name().map(str::to_owned))
.collect::<Vec<_>>();
names.sort();
assert_eq!(names, ["opa-list-location", "opa-list-visible"]);
let mut evaluations = BTreeSet::new();
for _ in 0..6 {
let request = opa.next_request().await?;
assert_eq!(request.pointer("/input/identity/account").and_then(Value::as_str), Some(user));
assert_eq!(request.pointer("/input/context/deny_only").and_then(Value::as_bool), Some(false));
let action = request
.pointer("/input/action")
.and_then(Value::as_str)
.ok_or("OPA ListBuckets input should include action")?;
let bucket = request
.pointer("/input/resource/bucket")
.and_then(Value::as_str)
.ok_or("OPA ListBuckets input should include resource.bucket")?;
if bucket.is_empty() {
assert_eq!(action, "s3:ListAllMyBuckets");
assert!(request.pointer("/input/context/conditions/prefix").is_none());
assert!(request.pointer("/input/context/conditions/delimiter").is_none());
} else {
let expected_arn = format!("arn:aws:s3:::{bucket}");
assert_eq!(request.pointer("/input/context/conditions/prefix"), Some(&serde_json::json!([""])));
assert_eq!(request.pointer("/input/context/conditions/delimiter"), Some(&serde_json::json!(["/"])));
assert_eq!(
request.pointer("/input/resource/arn").and_then(Value::as_str),
Some(expected_arn.as_str())
);
}
evaluations.insert((action.to_owned(), bucket.to_owned()));
}
assert_eq!(
evaluations,
BTreeSet::from([
("s3:GetBucketLocation".to_owned(), "opa-list-hidden".to_owned()),
("s3:GetBucketLocation".to_owned(), "opa-list-location".to_owned()),
("s3:ListAllMyBuckets".to_owned(), String::new()),
("s3:ListBucket".to_owned(), "opa-list-hidden".to_owned()),
("s3:ListBucket".to_owned(), "opa-list-location".to_owned()),
("s3:ListBucket".to_owned(), "opa-list-visible".to_owned()),
])
);
assert!(
matches!(opa.requests.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
"ListBuckets should not make redundant OPA evaluations"
);
env.stop_server();
Ok(())
}
#[tokio::test]
#[serial]
async fn test_sts_and_list_buckets_fail_closed_while_opa_is_initializing() -> TestResult {
init_logging();
let mut opa = OpaMock::start_blocked().await?;
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str())])
.await?;
opa.wait_for_validation().await?;
assert_opa_unavailable_denies_sts_and_list_buckets(&env, "configured OPA initialization").await?;
env.stop_server();
Ok(())
}
#[tokio::test]
#[serial]
async fn test_sts_and_list_buckets_fail_closed_after_opa_validation_failure() -> TestResult {
init_logging();
let mut opa = OpaMock::start_with_mode(OpaValidationMode::Unavailable, None).await?;
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str())])
.await?;
opa.wait_for_validation().await?;
assert_opa_unavailable_denies_sts_and_list_buckets(&env, "configured OPA validation failure").await?;
assert_chaining_denied(&sts_client(&env.url, &service_access_key, &service_secret_key, None), "service account").await?;
env.stop_server();
Ok(())
@@ -1,172 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression tests for Tier/ILM transition operations.
//!
//! Covers the recurring pattern where tier transition fails silently, the
//! free-version recovery task loops forever, or transitioned objects cannot
//! be read back. This has regressed 6+ times.
//!
//! ## Regression Issues
//!
//! - rustfs#5218: Remote tier mutation commit failed
//! - rustfs#5130: tier_free_version_recovery task loops forever
//! - rustfs#5011: Idle tier free-version recovery rescans every 60 seconds
//! - rustfs#4826: Full GET of multipart transitioned object fails
//! - rustfs#5024: Some files succeeded in tier offloading, others failed
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, admin_ok, init_logging};
use serde_json::Value;
use serial_test::serial;
use std::error::Error;
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
/// RT-13: Verify lifecycle rule with transition persists and is retrievable.
///
/// Note: Actual transition requires a configured remote tier. This test
/// validates that an expiration-only rule (the persistence path) survives
/// a server restart.
#[tokio::test]
#[serial]
async fn test_lifecycle_rule_persists_after_restart() -> TestResult {
init_logging();
info!("RT-13: lifecycle rule persists after restart");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "rt13-tier-persist";
client.create_bucket().bucket(bucket).send().await.expect("create bucket");
// Apply a lifecycle rule with expiration (transition needs a real tier)
let rule = aws_sdk_s3::types::LifecycleRule::builder()
.id("expire-after-90d")
.status(aws_sdk_s3::types::ExpirationStatus::Enabled)
.filter(aws_sdk_s3::types::LifecycleRuleFilter::builder().prefix("archive/").build())
.expiration(aws_sdk_s3::types::LifecycleExpiration::builder().days(90).build())
.build()
.expect("build rule");
client
.put_bucket_lifecycle_configuration()
.bucket(bucket)
.lifecycle_configuration(
aws_sdk_s3::types::BucketLifecycleConfiguration::builder()
.rules(rule)
.build()
.expect("build config"),
)
.send()
.await
.expect("put lifecycle");
// Restart server
env.restart_server_preserving_data(vec![], &[]).await.expect("restart RustFS");
// Verify the rule survived restart
let resp = client
.get_bucket_lifecycle_configuration()
.bucket(bucket)
.send()
.await
.expect("get lifecycle after restart");
let rules = resp.rules();
assert_eq!(rules.len(), 1, "RT-13 FAIL: expected 1 rule after restart");
let exp = rules[0].expiration().expect("expiration should be set");
assert_eq!(exp.days(), Some(90), "RT-13 FAIL: expiration days corrupted after restart");
info!("RT-13 PASS: lifecycle rule persists after restart");
Ok(())
}
/// RT-13b: Verify admin tier configuration API is functional.
///
/// Regression pattern: tier add/verify/delete API fails or the tier
/// configuration is not persisted (rustfs#5218).
#[tokio::test]
#[serial]
async fn test_admin_tier_list_endpoint_returns_json() -> TestResult {
init_logging();
info!("RT-13b: admin tier list endpoint returns JSON");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
// Query the tier list endpoint
let body = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/tier", None)
.await
.expect("list remote tiers");
let json: Value = serde_json::from_str(&body).expect("tier list response should be valid JSON");
// Should return an array (possibly empty)
assert!(json.is_array(), "RT-13b FAIL: tier list response is not an array: {json}");
info!("RT-13b PASS: admin tier list endpoint returns valid JSON array");
Ok(())
}
/// RT-13c: Verify scanner configuration persistence.
///
/// Regression pattern: scanner admin config update reports success but
/// is not persisted (rustfs#5013), causing the scanner to not run or
/// use stale settings.
#[tokio::test]
#[serial]
async fn test_scanner_config_persists_after_restart() -> TestResult {
init_logging();
info!("RT-13c: scanner config persists after restart");
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
.await
.expect("start RustFS");
// Get current scanner status
let body = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/scanner/status", None)
.await
.expect("get scanner status");
let json: Value = serde_json::from_str(&body).expect("scanner status should be valid JSON");
info!(" scanner status: {:?}", json.as_object().map(|o| o.keys().collect::<Vec<_>>()));
// Restart and verify config is still accessible
env.restart_server_preserving_data(vec![], &[]).await.expect("restart RustFS");
let body2 = admin_ok(&env, http::Method::GET, "/rustfs/admin/v3/scanner/status", None)
.await
.expect("get scanner status after restart");
let json2: Value = serde_json::from_str(&body2).expect("scanner status after restart should be valid JSON");
// Both should be valid JSON objects
assert!(json2.is_object(), "RT-13c FAIL: scanner status after restart is not a valid JSON object");
info!("RT-13c PASS: scanner/config persists across restart");
Ok(())
}
}
-2
View File
@@ -144,13 +144,11 @@ rustfs-lifecycle.workspace = true
rustfs-s3-types = { workspace = true }
rustfs-data-usage.workspace = true
rustfs-object-capacity.workspace = true
rustfs-object-data-cache = { workspace = true, features = ["runtime-memory"] }
arc-swap.workspace = true
async-trait.workspace = true
bytes = { workspace = true, features = ["serde"] }
byteorder = { workspace = true }
chrono = { workspace = true, features = ["serde"] }
jiff = { workspace = true, features = ["serde"] }
glob = { workspace = true }
thiserror.workspace = true
flatbuffers.workspace = true
+31 -47
View File
@@ -130,15 +130,13 @@ pub mod bucket {
pub mod metadata_sys {
pub use crate::bucket::metadata_sys::{
BucketMetadataMutationGuard, BucketMetadataSys, ObjectLockConfigState, acquire_bucket_metadata_transaction_lock,
capture_bucket_metadata_incarnation, delete, delete_if_incarnation, get, get_accelerate_config, get_bucket_policy,
BucketMetadataSys, acquire_bucket_metadata_transaction_lock, delete, get, get_accelerate_config, get_bucket_policy,
get_bucket_policy_raw, get_bucket_targets_config, get_config_from_disk, get_cors_config, get_durability_config,
get_global_bucket_metadata_sys, get_lifecycle_config, get_logging_config, get_notification_config,
get_object_lock_config, get_object_lock_config_state, get_public_access_block_config, get_quota_config,
get_replication_config, get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config,
get_website_config, init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
set_bucket_metadata, update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
update_under_transaction_lock,
get_object_lock_config, get_public_access_block_config, get_quota_config, get_replication_config,
get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
update, update_bucket_targets_under_transaction_lock, update_config_with, update_under_transaction_lock,
};
}
@@ -174,30 +172,20 @@ pub mod bucket {
}
pub mod replication {
pub use crate::bucket::replication::replication_pool::{
DurableMrfBacklogSummary, DurableMrfBucketBacklog, DurableMrfTargetBacklog, MrfBacklogObservabilitySummary,
MrfBucketBacklogObservability, durable_mrf_backlog_summary_snapshot, durable_mrf_target_backlog_snapshot,
mrf_backlog_observability_snapshot,
};
pub use crate::bucket::replication::{
BucketReplicationResyncStatus, BucketReplicationStats, BucketStats, DeleteReplicationConfigSnapshot,
DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool, MrfOpKind, MrfReplicateEntry,
MustReplicateOptions, ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS,
REMOTE_TARGET_WRITABLE_FIELDS, REPLICATE_INCOMING_DELETE, REPLICATION_CAPABILITY_CONTRACT_VERSION,
REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS, ReplicateDecision, ReplicateObjectInfo,
ReplicationBatchAdmission, ReplicationConfig, ReplicationConfigStructureError, ReplicationConfigurationExt,
ReplicationDeleteScheduleInput, ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge,
ReplicationObjectIO, ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission,
ReplicationScannerBridge, ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage,
ReplicationTargetValidationError, ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog,
TargetReplicationResyncStatus, VersionPurgeStatusType, commit_force_delete_intent, complete_force_delete_intent,
BucketReplicationResyncStatus, BucketStats, DeletedObjectReplicationInfo, DurableMrfBacklog, DynReplicationPool,
MrfOpKind, MrfReplicateEntry, MustReplicateOptions, ObjectOpts, REPLICATE_INCOMING_DELETE, ReplicateDecision,
ReplicateObjectInfo, ReplicationConfig, ReplicationConfigurationExt, ReplicationDeleteScheduleInput,
ReplicationDeleteStateSource, ReplicationHealQueueResult, ReplicationObjectBridge, ReplicationObjectIO,
ReplicationOperation, ReplicationPoolTrait, ReplicationPriority, ReplicationQueueAdmission, ReplicationScannerBridge,
ReplicationState, ReplicationStats, ReplicationStatusType, ReplicationStorage, ReplicationTargetValidationError,
ReplicationType, ResyncOpts, ResyncStatusType, TargetReplicationResyncStatus, VersionPurgeStatusType,
delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
get_global_replication_stats, init_background_replication, invalid_replication_config_status_field,
persist_force_delete_intent, read_durable_mrf_backlog, replication_state_to_filemeta, replication_status_to_filemeta,
replication_statuses_map, replication_target_arns, resync_start_conflict_id, should_remove_replication_target,
should_schedule_delete_replication, should_use_existing_delete_replication_info,
should_use_existing_delete_replication_source, unsupported_replication_config_field,
validate_replication_config_structure, validate_replication_config_target_arns, version_purge_status_to_filemeta,
get_global_replication_stats, init_background_replication, read_durable_mrf_backlog, replication_state_to_filemeta,
replication_status_to_filemeta, replication_statuses_map, replication_target_arns, resync_start_conflict_id,
should_remove_replication_target, should_schedule_delete_replication, should_use_existing_delete_replication_info,
should_use_existing_delete_replication_source, validate_replication_config_target_arns,
version_purge_status_to_filemeta,
};
}
@@ -206,9 +194,7 @@ pub mod bucket {
}
pub mod target {
pub use crate::bucket::target::{
ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials, LatencyStat, duration_from_secs_or_nanos,
};
pub use crate::bucket::target::{ARN, BucketTarget, BucketTargetType, BucketTargets, Credentials, LatencyStat};
}
pub mod utils {
@@ -310,8 +296,7 @@ pub mod config {
pub mod data_usage {
pub use crate::data_usage::{
DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
init_compression_total_memory_from_backend, invalidate_data_usage_snapshot_cache, live_bucket_usage_computations,
load_compression_total_from_memory, load_data_usage_from_backend, load_data_usage_from_backend_cached,
record_bucket_delete_marker_memory, record_bucket_object_delete_memory, record_bucket_object_version_write_memory,
record_bucket_object_write_memory, record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
@@ -408,11 +393,11 @@ pub mod notification {
pub mod object {
pub use crate::object_api::{
BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader,
ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest, StreamConsumer, get_object_body_cache_plaintext_len,
lookup_get_object_body_cache_hook, register_get_object_body_cache_hook, register_object_mutation_hook,
unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, ObjectEncryptionResolver, ObjectInfo,
ObjectMutationHook, ObjectOptions, PutObjReader, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode,
ReadEncryptionRequest, StreamConsumer, get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook,
register_get_object_body_cache_hook, register_object_mutation_hook, unregister_get_object_body_cache_hook,
unregister_object_mutation_hook,
};
pub use crate::store::PreparedGetObjectReader;
}
@@ -434,14 +419,13 @@ pub mod rio {
pub mod rpc {
pub use crate::cluster::rpc::{
AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS,
PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC,
ScannerBucketListing, ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer,
check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth,
normalize_tonic_rpc_audience, set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_tonic_rpc_response_proof,
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason,
verify_put_file_auth_trailer, verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
AuthenticatedChannel, LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient,
PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing,
ScannerPeerActivity, TONIC_RPC_PREFIX, TonicInterceptor, gen_signature_headers, gen_tonic_replay_scope_headers,
gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client,
node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, set_tonic_canonical_body_digest,
sign_ns_scanner_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_rpc_response_proof, verify_tonic_rpc_signature,
verify_tonic_rpc_signature_with_bootstrap,
};
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -12,127 +12,20 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use std::sync::Arc;
use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration};
use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration, ReplicationConfiguration};
use time::OffsetDateTime;
use uuid::Uuid;
use crate::bucket::metadata_sys::{self, ObjectLockConfigState};
use crate::error::{Error, Result};
#[derive(Debug)]
pub(crate) struct LifecycleExpiryConfigs {
pub(crate) lifecycle: Option<Arc<BucketLifecycleConfiguration>>,
pub(crate) object_lock: Option<Arc<ObjectLockConfiguration>>,
pub(crate) bucket_incarnation_id: Uuid,
}
pub(crate) async fn get_expiry_configs(api: &crate::store::ECStore, bucket: &str) -> Result<LifecycleExpiryConfigs> {
let bucket_incarnation_id = api.bucket_incarnation_id_from_disk(bucket).await?;
let sys = metadata_sys::bucket_metadata_sys_of(&api.ctx)?;
let sys = sys.read().await.clone();
let metadata = sys.get_authoritative_metadata(bucket).await?;
if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id != bucket_incarnation_id {
return Err(Error::other(format!("bucket lifecycle metadata is not authoritative: {bucket}")));
}
let lifecycle = if metadata.lifecycle_config.is_none() && !metadata.lifecycle_config_xml.is_empty() {
return Err(Error::other("persisted bucket lifecycle configuration is invalid"));
} else {
metadata
.lifecycle_config
.clone()
.filter(|config| !config.rules.is_empty())
.map(Arc::new)
};
if lifecycle.is_none() {
return Ok(LifecycleExpiryConfigs {
lifecycle: None,
object_lock: None,
bucket_incarnation_id,
});
}
let object_lock = match metadata_sys::object_lock_config_state_from_authoritative_metadata(&metadata)? {
ObjectLockConfigState::Configured { config, .. } => Some(Arc::new(config)),
ObjectLockConfigState::ConfirmedAbsent => None,
ObjectLockConfigState::Fabricated => {
return Err(Error::other(format!("bucket Object Lock metadata is not authoritative: {bucket}")));
}
};
Ok(LifecycleExpiryConfigs {
lifecycle,
object_lock,
bucket_incarnation_id,
})
}
use crate::bucket::metadata_sys;
use crate::error::Result;
pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
metadata_sys::get_lifecycle_config(bucket).await
}
#[cfg(test)]
mod tests {
use super::*;
use crate::bucket::metadata::BucketMetadata;
use crate::bucket::metadata_sys::{self, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::MakeBucketOptions;
use s3s::dto::{ExpirationStatus, LifecycleExpiration, LifecycleRule};
use serial_test::serial;
fn lifecycle_config() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(1),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("expire".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
}],
}
}
#[tokio::test]
#[serial]
async fn expiry_configs_are_resolved_from_the_owning_store() {
let (_dirs_a, store_a) = isolated_store_over_temp_disks().await;
let (_dirs_b, store_b) = isolated_store_over_temp_disks().await;
let bucket = "same-name-expiry-config";
store_a
.peer_sys
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.unwrap();
store_b
.peer_sys
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.unwrap();
metadata_sys::init_bucket_metadata_sys(store_a.clone(), vec![bucket.to_string()]).await;
metadata_sys::init_bucket_metadata_sys(store_b.clone(), vec![bucket.to_string()]).await;
let mut metadata = BucketMetadata::new(bucket);
let lifecycle = lifecycle_config();
metadata.lifecycle_config_xml = crate::bucket::utils::serialize(&lifecycle).unwrap();
metadata.lifecycle_config = Some(lifecycle);
metadata_sys::set_new_bucket_metadata_in(&store_a.ctx, metadata)
.await
.unwrap();
metadata_sys::set_new_bucket_metadata_in(&store_b.ctx, BucketMetadata::new(bucket))
.await
.unwrap();
assert!(get_expiry_configs(&store_a, bucket).await.unwrap().lifecycle.is_some());
assert!(get_expiry_configs(&store_b, bucket).await.unwrap().lifecycle.is_none());
}
pub(crate) async fn get_object_lock_config(bucket: &str) -> Result<(ObjectLockConfiguration, OffsetDateTime)> {
metadata_sys::get_object_lock_config(bucket).await
}
pub(crate) async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
metadata_sys::get_replication_config(bucket).await
}
@@ -19,7 +19,6 @@ pub mod core;
pub mod evaluator;
pub mod manual_transition_job;
mod metadata_boundary;
pub(crate) use metadata_boundary::get_expiry_configs;
mod object_lock_boundary;
pub use self::core as lifecycle;
mod replication_sink;
@@ -21,12 +21,12 @@ pub(crate) fn is_object_locked_by_metadata(user_defined: &HashMap<String, String
rustfs_lifecycle::object_lock::is_object_locked_by_metadata(user_defined, is_delete_marker)
}
pub(crate) fn check_object_lock_for_deletion_with_config(
config: Option<&s3s::dto::ObjectLockConfiguration>,
pub(crate) async fn check_object_lock_for_deletion(
bucket: &str,
obj_info: &ObjectInfo,
bypass_governance: bool,
) -> crate::error::Result<Option<ObjectLockBlockReason>> {
objectlock_sys::check_object_lock_for_deletion_with_config(config, obj_info, bypass_governance)
) -> Option<ObjectLockBlockReason> {
objectlock_sys::check_object_lock_for_deletion(bucket, obj_info, bypass_governance).await
}
#[cfg(test)]
@@ -15,14 +15,15 @@
use rustfs_common::metrics::IlmAction;
use crate::bucket::lifecycle::lifecycle::ObjectOpts;
pub(crate) use crate::bucket::replication::ReplicationStatusType;
#[cfg(test)]
pub(crate) use crate::bucket::replication::VersionPurgeStatusType;
pub(crate) use crate::bucket::replication::ReplicateTargetDecision;
pub(crate) use crate::bucket::replication::{
DeleteReplicationConfigSnapshot, ReplicationObjectBridge, replication_state_to_filemeta,
ReplicateDecision, ReplicationState, ReplicationStatusType, VersionPurgeStatusType, replication_state_to_filemeta,
replication_statuses_map, version_purge_statuses_map,
};
use crate::bucket::replication::{ReplicationLifecycleBridge, ReplicationLifecycleConfig};
use crate::storage_api_contracts::object::DeletedObject;
use crate::object_api::{ObjectInfo, ObjectOptions};
use crate::storage_api_contracts::object::{DeletedObject, ObjectToDelete};
pub(crate) type LifecycleReplicationConfig = ReplicationLifecycleConfig;
@@ -56,6 +57,15 @@ pub(crate) fn lifecycle_action_waits_for_replication(action: IlmAction) -> bool
)
}
pub(crate) async fn check_delete_replication(
bucket: &str,
object: ObjectToDelete,
source: &ObjectInfo,
opts: &ObjectOptions,
) -> ReplicateDecision {
ReplicationLifecycleBridge::check_delete_replication(bucket, &object, source, opts).await
}
pub(crate) async fn schedule_delete(bucket: String, delete_object: DeletedObject) {
ReplicationLifecycleBridge::schedule_delete(bucket, delete_object).await;
}
@@ -64,16 +74,7 @@ pub(crate) async fn schedule_delete(bucket: String, delete_object: DeletedObject
mod tests {
use std::collections::HashMap;
use crate::bucket::replication::{DeleteReplicationConfigSnapshot, ReplicationObjectBridge};
use crate::object_api::{ObjectInfo, ObjectOptions};
use crate::storage_api_contracts::object::ObjectToDelete;
use rustfs_common::metrics::IlmAction;
use s3s::dto::{
BucketVersioningStatus, DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication,
DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus,
VersioningConfiguration,
};
use uuid::Uuid;
use super::*;
@@ -138,97 +139,4 @@ mod tests {
assert!(lifecycle_action_waits_for_replication(IlmAction::TransitionVersionAction));
assert!(!lifecycle_action_waits_for_replication(IlmAction::NoneAction));
}
#[test]
fn lifecycle_delete_admission_uses_marker_and_version_switches_for_all_purges() {
for marker_enabled in [false, true] {
for purge_enabled in [false, true] {
let snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
..Default::default()
},
Some(ReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
delete_marker_replication: Some(DeleteMarkerReplication {
status: Some(if marker_enabled {
DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)
} else {
DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)
}),
}),
delete_replication: Some(DeleteReplication {
status: if purge_enabled {
DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED)
} else {
DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED)
},
}),
destination: Destination {
bucket: "arn:rustfs:replication:target".to_string(),
..Default::default()
},
existing_object_replication: None,
filter: None,
id: Some("lifecycle-delete-switches".to_string()),
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
}],
}),
);
let source = ObjectInfo {
bucket: "bucket".to_string(),
name: "logs/object".to_string(),
..Default::default()
};
let marker = ObjectToDelete {
object_name: source.name.clone(),
..Default::default()
};
let marker_opts = ObjectOptions {
versioned: true,
..Default::default()
};
assert_eq!(
ReplicationObjectBridge::check_delete_with_snapshot(&marker, &source, &marker_opts, false, &snapshot)
.replicate_any(),
marker_enabled
);
for delete_marker in [false, true] {
for version_id in [Uuid::new_v4(), Uuid::nil()] {
let purge = ObjectToDelete {
object_name: source.name.clone(),
version_id: Some(version_id),
..Default::default()
};
let purge_source = ObjectInfo {
delete_marker,
..source.clone()
};
let purge_opts = ObjectOptions {
version_id: Some(version_id.to_string()),
versioned: true,
..Default::default()
};
assert_eq!(
ReplicationObjectBridge::check_delete_with_snapshot(
&purge,
&purge_source,
&purge_opts,
false,
&snapshot,
)
.replicate_any(),
purge_enabled,
"delete marker={delete_marker}, version_id={version_id}"
);
}
}
}
}
}
}
@@ -20,10 +20,8 @@ use tokio_util::sync::CancellationToken;
use tracing::{debug, warn};
use crate::bucket::lifecycle::config_boundary;
use crate::bucket::lifecycle::runtime_boundary;
use crate::bucket::lifecycle::tier_sweeper::{
Jentry, TierDeleteJournalState, TierDeleteSourceIdentity,
delete_confirmed_transition_candidate_exact_with_manager_and_identity,
Jentry, delete_confirmed_transition_candidate_exact_with_manager_and_identity,
delete_object_from_remote_tier_idempotent_with_manager_and_identity,
};
use crate::disk::RUSTFS_META_BUCKET;
@@ -32,7 +30,7 @@ use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader
use crate::services::tier::tier::tier_destination_id_from_metadata;
use crate::storage_api_contracts::{
list::ListOperations as _,
object::{DeletedObject, HTTPPreconditions, ObjectIO, ObjectOperations, ObjectToDelete},
object::{DeletedObject, ObjectIO, ObjectOperations, ObjectToDelete},
range::HTTPRangeSpec,
};
use crate::store::ECStore;
@@ -48,7 +46,6 @@ const TIER_DELETE_JOURNAL_RECOVERY_TIMEOUT: Duration = Duration::from_secs(300);
const TIER_DELETE_JOURNAL_VERSION: u8 = 2;
const TIER_DELETE_JOURNAL_EXACT_VERSION: u8 = 3;
const TIER_DELETE_JOURNAL_STATE_VERSION: u8 = 4;
const TIER_DELETE_JOURNAL_TRANSACTION_VERSION: u8 = 5;
pub(crate) const TIER_DELETE_JOURNAL_PREFIX: &str = "ilm/tier-delete-journal/";
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
@@ -64,22 +61,13 @@ struct PersistedTierDeleteJournalEntry {
version_id_exact: Option<bool>,
#[serde(default, skip_serializing_if = "Option::is_none")]
version_state: Option<rustfs_filemeta::TransitionVersionState>,
#[serde(default, skip_serializing_if = "Option::is_none")]
state: Option<TierDeleteJournalState>,
#[serde(default, skip_serializing_if = "Option::is_none")]
source: Option<TierDeleteSourceIdentity>,
}
impl PersistedTierDeleteJournalEntry {
fn from_jentry(je: &Jentry) -> Result<Self> {
validate_version_state(je.version_state, &je.version_id, je.version_id_exact)?;
let legacy_unknown = je.version_state == rustfs_filemeta::TransitionVersionState::Unknown;
let version = if je.source.is_some() || je.state == TierDeleteJournalState::Prepared {
if je.backend_identity.is_none() {
return Err(Error::other("tier delete transaction is missing its backend identity"));
}
TIER_DELETE_JOURNAL_TRANSACTION_VERSION
} else if legacy_unknown {
let version = if legacy_unknown {
if je.backend_identity.is_some() {
TIER_DELETE_JOURNAL_VERSION
} else {
@@ -99,10 +87,6 @@ impl PersistedTierDeleteJournalEntry {
backend_identity: je.backend_identity,
version_id_exact: je.version_id_exact.then_some(true),
version_state: (!legacy_unknown).then_some(je.version_state),
state: (version == TIER_DELETE_JOURNAL_TRANSACTION_VERSION).then_some(je.state),
source: (version == TIER_DELETE_JOURNAL_TRANSACTION_VERSION)
.then(|| je.source.clone())
.flatten(),
})
}
@@ -117,21 +101,14 @@ impl PersistedTierDeleteJournalEntry {
}
if self.version != TIER_DELETE_JOURNAL_EXACT_VERSION
&& self.version != TIER_DELETE_JOURNAL_STATE_VERSION
&& self.version != TIER_DELETE_JOURNAL_TRANSACTION_VERSION
&& self.version_id_exact.unwrap_or(false)
{
return Err(Error::other(
"legacy tier delete journal entry has an unsupported exact version constraint",
));
}
let (backend_identity, version_id_exact, version_state, state, source) = match self.version {
1 => (
None,
false,
rustfs_filemeta::TransitionVersionState::Unknown,
TierDeleteJournalState::Committed,
None,
),
let (backend_identity, version_id_exact, version_state) = match self.version {
1 => (None, false, rustfs_filemeta::TransitionVersionState::Unknown),
TIER_DELETE_JOURNAL_VERSION => (
Some(
self.backend_identity
@@ -139,8 +116,6 @@ impl PersistedTierDeleteJournalEntry {
),
false,
rustfs_filemeta::TransitionVersionState::Unknown,
TierDeleteJournalState::Committed,
None,
),
TIER_DELETE_JOURNAL_EXACT_VERSION => {
if self.version_id.is_empty() || self.version_id_exact != Some(true) {
@@ -153,8 +128,6 @@ impl PersistedTierDeleteJournalEntry {
),
true,
rustfs_filemeta::TransitionVersionState::Exact,
TierDeleteJournalState::Committed,
None,
)
}
TIER_DELETE_JOURNAL_STATE_VERSION => {
@@ -170,31 +143,6 @@ impl PersistedTierDeleteJournalEntry {
),
exact,
state,
TierDeleteJournalState::Committed,
None,
)
}
TIER_DELETE_JOURNAL_TRANSACTION_VERSION => {
let state = self
.state
.ok_or_else(|| Error::other("tier delete journal v5 entry is missing its state"))?;
let source = self
.source
.ok_or_else(|| Error::other("tier delete journal v5 entry is missing its source identity"))?;
let exact = self.version_id_exact.unwrap_or(false);
let version_state = self
.version_state
.ok_or_else(|| Error::other("tier delete journal v5 entry is missing its version state"))?;
validate_version_state(version_state, &self.version_id, exact)?;
(
Some(
self.backend_identity
.ok_or_else(|| Error::other("tier delete journal v5 entry is missing its backend identity"))?,
),
exact,
version_state,
state,
Some(source),
)
}
version => return Err(Error::other(format!("unsupported tier delete journal version {version}"))),
@@ -206,8 +154,6 @@ impl PersistedTierDeleteJournalEntry {
backend_identity,
version_id_exact,
version_state,
state,
source,
})
}
}
@@ -255,20 +201,6 @@ pub(crate) fn tier_delete_journal_object_name(je: &Jentry) -> String {
hasher.update([0]);
hasher.update(b"exact-version-id");
}
if let Some(source) = &je.source {
hasher.update([0]);
hasher.update(source.bucket.as_bytes());
hasher.update([0]);
hasher.update(source.object.as_bytes());
hasher.update([0]);
hasher.update(source.version_id.as_deref().unwrap_or_default().as_bytes());
hasher.update([0]);
hasher.update(source.data_dir.as_deref().unwrap_or_default().as_bytes());
hasher.update([0]);
hasher.update(source.etag.as_deref().unwrap_or_default().as_bytes());
hasher.update([0]);
hasher.update(source.mod_time.as_deref().unwrap_or_default().as_bytes());
}
format!(
"{TIER_DELETE_JOURNAL_PREFIX}{}.json",
rustfs_utils::crypto::hex(hasher.finalize().as_slice())
@@ -314,66 +246,6 @@ where
.map_err(std::io::Error::other)
}
pub async fn commit_tier_delete_journal_entry<S>(api: Arc<S>, je: &Jentry) -> std::io::Result<()>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = http::HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
>,
{
let mut committed = je.clone();
committed.state = TierDeleteJournalState::Committed;
persist_tier_delete_journal_entry(api, &committed).await
}
pub async fn abort_tier_delete_journal_entry<S>(api: Arc<S>, je: &Jentry) -> std::io::Result<()>
where
S: ObjectOperations<
Error = Error,
ObjectInfo = ObjectInfo,
ObjectOptions = ObjectOptions,
FileInfo = FileInfo,
ObjectToDelete = ObjectToDelete,
DeletedObject = DeletedObject,
>,
{
remove_tier_delete_journal_entry(api, je).await
}
pub async fn abort_prepared_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
let name = tier_delete_journal_object_name(je);
let (data, metadata) = match config_boundary::read_config_with_metadata(api.clone(), &name, &ObjectOptions::default()).await {
Ok(result) => result,
Err(Error::ConfigNotFound) | Err(Error::FileNotFound) => return Ok(()),
Err(err) => return Err(std::io::Error::other(err)),
};
let current = decode_tier_delete_journal_entry(&data).map_err(std::io::Error::other)?;
if current.state != TierDeleteJournalState::Prepared {
return Ok(());
}
let etag = metadata
.etag
.ok_or_else(|| std::io::Error::other("prepared tier delete journal has no entity tag"))?;
match config_boundary::delete_config_if_match(api, &name, &etag).await {
Ok(()) | Err(Error::ConfigNotFound) => Ok(()),
Err(Error::PreconditionFailed) => Err(std::io::Error::new(
std::io::ErrorKind::WouldBlock,
"prepared tier delete journal changed before abort",
)),
Err(err) => Err(std::io::Error::other(err)),
}
}
pub(crate) async fn enqueue_committed_tier_delete_journal_entry(je: &Jentry) -> std::io::Result<()> {
let expiry_state = runtime_boundary::expiry_state_handle();
expiry_state.write().await.enqueue_tier_journal_entry(je)
}
pub async fn remove_tier_delete_journal_entry<S>(api: Arc<S>, je: &Jentry) -> std::io::Result<()>
where
S: ObjectOperations<
@@ -392,13 +264,6 @@ where
}
pub async fn process_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
if je.state == TierDeleteJournalState::Prepared {
return reconcile_prepared_tier_delete_journal_entry(api, je).await;
}
process_committed_tier_delete_journal_entry(api, je).await
}
async fn process_committed_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
if je.version_state == rustfs_filemeta::TransitionVersionState::Unknown {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
@@ -431,87 +296,6 @@ async fn process_committed_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jen
remove_tier_delete_journal_entry(api, je).await
}
async fn reconcile_prepared_tier_delete_journal_entry(api: Arc<ECStore>, je: &Jentry) -> std::io::Result<()> {
let (data, metadata) =
config_boundary::read_config_with_metadata(api.clone(), &tier_delete_journal_object_name(je), &ObjectOptions::default())
.await
.map_err(std::io::Error::other)?;
let current = decode_tier_delete_journal_entry(&data).map_err(std::io::Error::other)?;
if current.state != TierDeleteJournalState::Prepared {
return Err(std::io::Error::new(
std::io::ErrorKind::WouldBlock,
"prepared tier delete journal changed before reconciliation",
));
}
let Some(etag) = metadata.etag else {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"prepared tier delete journal has no entity tag",
));
};
let source = je
.source
.as_ref()
.ok_or_else(|| std::io::Error::new(std::io::ErrorKind::InvalidData, "prepared tier delete journal has no source"))?;
match api
.get_object_info(&source.bucket, &source.object, &source.lookup_options())
.await
{
Ok(info) if source.matches(&info) => {
match config_boundary::delete_config_if_match(api, &tier_delete_journal_object_name(&current), &etag).await {
Ok(()) => Ok(()),
Err(Error::PreconditionFailed) => Err(std::io::Error::new(
std::io::ErrorKind::WouldBlock,
"prepared tier delete journal changed before abort",
)),
Err(err) => Err(std::io::Error::other(err)),
}
}
Ok(_info) if source.has_stable_identity() => {
commit_prepared_tier_delete_journal_entry_if_current(api, current, etag).await
}
Ok(_) => Err(std::io::Error::new(
std::io::ErrorKind::WouldBlock,
"prepared tier delete journal source identity is not sufficient to confirm deletion",
)),
Err(Error::ObjectNotFound(_, _)) | Err(Error::FileNotFound) | Err(Error::FileVersionNotFound) => {
commit_prepared_tier_delete_journal_entry_if_current(api, current, etag).await
}
Err(err) => Err(std::io::Error::other(err)),
}
}
async fn commit_prepared_tier_delete_journal_entry_if_current(
api: Arc<ECStore>,
mut committed: Jentry,
etag: String,
) -> std::io::Result<()> {
committed.state = TierDeleteJournalState::Committed;
let data = encode_tier_delete_journal_entry(&committed).map_err(std::io::Error::other)?;
match config_boundary::save_config_with_opts(
api.clone(),
&tier_delete_journal_object_name(&committed),
data,
&ObjectOptions {
max_parity: true,
http_preconditions: Some(HTTPPreconditions {
if_match: Some(etag),
..Default::default()
}),
..Default::default()
},
)
.await
{
Ok(()) => process_committed_tier_delete_journal_entry(api, &committed).await,
Err(Error::PreconditionFailed) => Err(std::io::Error::new(
std::io::ErrorKind::WouldBlock,
"prepared tier delete journal changed before commit",
)),
Err(err) => Err(std::io::Error::other(err)),
}
}
pub async fn recover_tier_delete_journal_entries(
api: Arc<ECStore>,
limit: usize,
@@ -698,13 +482,10 @@ mod tests {
decode_tier_delete_journal_entry, encode_tier_delete_journal_entry, record_tier_delete_journal_backend_identity,
tier_delete_journal_object_name,
};
use crate::bucket::lifecycle::tier_sweeper::{Jentry, TierDeleteJournalState, TierDeleteSourceIdentity};
use crate::bucket::lifecycle::tier_sweeper::Jentry;
use crate::error::Result;
use crate::object_api::ObjectInfo;
use std::time::Duration;
use time::OffsetDateTime;
use tokio_util::sync::CancellationToken;
use uuid::Uuid;
fn journal_entry() -> Jentry {
Jentry {
@@ -714,8 +495,6 @@ mod tests {
backend_identity: Some([7; 32]),
version_id_exact: true,
version_state: rustfs_filemeta::TransitionVersionState::Exact,
state: TierDeleteJournalState::Committed,
source: None,
}
}
@@ -734,55 +513,6 @@ mod tests {
assert_eq!(decoded.version_state, je.version_state);
}
#[test]
fn tier_delete_transaction_roundtrips_prepared_source_identity() {
let mut je = journal_entry();
je.state = TierDeleteJournalState::Prepared;
je.source = Some(TierDeleteSourceIdentity {
bucket: "bucket".to_string(),
object: "object".to_string(),
version_id: Some("version".to_string()),
versioned: true,
version_suspended: false,
data_dir: Some("data-dir".to_string()),
etag: Some("etag".to_string()),
mod_time: Some("mod-time".to_string()),
});
let encoded = encode_tier_delete_journal_entry(&je).expect("prepared transaction should encode");
let value: serde_json::Value = serde_json::from_slice(&encoded).expect("transaction should be JSON");
assert_eq!(value["version"], serde_json::json!(5));
assert_eq!(value["state"], serde_json::json!("Prepared"));
assert!(value["source"].is_object());
let decoded = decode_tier_delete_journal_entry(&encoded).expect("prepared transaction should decode");
assert_eq!(decoded.state, TierDeleteJournalState::Prepared);
assert_eq!(decoded.source, je.source);
}
#[test]
fn tier_delete_source_identity_rejects_recreated_object() {
let version_id = Uuid::from_u128(1);
let data_dir = Uuid::from_u128(2);
let mod_time = OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1);
let info = ObjectInfo {
bucket: "bucket".to_string(),
name: "object".to_string(),
version_id: Some(version_id),
data_dir: Some(data_dir),
mod_time: Some(mod_time),
..Default::default()
};
let source = TierDeleteSourceIdentity::from_object_info("bucket", "object", &info, true, false);
assert!(source.matches(&info));
let recreated = ObjectInfo {
data_dir: Some(Uuid::from_u128(3)),
..info
};
assert!(!source.matches(&recreated));
}
#[test]
fn tier_delete_journal_roundtrips_exact_put_response_constraint() {
let mut exact = journal_entry();
@@ -23,12 +23,10 @@ use crate::bucket::lifecycle::bucket_lifecycle_ops::ExpiryOp;
use crate::bucket::lifecycle::lifecycle::{self, ObjectOpts};
use crate::bucket::lifecycle::tier_delete_journal::persist_tier_delete_journal_entry;
use crate::client::signer_error::error_chain_contains_signer_header_marker;
use crate::object_api::ObjectInfo;
use crate::services::tier::tier::{TierConfigMgr, TierDestinationId, TierOperationLease};
use crate::storage_api_contracts::lifecycle::TransitionedObject;
use crate::store::ECStore;
use rustfs_utils::get_env_usize;
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::any::Any;
use std::collections::VecDeque;
@@ -259,8 +257,6 @@ impl ObjSweeper {
rustfs_filemeta::TransitionVersionState::SuspendedNull | rustfs_filemeta::TransitionVersionState::Exact
),
version_state: self.transition_version_state,
state: TierDeleteJournalState::Committed,
source: None,
});
}
None
@@ -289,76 +285,6 @@ impl ObjSweeper {
}
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
pub(crate) enum TierDeleteJournalState {
Prepared,
Committed,
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
#[serde(deny_unknown_fields)]
pub(crate) struct TierDeleteSourceIdentity {
pub(crate) bucket: String,
pub(crate) object: String,
pub(crate) version_id: Option<String>,
pub(crate) versioned: bool,
pub(crate) version_suspended: bool,
pub(crate) data_dir: Option<String>,
pub(crate) etag: Option<String>,
pub(crate) mod_time: Option<String>,
}
impl TierDeleteSourceIdentity {
pub(crate) fn from_object_info(
bucket: &str,
object: &str,
info: &ObjectInfo,
versioned: bool,
version_suspended: bool,
) -> Self {
Self {
bucket: bucket.to_string(),
object: object.to_string(),
version_id: info.version_id.map(|id| id.to_string()),
versioned,
version_suspended,
data_dir: info.data_dir.map(|id| id.to_string()),
etag: info.etag.clone(),
mod_time: info.mod_time.map(|time| time.to_string()),
}
}
pub(crate) fn lookup_options(&self) -> crate::object_api::ObjectOptions {
crate::object_api::ObjectOptions {
version_id: self.version_id.clone(),
versioned: self.versioned,
version_suspended: self.version_suspended,
..Default::default()
}
}
pub(crate) fn matches(&self, info: &ObjectInfo) -> bool {
if self.bucket != info.bucket {
return false;
}
if let Some(version_id) = &self.version_id {
return info.version_id.map(|id| id.to_string()).as_deref() == Some(version_id.as_str())
&& self.data_dir == info.data_dir.map(|id| id.to_string());
}
if self.data_dir.is_some() {
return self.data_dir == info.data_dir.map(|id| id.to_string());
}
self.etag.is_some()
&& self.etag == info.etag
&& self.mod_time.is_some()
&& self.mod_time == info.mod_time.map(|time| time.to_string())
}
pub(crate) fn has_stable_identity(&self) -> bool {
self.version_id.is_some() || self.data_dir.is_some() || (self.etag.is_some() && self.mod_time.is_some())
}
}
#[derive(Debug, Clone)]
#[allow(unused_assignments)]
pub struct Jentry {
@@ -368,8 +294,6 @@ pub struct Jentry {
pub(crate) backend_identity: Option<TierDestinationId>,
pub(crate) version_id_exact: bool,
pub(crate) version_state: rustfs_filemeta::TransitionVersionState,
pub(crate) state: TierDeleteJournalState,
pub(crate) source: Option<TierDeleteSourceIdentity>,
}
impl ExpiryOp for Jentry {
@@ -630,48 +554,9 @@ pub fn transitioned_force_delete_journal_entry(
rustfs_filemeta::TransitionVersionState::SuspendedNull | rustfs_filemeta::TransitionVersionState::Exact
),
version_state: transition_version_state,
state: TierDeleteJournalState::Committed,
source: None,
})
}
pub(crate) fn attach_tier_delete_source(
je: &mut Jentry,
bucket: &str,
object: &str,
info: &ObjectInfo,
versioned: bool,
version_suspended: bool,
) {
je.state = TierDeleteJournalState::Prepared;
je.source = Some(TierDeleteSourceIdentity::from_object_info(
bucket,
object,
info,
versioned,
version_suspended,
));
}
pub(crate) fn transitioned_delete_journal_entry_for_source(
version_id: Option<Uuid>,
versioned: bool,
suspended: bool,
bucket: &str,
object: &str,
source: &ObjectInfo,
) -> Option<Jentry> {
let mut je = transitioned_delete_journal_entry(
version_id,
versioned,
suspended,
&source.transitioned_object,
source.transition_version_state,
)?;
attach_tier_delete_source(&mut je, bucket, object, source, versioned, suspended);
Some(je)
}
#[cfg(test)]
mod test {
use crate::client::signer_error::invalid_utf8_header_error;
+7 -189
View File
@@ -16,9 +16,8 @@ use super::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
use super::object_lock::ObjectLockApi;
use super::versioning::VersioningApi;
use super::{quota::BucketQuota, target::BucketTargets};
use crate::bucket::replication::invalid_replication_config_status_field;
use crate::bucket::utils::deserialize;
use crate::config::com::{read_config, read_config_preserve_empty, save_config};
use crate::config::com::{read_config, save_config};
use crate::disk::BUCKET_META_PREFIX;
use crate::error::{Error, Result};
use crate::runtime::sources as runtime_sources;
@@ -26,9 +25,9 @@ use crate::store::ECStore;
use byteorder::{BigEndian, ByteOrder, LittleEndian};
use rustfs_policy::policy::BucketPolicy;
use s3s::dto::{
AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, BucketVersioningStatus, CORSConfiguration,
NotificationConfiguration, ObjectLockConfiguration, PublicAccessBlockConfiguration, ReplicationConfiguration,
RequestPaymentConfiguration, ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, CORSConfiguration, NotificationConfiguration,
ObjectLockConfiguration, PublicAccessBlockConfiguration, ReplicationConfiguration, RequestPaymentConfiguration,
ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
};
use serde::Serializer;
use sha2::{Digest, Sha256};
@@ -37,7 +36,6 @@ use std::io::{Read, Write};
use std::sync::Arc;
use time::{Date, OffsetDateTime, PrimitiveDateTime, Time as CivilTime, UtcOffset};
use tracing::error;
use uuid::Uuid;
fn read_msgp_str<R: Read>(rd: &mut R) -> Result<String> {
let len = rmp::decode::read_str_len(rd)? as usize;
@@ -227,7 +225,6 @@ fn write_bin_field<W: Write>(wr: &mut W, key: &str, val: &[u8]) -> Result<()> {
}
pub const BUCKET_METADATA_FILE: &str = ".metadata.bin";
pub const BUCKET_INCARNATION_FILE: &str = ".bucket-incarnation";
pub const BUCKET_METADATA_FORMAT: u16 = 1;
pub const BUCKET_METADATA_VERSION: u16 = 1;
@@ -279,8 +276,6 @@ pub struct BucketMetadata {
pub name: String,
pub created: OffsetDateTime,
pub lock_enabled: bool, // While marked as unused, it may need to be retained
pub bucket_incarnation_id: Uuid,
pub(crate) bucket_incarnation_sidecar: bool,
pub policy_config_json: Vec<u8>,
pub notification_config_xml: Vec<u8>,
pub lifecycle_config_xml: Vec<u8>,
@@ -351,8 +346,6 @@ impl Default for BucketMetadata {
name: Default::default(),
created: OffsetDateTime::UNIX_EPOCH,
lock_enabled: Default::default(),
bucket_incarnation_id: Uuid::nil(),
bucket_incarnation_sidecar: false,
policy_config_json: Default::default(),
notification_config_xml: Default::default(),
lifecycle_config_xml: Default::default(),
@@ -420,7 +413,6 @@ impl BucketMetadata {
pub fn new(name: &str) -> Self {
BucketMetadata {
name: name.to_string(),
bucket_incarnation_id: Uuid::new_v4(),
..Default::default()
}
}
@@ -486,11 +478,6 @@ impl BucketMetadata {
"Name" => self.name = read_msgp_str(rd)?,
"Created" => self.created = read_msgp_time_value(rd)?,
"LockEnabled" => self.lock_enabled = read_msgp_bool(rd)?,
"BucketIncarnationID" => {
let bytes = read_msgp_bin(rd)?;
self.bucket_incarnation_id =
Uuid::from_slice(&bytes).map_err(|err| Error::other(format!("invalid BucketIncarnationID: {err}")))?;
}
"PolicyConfigJSON" | "PolicyConfigJson" => self.policy_config_json = read_msgp_bin(rd)?,
"NotificationConfigXML" | "NotificationConfigXml" => self.notification_config_xml = read_msgp_bin(rd)?,
"LifecycleConfigXML" | "LifecycleConfigXml" => self.lifecycle_config_xml = read_msgp_bin(rd)?,
@@ -547,8 +534,8 @@ impl BucketMetadata {
/// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
// Map size: MinIO fields (25) + RustFS extensions (19)
let map_len: u32 = 44;
// Map size: MinIO fields (25) + RustFS extensions (18)
let map_len: u32 = 43;
rmp::encode::write_map_len(wr, map_len)?;
// MinIO field order (same as Go struct)
@@ -561,8 +548,6 @@ impl BucketMetadata {
rmp::encode::write_str(wr, "LockEnabled")?;
rmp::encode::write_bool(wr, self.lock_enabled)?;
write_bin_field(wr, "BucketIncarnationID", self.bucket_incarnation_id.as_bytes())?;
write_bin_field(wr, "PolicyConfigJSON", &self.policy_config_json)?;
write_bin_field(wr, "NotificationConfigXML", &self.notification_config_xml)?;
write_bin_field(wr, "LifecycleConfigXML", &self.lifecycle_config_xml)?;
@@ -762,41 +747,15 @@ impl BucketMetadata {
self.quota_config_updated_at = updated;
}
OBJECT_LOCK_CONFIG => {
self.object_lock_config = None;
if !data.is_empty() {
self.lock_enabled = true;
}
self.object_lock_config_xml = data;
self.object_lock_config_updated_at = updated;
}
BUCKET_VERSIONING_CONFIG => {
let config = if data.is_empty() {
None
} else {
let config = deserialize::<VersioningConfiguration>(&data)?;
if config.status.as_ref().is_some_and(|status| {
!matches!(status.as_str(), BucketVersioningStatus::ENABLED | BucketVersioningStatus::SUSPENDED)
}) {
return Err(Error::other("bucket versioning configuration has an invalid status"));
}
Some(config)
};
self.versioning_config_xml = data;
self.versioning_config = config;
self.versioning_config_updated_at = updated;
}
BUCKET_REPLICATION_CONFIG => {
let config = if data.is_empty() {
None
} else {
let config = deserialize::<ReplicationConfiguration>(&data)?;
if let Some(field) = invalid_replication_config_status_field(&config) {
return Err(Error::other(format!("replication field {field} has an invalid status")));
}
Some(config)
};
self.replication_config_xml = data;
self.replication_config = config;
self.replication_config_updated_at = updated;
}
BUCKET_TARGETS_FILE => {
@@ -866,6 +825,7 @@ impl BucketMetadata {
/// ambient (first) one. [`BucketMetadata::save`] keeps the ambient default.
pub async fn save_with_store(&mut self, store: std::sync::Arc<crate::store::ECStore>) -> Result<()> {
self.parse_all_configs()?;
let mut buf: Vec<u8> = vec![0; 4];
LittleEndian::write_u16(&mut buf[0..2], BUCKET_METADATA_FORMAT);
@@ -946,7 +906,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.versioning_config = None;
if !self.versioning_config_xml.is_empty()
&& let Err(e) =
deserialize::<VersioningConfiguration>(&self.versioning_config_xml).map(|c| self.versioning_config = Some(c))
@@ -1001,7 +960,6 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.replication_config = None;
if !self.replication_config_xml.is_empty()
&& let Err(e) =
deserialize::<ReplicationConfiguration>(&self.replication_config_xml).map(|c| self.replication_config = Some(c))
@@ -1133,29 +1091,6 @@ impl BucketMetadata {
}
}
pub(crate) async fn load_bucket_incarnation(api: Arc<ECStore>, bucket: &str) -> Result<Option<Uuid>> {
let path = format!("{BUCKET_META_PREFIX}/{bucket}/{BUCKET_INCARNATION_FILE}");
let data = match read_config_preserve_empty(api, &path).await {
Ok(data) => data,
Err(Error::ConfigNotFound) => return Ok(None),
Err(err) => return Err(err),
};
let incarnation =
Uuid::from_slice(&data).map_err(|err| Error::other(format!("persisted bucket incarnation is invalid: {err}")))?;
if incarnation.is_nil() {
return Err(Error::other("persisted bucket incarnation is nil"));
}
Ok(Some(incarnation))
}
pub(crate) async fn save_bucket_incarnation(api: Arc<ECStore>, bucket: &str, incarnation: Uuid) -> Result<()> {
if incarnation.is_nil() {
return Err(Error::other("cannot persist a nil bucket incarnation"));
}
let path = format!("{BUCKET_META_PREFIX}/{bucket}/{BUCKET_INCARNATION_FILE}");
save_config(api, &path, incarnation.as_bytes().to_vec()).await
}
pub async fn load_bucket_metadata(api: Arc<ECStore>, bucket: &str) -> Result<BucketMetadata> {
load_bucket_metadata_parse(api, bucket, true).await
}
@@ -1183,23 +1118,6 @@ pub(crate) async fn load_bucket_metadata_parse_with_presence(
}
};
let incarnation = load_bucket_incarnation(api, bucket).await?;
if persisted {
if let Some(incarnation) = incarnation {
if !bm.bucket_incarnation_id.is_nil() && bm.bucket_incarnation_id != incarnation {
return Err(Error::other("bucket incarnation sidecar does not match bucket metadata"));
}
bm.bucket_incarnation_id = incarnation;
bm.bucket_incarnation_sidecar = true;
} else if !bm.bucket_incarnation_id.is_nil() {
return Err(Error::other(format!(
"bucket incarnation sidecar is missing for new-format metadata: {bucket}"
)));
}
} else if incarnation.is_some() {
return Err(Error::other("bucket incarnation sidecar exists without bucket metadata"));
}
bm.default_timestamps();
if parse {
@@ -1267,10 +1185,6 @@ mod test {
// Same 4-byte format|version header (1|1) and msgpack layout as MinIO.
BucketMetadata::check_header(&blob).expect("valid .metadata.bin header");
let mut bm = BucketMetadata::unmarshal(&blob[4..]).expect("unmarshal MinIO bucket metadata");
assert!(
bm.bucket_incarnation_id.is_nil(),
"legacy MinIO metadata has no RustFS bucket incarnation field"
);
// Raw config fields survive the msgpack decode (PascalCase MinIO field names).
assert_eq!(bm.name, "interop");
@@ -1353,42 +1267,6 @@ mod test {
let new = BucketMetadata::unmarshal(&buf).unwrap();
assert_eq!(bm.name, new.name);
assert!(!bm.bucket_incarnation_id.is_nil());
assert_eq!(bm.bucket_incarnation_id, new.bucket_incarnation_id);
}
#[test]
fn bucket_incarnation_msgpack_rejects_invalid_binary_length() {
let mut fixture = Vec::new();
rmp::encode::write_map_len(&mut fixture, 1).unwrap();
rmp::encode::write_str(&mut fixture, "BucketIncarnationID").unwrap();
rmp::encode::write_bin(&mut fixture, &[0_u8; 15]).unwrap();
let err = BucketMetadata::unmarshal(&fixture).expect_err("non-UUID incarnation bytes must fail closed");
assert!(err.to_string().contains("invalid BucketIncarnationID"));
}
#[test]
fn same_name_bucket_metadata_gets_a_new_incarnation() {
let old = BucketMetadata::new("recreated");
let new = BucketMetadata::new("recreated");
assert!(!old.bucket_incarnation_id.is_nil());
assert!(!new.bucket_incarnation_id.is_nil());
assert_ne!(old.bucket_incarnation_id, new.bucket_incarnation_id);
}
#[test]
fn site_replication_config_updates_cannot_replace_bucket_incarnation() {
let mut metadata = BucketMetadata::new("site-replication-update");
let incarnation = metadata.bucket_incarnation_id;
metadata
.update_config(BUCKET_POLICY_CONFIG, br#"{"Version":"2012-10-17","Statement":[]}"#.to_vec())
.unwrap();
metadata.update_config(OBJECT_LOCK_CONFIG, Vec::new()).unwrap();
assert_eq!(metadata.bucket_incarnation_id, incarnation);
}
#[test]
@@ -1467,66 +1345,6 @@ mod test {
assert!(bm.tagging_config.is_none());
}
#[test]
fn delete_admission_configs_update_parsed_state_atomically() {
let mut bm = BucketMetadata::new("test-bucket");
let versioning_xml = b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>";
let replication_xml = b"<ReplicationConfiguration><Role>arn:aws:s3:::target-bucket</Role><Rule><ID>rule1</ID><Status>Enabled</Status><Prefix></Prefix><Destination><Bucket>arn:aws:s3:::target-bucket</Bucket></Destination></Rule></ReplicationConfiguration>";
bm.update_config(BUCKET_VERSIONING_CONFIG, versioning_xml.to_vec())
.expect("valid versioning config should update parsed state");
bm.update_config(BUCKET_REPLICATION_CONFIG, replication_xml.to_vec())
.expect("valid replication config should update parsed state");
assert!(bm.versioning_config.as_ref().is_some_and(VersioningConfiguration::enabled));
assert_eq!(
bm.replication_config.as_ref().map(|config| config.role.as_str()),
Some("arn:aws:s3:::target-bucket")
);
assert!(
bm.update_config(BUCKET_VERSIONING_CONFIG, b"<VersioningConfiguration>".to_vec())
.is_err()
);
assert!(
bm.update_config(BUCKET_REPLICATION_CONFIG, b"<ReplicationConfiguration>".to_vec())
.is_err()
);
assert_eq!(bm.versioning_config_xml, versioning_xml);
assert_eq!(bm.replication_config_xml, replication_xml);
assert!(bm.versioning_config.as_ref().is_some_and(VersioningConfiguration::enabled));
assert_eq!(
bm.replication_config.as_ref().map(|config| config.role.as_str()),
Some("arn:aws:s3:::target-bucket")
);
assert!(
bm.update_config(
BUCKET_VERSIONING_CONFIG,
b"<VersioningConfiguration><Status>Enabld</Status></VersioningConfiguration>".to_vec(),
)
.is_err()
);
assert!(
bm.update_config(
BUCKET_REPLICATION_CONFIG,
b"<ReplicationConfiguration><Role>arn:aws:s3:::target-bucket</Role><Rule><ID>rule1</ID><Status>Enabld</Status><Prefix></Prefix><Destination><Bucket>arn:aws:s3:::target-bucket</Bucket></Destination></Rule></ReplicationConfiguration>".to_vec(),
)
.is_err()
);
assert_eq!(bm.versioning_config_xml, versioning_xml);
assert_eq!(bm.replication_config_xml, replication_xml);
bm.versioning_config_xml = b"<VersioningConfiguration>".to_vec();
bm.replication_config_xml = b"<ReplicationConfiguration>".to_vec();
bm.parse_all_configs()
.expect("bulk config parsing reports malformed fields through cleared typed state");
assert!(bm.versioning_config.is_none());
assert!(bm.replication_config.is_none());
}
#[tokio::test]
async fn marshal_msg_complete_example() {
// Create a complete BucketMetadata with various configurations
File diff suppressed because it is too large Load Diff
@@ -12,12 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::bucket::metadata_sys::{ObjectLockConfigState, get_object_lock_config, get_object_lock_config_state};
use crate::bucket::metadata_sys::get_object_lock_config;
use crate::bucket::object_lock::objectlock;
use crate::error::{Error, Result, StorageError};
use crate::object_api::ObjectInfo;
use s3s::dto::{Date, DefaultRetention, ObjectLockConfiguration, ObjectLockLegalHoldStatus, ObjectLockRetentionMode};
use s3s::header::{X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE};
use s3s::dto::{DefaultRetention, ObjectLockLegalHoldStatus, ObjectLockRetentionMode};
use std::sync::Arc;
use time::OffsetDateTime;
@@ -39,20 +37,6 @@ impl BucketObjectLockSys {
}
}
pub(crate) fn ensure_recursive_force_delete_allowed_for_state(bucket: &str, state: &ObjectLockConfigState) -> Result<()> {
match state {
ObjectLockConfigState::ConfirmedAbsent => Ok(()),
ObjectLockConfigState::Configured { .. } => Err(StorageError::InvalidArgument(
bucket.to_string(),
String::new(),
"force-delete is forbidden on Object Locking enabled buckets".to_string(),
)),
ObjectLockConfigState::Fabricated => {
Err(Error::other(format!("bucket Object Lock metadata is not authoritative: {bucket}")))
}
}
}
/// Check if a retention period is still active based on mode and retain_until_date
pub fn is_retention_active(mode: &str, retain_until_date: Option<&s3s::dto::Date>) -> bool {
if mode != ObjectLockRetentionMode::COMPLIANCE && mode != ObjectLockRetentionMode::GOVERNANCE {
@@ -221,122 +205,71 @@ fn check_retention_blocks_deletion(
None
}
/// Check an object's lock metadata using an already resolved bucket Object
/// Lock configuration. `None` means the configuration is confirmed absent.
///
/// # S3 Standard Behavior
/// - COMPLIANCE mode: Cannot be deleted even with bypass header
/// - GOVERNANCE mode: Can be deleted if bypass_governance is true (caller must verify s3:BypassGovernanceRetention permission)
/// - Legal Hold: Cannot be bypassed regardless of mode
pub(crate) fn check_object_lock_for_deletion_with_config(
config: Option<&ObjectLockConfiguration>,
obj_info: &ObjectInfo,
bypass_governance: bool,
) -> Result<Option<ObjectLockBlockReason>> {
if obj_info.delete_marker {
return Ok(None);
}
if let Some(status) = obj_info.user_defined.get(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()) {
if status.eq_ignore_ascii_case(ObjectLockLegalHoldStatus::ON) {
return Ok(Some(ObjectLockBlockReason::LegalHold));
}
if !status.eq_ignore_ascii_case(ObjectLockLegalHoldStatus::OFF) {
return Err(Error::other("persisted object legal-hold metadata is invalid"));
}
}
let mode = obj_info.user_defined.get(X_AMZ_OBJECT_LOCK_MODE.as_str());
let retain_until = obj_info.user_defined.get(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str());
let explicit_ret = match (mode, retain_until) {
(None, None) => None,
(Some(mode), Some(retain_until)) => {
let mode =
objectlock::parse_ret_mode(mode).ok_or_else(|| Error::other("persisted object retention mode is invalid"))?;
let retain_until = OffsetDateTime::parse(retain_until, &time::format_description::well_known::Iso8601::DEFAULT)
.map(Date::from)
.map_err(|_| Error::other("persisted object retention date is invalid"))?;
Some((mode, retain_until))
}
_ => return Err(Error::other("persisted object retention metadata is incomplete")),
};
if let Some((mode, retain_until)) = &explicit_ret {
let mode_str = mode.as_str();
if is_retention_active(mode_str, Some(retain_until))
&& let Some(reason) =
check_retention_blocks_deletion(mode_str, Some(OffsetDateTime::from(retain_until.clone())), bypass_governance)
{
return Ok(Some(reason));
}
}
if explicit_ret.is_none()
&& let Some(default_retention) = config.and_then(|config| config.rule.as_ref()?.default_retention.as_ref())
&& let Some(mode) = &default_retention.mode
{
let mode_str = mode.as_str();
if mode_str == ObjectLockRetentionMode::COMPLIANCE || mode_str == ObjectLockRetentionMode::GOVERNANCE {
// Calculate retention expiration date from object modification time
let mod_time = obj_info
.mod_time
.ok_or_else(|| Error::other("persisted object modification time is missing"))?;
let now = objectlock::utc_now_ntp();
let retain_until = if let Some(days) = default_retention.days {
mod_time.saturating_add(time::Duration::days(i64::from(days)))
} else {
let years = default_retention
.years
.ok_or_else(|| Error::other("persisted bucket Object Lock retention period is invalid"))?;
add_years(mod_time, years)
};
if retain_until.unix_timestamp() > now.unix_timestamp()
&& let Some(reason) = check_retention_blocks_deletion(mode_str, Some(retain_until), bypass_governance)
{
return Ok(Some(reason));
}
}
}
Ok(None)
}
pub(crate) fn check_object_lock_for_deletion_with_state(
state: &ObjectLockConfigState,
obj_info: &ObjectInfo,
bypass_governance: bool,
) -> Result<Option<ObjectLockBlockReason>> {
match state {
ObjectLockConfigState::Configured { config, .. } => {
check_object_lock_for_deletion_with_config(Some(config), obj_info, bypass_governance)
}
ObjectLockConfigState::ConfirmedAbsent => check_object_lock_for_deletion_with_config(None, obj_info, bypass_governance),
ObjectLockConfigState::Fabricated => Err(Error::other("bucket Object Lock metadata is not authoritative")),
}
}
/// Compatibility wrapper for callers that predate fallible metadata lookup.
/// An authority/read/parse failure is represented as a blocking reason rather
/// than the old fail-open `None` result.
pub async fn check_object_lock_for_deletion(
bucket: &str,
obj_info: &ObjectInfo,
bypass_governance: bool,
) -> Option<ObjectLockBlockReason> {
match get_object_lock_config_state(bucket)
.await
.and_then(|state| check_object_lock_for_deletion_with_state(&state, obj_info, bypass_governance))
{
Ok(reason) => reason,
Err(_) => Some(ObjectLockBlockReason::LegalHold),
if obj_info.delete_marker {
return None;
}
// 1. Check legal hold - cannot be bypassed (reuse has_legal_hold)
if has_legal_hold(&obj_info.user_defined) {
return Some(ObjectLockBlockReason::LegalHold);
}
// 2. Check explicit retention
let explicit_ret = objectlock::get_object_retention_meta(&obj_info.user_defined);
if let Some(mode) = &explicit_ret.mode {
let mode_str = mode.as_str();
if is_retention_active(mode_str, explicit_ret.retain_until_date.as_ref())
&& let Some(reason) = check_retention_blocks_deletion(
mode_str,
explicit_ret.retain_until_date.map(OffsetDateTime::from),
bypass_governance,
)
{
return Some(reason);
}
}
// 3. Check default retention only if no explicit retention is set
if explicit_ret.mode.is_none()
&& let Some(default_retention) = BucketObjectLockSys::get(bucket).await
&& let Some(mode) = &default_retention.mode
{
let mode_str = mode.as_str();
if mode_str == ObjectLockRetentionMode::COMPLIANCE || mode_str == ObjectLockRetentionMode::GOVERNANCE {
// Calculate retention expiration date from object modification time
if let Some(mod_time) = obj_info.mod_time {
let now = objectlock::utc_now_ntp();
let retain_until = if let Some(days) = default_retention.days {
mod_time.saturating_add(time::Duration::days(days as i64))
} else {
let years = default_retention.years?;
add_years(mod_time, years)
};
if retain_until.unix_timestamp() > now.unix_timestamp()
&& let Some(reason) = check_retention_blocks_deletion(mode_str, Some(retain_until), bypass_governance)
{
return Some(reason);
}
}
}
}
None
}
#[cfg(test)]
mod tests {
use super::*;
use s3s::dto::{ObjectLockEnabled, ObjectLockRule};
use time::{Date, Month, PrimitiveDateTime, Time};
fn make_datetime(year: i32, month: u8, day: u8) -> OffsetDateTime {
@@ -345,160 +278,6 @@ mod tests {
PrimitiveDateTime::new(date, time).assume_utc()
}
fn default_retention_config(mode: &'static str) -> ObjectLockConfiguration {
ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
mode: Some(ObjectLockRetentionMode::from_static(mode)),
days: Some(30),
years: None,
}),
}),
}
}
#[test]
fn deletion_with_config_blocks_active_default_compliance_even_with_bypass() {
let config = default_retention_config(ObjectLockRetentionMode::COMPLIANCE);
let obj_info = ObjectInfo {
mod_time: Some(OffsetDateTime::now_utc()),
..Default::default()
};
let result = check_object_lock_for_deletion_with_config(Some(&config), &obj_info, true);
assert!(matches!(result, Ok(Some(ObjectLockBlockReason::Retention { .. }))));
}
#[test]
fn deletion_with_config_allows_active_default_governance_with_bypass() {
let config = default_retention_config(ObjectLockRetentionMode::GOVERNANCE);
let obj_info = ObjectInfo {
mod_time: Some(OffsetDateTime::now_utc()),
..Default::default()
};
assert!(matches!(
check_object_lock_for_deletion_with_config(Some(&config), &obj_info, true),
Ok(None)
));
}
#[test]
fn deletion_with_default_retention_rejects_missing_object_mod_time() {
let config = default_retention_config(ObjectLockRetentionMode::COMPLIANCE);
let err = check_object_lock_for_deletion_with_config(Some(&config), &ObjectInfo::default(), false)
.expect_err("default retention needs an authoritative object modification time");
assert!(err.to_string().contains("modification time"));
}
#[test]
fn deletion_with_confirmed_absence_still_blocks_explicit_compliance() {
let retain_until = OffsetDateTime::now_utc() + time::Duration::days(30);
let mut user_defined = std::collections::HashMap::new();
user_defined.insert("x-amz-object-lock-mode".to_string(), ObjectLockRetentionMode::COMPLIANCE.to_string());
user_defined.insert(
"x-amz-object-lock-retain-until-date".to_string(),
retain_until
.format(&time::format_description::well_known::Rfc3339)
.expect("retain-until date should format"),
);
let obj_info = ObjectInfo {
user_defined: Arc::new(user_defined),
..Default::default()
};
let result = check_object_lock_for_deletion_with_config(None, &obj_info, true);
assert!(matches!(result, Ok(Some(ObjectLockBlockReason::Retention { .. }))));
}
#[test]
fn deletion_with_fabricated_bucket_metadata_fails_closed() {
let err = check_object_lock_for_deletion_with_state(&ObjectLockConfigState::Fabricated, &ObjectInfo::default(), false)
.expect_err("non-authoritative Object Lock metadata must block deletion");
assert!(err.to_string().contains("not authoritative"));
}
#[test]
fn recursive_force_delete_with_fabricated_bucket_metadata_fails_closed() {
let err = ensure_recursive_force_delete_allowed_for_state("bucket", &ObjectLockConfigState::Fabricated)
.expect_err("non-authoritative Object Lock metadata must block recursive deletion");
assert!(err.to_string().contains("not authoritative"));
}
#[test]
fn deletion_rejects_incomplete_persisted_retention_metadata() {
let mut user_defined = std::collections::HashMap::new();
user_defined.insert(
X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(),
ObjectLockRetentionMode::COMPLIANCE.to_string(),
);
let obj_info = ObjectInfo {
user_defined: Arc::new(user_defined),
..Default::default()
};
let err = check_object_lock_for_deletion_with_config(None, &obj_info, false)
.expect_err("mode without retain-until date must fail closed");
assert!(err.to_string().contains("incomplete"));
}
#[test]
fn deletion_rejects_each_malformed_persisted_retention_shape() {
let valid_date = (OffsetDateTime::now_utc() + time::Duration::days(30))
.format(&time::format_description::well_known::Rfc3339)
.expect("retain-until date should format");
let cases = [
("invalid mode", Some("INVALID"), Some(valid_date.as_str()), "retention mode"),
(
"invalid date",
Some(ObjectLockRetentionMode::COMPLIANCE),
Some("not-a-date"),
"retention date",
),
("date only", None, Some(valid_date.as_str()), "incomplete"),
];
for (case, mode, retain_until, expected) in cases {
let mut user_defined = std::collections::HashMap::new();
if let Some(mode) = mode {
user_defined.insert(X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), mode.to_string());
}
if let Some(retain_until) = retain_until {
user_defined.insert(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), retain_until.to_string());
}
let obj_info = ObjectInfo {
user_defined: Arc::new(user_defined),
..Default::default()
};
let err = check_object_lock_for_deletion_with_config(None, &obj_info, false).expect_err(case);
assert!(err.to_string().contains(expected), "unexpected {case} error: {err}");
}
}
#[test]
fn deletion_rejects_invalid_persisted_legal_hold_metadata() {
let mut user_defined = std::collections::HashMap::new();
user_defined.insert(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), "INVALID".to_string());
let obj_info = ObjectInfo {
user_defined: Arc::new(user_defined),
..Default::default()
};
let err = check_object_lock_for_deletion_with_config(None, &obj_info, false)
.expect_err("invalid legal-hold value must fail closed");
assert!(err.to_string().contains("legal-hold"));
}
#[test]
fn test_add_years_normal() {
// Normal case: add 1 year to a regular date
+9 -125
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use super::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation};
use crate::bucket::metadata_sys::{BucketMetadataSys, update, update_if_incarnation};
use crate::bucket::metadata_sys::{BucketMetadataSys, update};
use crate::data_usage::get_bucket_usage_memory;
use rustfs_common::metrics::Metric;
use rustfs_config::QUOTA_CONFIG_FILE;
@@ -145,35 +145,14 @@ impl QuotaChecker {
}
pub async fn set_quota_config(&mut self, bucket: &str, quota: BucketQuota) -> Result<OffsetDateTime, QuotaError> {
self.set_quota_config_for_incarnation(bucket, quota, None).await
}
pub async fn set_quota_config_if_incarnation(
&mut self,
bucket: &str,
quota: BucketQuota,
expected_incarnation_id: uuid::Uuid,
) -> Result<OffsetDateTime, QuotaError> {
self.set_quota_config_for_incarnation(bucket, quota, Some(expected_incarnation_id))
.await
}
async fn set_quota_config_for_incarnation(
&mut self,
bucket: &str,
quota: BucketQuota,
expected_incarnation_id: Option<uuid::Uuid>,
) -> Result<OffsetDateTime, QuotaError> {
let json_data = serde_json::to_vec(&quota).map_err(|e| QuotaError::InvalidConfig {
reason: format!("Failed to serialize quota config: {}", e),
})?;
let start_time = Instant::now();
let updated_at = match expected_incarnation_id {
Some(incarnation_id) => update_if_incarnation(bucket, QUOTA_CONFIG_FILE, json_data, incarnation_id).await,
None => update(bucket, QUOTA_CONFIG_FILE, json_data).await,
}
.map_err(QuotaError::StorageError)?;
let updated_at = update(bucket, QUOTA_CONFIG_FILE, json_data)
.await
.map_err(QuotaError::StorageError)?;
rustfs_common::metrics::Metrics::inc_time(Metric::QuotaSync, start_time.elapsed());
Ok(updated_at)
@@ -198,36 +177,11 @@ impl QuotaChecker {
}
pub async fn get_real_time_usage(&self, bucket: &str) -> Result<u64, QuotaError> {
if let Some(usage) = get_bucket_usage_memory(bucket).await {
return Ok(usage);
}
// Degraded window (issue #5716): with no authoritative usage — most
// prominently after upgrading from a pre-v2 release, whose legacy
// `.usage.json` is demoted to non-authoritative until the scanner's
// first complete cycle persists `.usage.v2.json` — failing closed
// turned every write to a quota-enabled bucket into a retryable 503
// for the whole window. Quota admission instead degrades to the last
// persisted per-bucket size. That baseline is static between snapshot
// loads (live writes do not advance it), so hard-quota enforcement is
// advisory for the duration of the window: the overrun is bounded by
// the writes issued before the next complete scanner cycle. Buckets
// with no persisted baseline anywhere keep failing closed.
let store = self.metadata_sys.read().await.object_store();
// Box the fallback: it embeds the whole snapshot-load future, and every
// object write nests a quota check several futures deep, so keeping it
// inline would grow each write's state machine by the loader's full
// size — the debug-build 2MiB worker-stack overflow class fixed for
// bucket-config writes in #5648. The allocation only happens on the
// degraded path; the authoritative fast path returns above.
if let Some(baseline) = Box::pin(crate::data_usage::lookup_degraded_bucket_usage_baseline(store, bucket)).await {
debug!(bucket, baseline, "Bucket quota admission using degraded persisted usage baseline");
return Ok(baseline);
}
Err(QuotaError::UsageUnavailable {
bucket: bucket.to_string(),
})
get_bucket_usage_memory(bucket)
.await
.ok_or_else(|| QuotaError::UsageUnavailable {
bucket: bucket.to_string(),
})
}
}
@@ -257,76 +211,6 @@ mod tests {
assert_eq!(result.quota_limit, None);
}
/// Regression (issue #5716): an upgrade from a pre-v2 release leaves only
/// the legacy `.usage.json` snapshot, which has no completeness marker and
/// is demoted to non-authoritative, and the scanner's first complete cycle
/// can be a long way off. Quota admission must degrade to that persisted
/// baseline instead of failing every write to a quota-enabled bucket with
/// a retryable 503 for the whole window.
#[tokio::test]
#[serial]
async fn quota_admission_falls_back_to_legacy_snapshot_baseline() {
let (_dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore.clone())));
let checker = QuotaChecker::new(sys);
let bucket = format!("quota-legacy-{}", Uuid::new_v4().simple());
let mut legacy = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::now()),
buckets_count: 1,
..Default::default()
};
legacy.buckets_usage.insert(
bucket.clone(),
rustfs_data_usage::BucketUsageInfo {
size: 1_234,
..Default::default()
},
);
legacy.bucket_sizes.insert(bucket.clone(), 1_234);
// usage_snapshot_complete stays false: pre-v2 snapshots do not carry
// the field at all, so they always deserialize as incomplete.
let legacy_path = format!("{}/{}", crate::disk::BUCKET_META_PREFIX, rustfs_data_usage::LEGACY_DATA_USAGE_OBJECT_NAME);
crate::config::com::save_config(
ecstore.clone(),
&legacy_path,
serde_json::to_vec(&legacy).expect("legacy snapshot should encode"),
)
.await
.expect("legacy snapshot fixture should be stored");
crate::data_usage::invalidate_data_usage_snapshot_cache().await;
let usage = checker
.get_real_time_usage(&bucket)
.await
.expect("quota admission must degrade to the persisted legacy baseline");
assert_eq!(usage, 1_234);
// A bucket absent from every persisted snapshot still has no grounded
// baseline and must keep failing closed.
let unknown = format!("quota-unknown-{}", Uuid::new_v4().simple());
assert!(matches!(
checker.get_real_time_usage(&unknown).await,
Err(QuotaError::UsageUnavailable { .. })
));
// Deleting the bucket's usage from the backend must purge the
// baseline: a recreated bucket may not inherit the dead incarnation's
// size, so with no persisted trace left it fails closed again.
crate::data_usage::remove_bucket_usage_from_backend(ecstore.clone(), &bucket)
.await
.expect("bucket usage removal should succeed");
assert!(matches!(
checker.get_real_time_usage(&bucket).await,
Err(QuotaError::UsageUnavailable { .. })
));
crate::data_usage::prepare_bucket_usage_for_namespace_change(&bucket, None)
.await
.expect("test usage cache cleanup should succeed");
crate::data_usage::invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test]
#[serial]
async fn quota_usage_rejects_an_unknown_mutation_baseline() {
@@ -100,41 +100,11 @@ paths.
behind the ECStore replication facade; only `rustfs/src/app/storage_api.rs`
may retain direct object/delete replication helper calls.
## Completion Criteria
## First Code-Bearing Step
The split is complete when the "Current dependency to remove" column in the
Required Contracts table above is empty: every row is either deleted because
the dependency is gone, or reduced to "none". No other signal — file count,
boundary count, line count — measures completion.
Target end state:
- `replication_pool.rs`, `replication_resyncer.rs`, and `replication_state.rs`
move into `crates/replication` behind the contracts above;
- the `*_boundary.rs` and `*_bridge.rs` micro-files dissolve naturally as the
code they fence moves across the crate boundary. They are the mechanical
seams of the migration ratchet — the architecture guard scripts anchor on
their file names — so batch-merging them beforehand is explicitly rejected:
it forces synchronized guard-script/mod/import churn with zero functional
gain;
- the only module that can retire early is `datatypes.rs`: delete it once its
facade consumers import the resync status enums through `rustfs-replication`
directly.
## Milestones
| Milestone | Scope | Status |
|---|---|---|
| M0 | Record the completion criteria and end state (this section). | Done |
| M1 | Contract extraction: resync/queue/stats/object-decision/filemeta/storage wire contracts owned by `crates/replication`; ECStore imports concentrated in `*_boundary.rs`; event sink and runtime access behind local contracts. | Done — see Required Contracts |
| M2 | Move resyncer pure decision logic (no IO) into `crates/replication`. | Pending; sequence after splitting the oversized resyncer/pool functions (`resync_bucket`, `replicate_all`, `start_mrf_processor`) so moves stay mechanical |
| M3 | Move the worker runtime (`replication_pool.rs`, the IO paths of `replication_resyncer.rs`, `replication_state.rs`) once the contract traits are stable. Highest-risk step of the whole plan; do it last. | Pending |
| M4 | Retire the boundary modules together with their guard-script entries; delete `datatypes.rs`. | Pending |
The original first code-bearing step (narrow `ReplicationEventSink` /
`ReplicationRuntime` contracts) has landed — `replication_event_sink.rs`
exists and runtime access goes through local boundary aliases — so new work
starts from M2.
Start with `ReplicationRuntime` or `ReplicationEventSink`. Both can be added as
narrow internal contracts while keeping current queue, MRF, resync, and target
behavior unchanged. Do not start with a crate move.
Current compatibility guard: `crates/ecstore/tests/replication_facade_compat_test.rs`
keeps the ECStore replication facade types covered while architecture rules

Some files were not shown because too many files have changed in this diff Show More