fix(ilm): replace whole-copy-back restore lock with accept-path CAS (#4956)

fix(ilm): serialize RestoreObject accepts with a CAS guard, not the whole copy-back

Implements the backlog#1304 decision: replace #4877's object write lock
held across the entire tier copy-back with an atomic compare-and-set on
the restore ongoing flag.

- ecstore: add ECStore::acquire_restore_accept_guard + RestoreAcceptGuard
  (opaque, purpose-scoped object write lock with an is_lock_lost fence);
  handle_restore_transitioned_object no longer locks the copy-back, so
  HEAD/get_object_info stay non-blocking during a restore and the inner
  put_object/complete_multipart_upload commit locks no longer self-deadlock.
- API: execute_restore_object holds the guard across the restore-status
  read-check-write (no_lock inside the scope), drops it before spawning
  the copy-back; SELECT-type restores keep the plain read-locked path;
  the copy-back pins the resolved version so a concurrent PUT cannot
  strand the flagged version at ongoing=true; concurrent restores are
  rejected with 409 RestoreAlreadyInProgress (was a retryable 500) and
  guard contention maps to 503 SlowDown.
- tests: drop the #4877 entry-blocking unit test (semantics deliberately
  reversed), pin accept-guard mutual exclusion, update the ilm-8 restore
  integration test to the final semantics, and add a concurrent
  double-POST test asserting exactly one acceptance and one tier GET.
This commit is contained in:
Zhengchao An
2026-07-17 17:56:45 +08:00
committed by GitHub
parent a9e3613cfd
commit 701c3eee5b
4 changed files with 246 additions and 53 deletions
+8 -9
View File
@@ -221,10 +221,13 @@ jobs:
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# The #4877 restore self-deadlock is fixed in this PR, which re-enabled
# test_multipart_restore_preserves_parts_and_etag. The remaining exclusions
# each hit a DIFFERENT, independent issue (all tracked under
# rustfs/backlog#1148; they keep #[ignore] with a backlog reference):
# restore_object_usecase_reports_ongoing_conflict_and_completion was
# re-enabled by backlog#1304 (restore accepts serialize on a short CAS
# guard; the copy-back no longer holds the #4877 whole-copy-back lock,
# so the mid-restore ongoing read and fast 409 rejection it asserts are
# the implemented contract). The remaining exclusions each hit a
# DIFFERENT, independent issue (all tracked under rustfs/backlog#1148;
# they keep #[ignore] with a backlog reference):
# - test_noncurrent_{expiry,transition}_still_works_after_immediate_compensation_transition:
# noncurrent transition/expiry after an immediate compensation transition.
# - test_transition_and_restore_flows: transition metadata is missing on
@@ -235,15 +238,11 @@ jobs:
# delete path reads that flag, so cleanup deletes the whole object
# instead of only the local restored copy (ObjectNotFound afterwards).
# The expire_restored delete semantics are unimplemented.
# - restore_object_usecase_reports_ongoing_conflict_and_completion: asserts
# a concurrent get_object_info observes ongoing-request=true mid-restore,
# which #4877's read-vs-restore serialization rules out (see backlog#1148
# ilm-8 criterion 1 - an API-semantics decision, not a bug).
- name: Run ignored ILM integration tests serially
run: |
cargo nextest run -j1 --run-ignored ignored-only \
-p rustfs-scanner -p rustfs \
-E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition) or test(restore_object_usecase_reports_ongoing_conflict_and_completion) or test(test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore))'
-E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition) or test(test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore))'
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)