mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-07 11:10:47 +00:00
fdb29b62747130c447b40dfb2edc9607553e0392
1003 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris An
|
d6210ac17b |
fix(s3,ecstore,table-catalog): bound work driven by client input (#9172)
* fix(ecstore,heal,select): bound storage-layer work driven by client input Scan findings in the erasure store, heal, file metadata and S3 Select layers, grouped because they share the disk walk, the multipart staging area, delete recovery and the select snapshot lease: - Multipart: upload ids are canonicalized before they name a staging path or a lock key (backlog#3027); trailing-slash keys are encoded like every other handler (backlog#3055); unlisted parts are removed at completion (backlog#3106); per-bucket uploads are found through a signed per-disk index instead of a scan of the shared staging area (backlog#3121); encrypted part ETags carry a per-upload secret so a write-only user cannot confirm plaintext guesses (backlog#3044). - Delete and recovery: recovery walks are bounded per request (backlog#3038); a delete marker on a key whose xl.meta path is a directory, and sidecar names used as child keys, no longer wedge the journal (backlog#3035, backlog#3105); over-quota PUTs no longer list the bucket under the ledger lock (backlog#3128). - Listing: scan_dir sends metadata without inline data (backlog#3052), counts delete markers against max-keys (backlog#3108), and rejects a marker that carries the cursor tag (backlog#3059). - Heal: bucket metadata and heal records live in separate namespaces so DeleteBucket cannot erase a drive rebuild (backlog#3039), a deleted bucket is skipped instead of failing the rebuild (backlog#3040), and directory objects are encoded on readback (backlog#3061). - Restore and tiering: RestoreObject preserves tags and the version's modification time (backlog#3073, backlog#3074); Snowball transport headers are not forwarded to the tier (backlog#3090); an epoch modification time round-trips through xl.meta (backlog#3063). - Locks: PutObject takes its bucket locks after the body is staged (backlog#3131); SelectObjectContent releases its namespace locks once the snapshot is taken (backlog#3132). - RPC and peers: the v3 signature check consumes the v2 nonce (backlog#3129); a bucket-metadata reload timeout no longer marks the peer offline (backlog#3124); replication target requests carry an operation deadline (backlog#3102); the scanner resumes its interrupted wait after a lifecycle write (backlog#3118). - S3 Select: bounded schema inference, column cap and budgeted CSV batches, Parquet dictionary bounds and pre-planner projection checks (backlog#3075, backlog#3076, backlog#3077, backlog#3078). * test(e2e): seed legacy duplicate bucket tags offline for the OPA contract * fix(s3): authorize what handlers store and bound request-driven work Scan findings in the S3 request layer (access hooks, object use cases, HTTP server layers), grouped because they share the authorization condition builder, the PutObject admission path and the response layers. Builds on the storage-layer batch for the bounded recovery and multipart APIs it calls. Authorization and policy conditions: - copy source is percent-decoded and version-canonical before the s3:x-amz-copy-source condition (backlog#3135); CopyObject authorizes the tag set it copies (backlog#3034); HeadObject with the replication check header still authorizes s3:GetObject (backlog#3047); the x-rustfs-source-version-id header feeds s3:versionid (backlog#3072); recursive force-delete with a REPLICA header authorizes every descendant as a delete (backlog#3070); include-deleted listings require s3:ListBucketVersions (backlog#3099); the metadata listing route keeps table isolation and uses the same prefix it authorizes (backlog#3053, backlog#3054); If-Match probing needs read permission (backlog#3097); Object Lambda checks s3:GetObjectVersion (backlog#3058); replication proxy fallback keeps ExistingObjectTag denies (backlog#3089); HeadObject returns lock and tag headers only with permission (backlog#3113); IPv4 clients on the dual-stack listener match IPv4 SourceIp conditions (backlog#3049); bucket policies are parsed once and cached (backlog#3032); keystone: access keys no longer verify with an empty secret (backlog#3043). Admission and resource bounds: - GetObject releases disk-read permits when the client stops reading (backlog#3023); PutObject takes buffer and foreground admission only after body bytes arrive and bounds the wait (backlog#3031, backlog#3084); xl.meta size is bounded on the write path (backlog#3100); XML compat layers stream instead of buffering whole bodies (backlog#3094); tier GETs stream the remote body (backlog#3046); on-demand migration pulls release permits on unread GETs (backlog#3110); SSE-KMS key policy covers UploadPart and completion (backlog#3130); MinIO admin decryption runs Argon2id off the async worker (backlog#3024). Input validation and leaks: - replication headers are validated (source ETag shape, original size) (backlog#3092, backlog#3093); control characters in keys and lifecycle rule IDs are rejected (backlog#3101, backlog#3117); Snowball member mtime needs s3:ReplicateObject (backlog#3083); DeleteObjects checks Content-MD5 and admin/STS reject SigV2 replays with altered query or body (backlog#3037, backlog#3026); request logs redact presigned queries and handler spans skip credentials (backlog#3068, backlog#3107); SSE-C listings and completions withhold plaintext metadata and checksums (backlog#3120, backlog#3050); CORS resolves the bucket like s3s does (backlog#3104); PutBucketTagging rejects duplicate keys (backlog#3126); notifications fire after a client disconnect (backlog#3127); ListenNotification streams re-check access (backlog#3119). * fix(api): drop imports left unused by the base merge * test(s3): move the KMS multipart denial test to the app layer The test drives app-layer multipart use cases, so it belongs with them; living in the SSE module it imported app code from the infra layer and failed the layer dependency guard. The SSE test helpers it borrows are exposed through the storage and app facades. * fix(s3): keep the SSE-C key fingerprint on keyless multipart completion The backlog#3050 fix withheld both the plaintext checksums and the customer-key MD5 header from a CompleteMultipartUpload that did not present the key. The checksums are a digest of the plaintext and stay hidden; the key MD5 identifies the key only, and the S3 conformance suite (test_copy_part_enc) expects it on a keyless completion, so it is echoed again. * fix(tier): stream GET bodies and document replication budgets * fix(ecstore,s3): keep the multipart index and cold fill off disk lifetimes Two lifecycle regressions surfaced once the storage and request batches ran together against main: - The multipart index owner cloned the disk's pinned publication root and bootstrapped on a task of its own. That kept the mount alive past the last LocalDisk owner (the disk lifecycle allows exactly three holders of the root FD) and left a stray task behind, which the local-disk and read-repair tests detect. The owner now keeps only the root path and reopens it for identity checks, and the bootstrap scan rides the disk's existing cleanup task, where stopping the disk abandons it and a reconnect resumes it. - The disk-read admission wrapper defers releasing its permit until an in-flight read completes, so a cancelled client cannot widen disk concurrency. The cold-fill producer, however, drops a read it has already classified as stalled when its deadline fires, and that permit stayed pinned for the lifetime of the stuck read. The producer now abandons the admission explicitly on timeout. * test(ecstore): pass the index owner to the Linux cleanup-loop test * fix(s3): release cold-fill admission after watchdog handoff * fix(s3,ecstore): close the round-1 adversarial review findings Adversarial review of the combined request- and storage-layer batch found fixes that fell short of their issues, two behaviour regressions for legitimate clients and several unrequested costs: - backlog#3038: a bulk DeleteObjects reserved the whole per-disk inspection allowance up front for every key in flight and exhausted the 16 Mi request budget on any set wider than eight drives, failing healthy keys with SlowDown. Calls are now admitted while budget remains and charged for the entries they actually visited. Object-scoped discovery charges plain child keys at one entry per 64 names instead of one each, so deleting a missing key that names a folder prefix completes instead of answering 503 forever while a hostile prefix stays bounded. A peer still on the previous binary is reported as an unsupported participant, the repository's rolling-upgrade contract, instead of as a budget verdict. - backlog#3037/#3026: a SigV2 configuration PUT without Content-MD5 was still replayable with another body; it is refused with MissingContentMD5 while SigV4 keeps the optional-MD5 contract. - backlog#3072: DeleteObject now resolves x-rustfs-source-version-id before authorization like PUT and COPY, so a replica delete cannot remove a version the policy never saw. - backlog#3121: the multipart index no longer fsyncs a marker and four directories per multipart metadata write (it is rebuilt on every start), and ListMultipartUploads discovers roots concurrently. - backlog#3077: Parquet SelectObjectContent decodes budget-sized batches instead of one row per batch. - backlog#3031: a full upload buffer pool hands out an unpooled buffer instead of refusing the upload; the progress floor already bounds stalls. - backlog#3023: the GET disk-read idle cut-off is configurable (RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS, default 120 s) so a paused client is not truncated after 30 s. - backlog#3075: JSON Lines schema inference keeps the sample collected at the 4 MiB bound instead of failing the query. - backlog#3024: compat admin decryption queues briefly for a slot instead of refusing the third concurrent request. - backlog#3113: HEAD skips its permission probes for the bucket owner. - backlog#3117: control characters in a lifecycle rule ID are rejected on PutBucketLifecycle as well. The lock-crate publication FIFO serves backlog#3111 and moves to the table-catalog PR. Also fixed: a lossy admin error message, an unused s3-client function, two unused test-util features, a duplicated key validation, the s3s footprint baseline, and a stale log event name in the reverse-proxy runbook. The ODM multipart write-back test runs on the large test stack like its sibling. * fix(table-catalog): bound maintenance work and isolate table namespaces (#9174) * fix(table-catalog): bound maintenance work and isolate table namespaces Scan findings in the Iceberg table catalog: - Metadata maintenance deleted every unreferenced *.metadata.json in a table's directory without checking whether a renamed table in another namespace still owned those files (backlog#3060); it now resolves the owning table through the directory index before deleting. - Manifest lists and manifests were decoded with the embedded Avro writer schema and no recursion depth limit; a self-referencing record schema overflowed the stack (backlog#3080). Schemas are validated for depth before decoding. - Compaction decoded every input Parquet file into Arrow batches held in memory; a decompression bomb OOM-killed the server (backlog#3085). Row groups are streamed through a byte budget and oversized inputs are refused. - An S3 request for a key outside an indexed table prefix listed the bucket's whole namespaces/ catalog prefix and failed once it held more than 4096 objects (backlog#3086); a per-bucket directory index answers index misses without the listing. - The maintenance planner read whole orphan files to obtain their modification time (backlog#3087); it reads metadata only. - A client disconnect between persisting active_rename_id and clearing it left the bucket's catalog entry locked at 503 (backlog#3088); the rename runs detached from the request future and clears the flag on every exit. - Table commits held the bucket publication fence until publication, starving every object write (backlog#3111); commits now hold it under a deadline and object writes are admitted between commits. The directory index is managed through three new admin routes under {warehouse}/catalog/directory, registered in the admin route policy. The durable-strong store is left as is (its snapshot partitioning for backlog#3109 and the strong-path deadline are not ported): that backing is being retired separately, so the new tests cover the object backing. * fix(table-catalog): close the review gaps in the scan-fix batch Adversarial review of the table-catalog batch found the fixes narrower than the issues they close and two cost regressions: - backlog#3085: the compaction footer check bounded nothing a page header could not override; parquet sizes its page buffers from the header. Walk every column chunk's page headers (Thrift compact, no decompression) and hold each page and the chunk total to the budget before decoding. The stored-input budget is per rewrite group, so a multi-group job no longer fails after writing earlier outputs. - backlog#3086: the default index-miss scan still counted maintenance reports toward its 4096-object limit; only table entries count now, and the listing itself is bounded to 64 pages. The directory index remains an optional accelerator. - backlog#3111: a create wrote its metadata file under the fence as the first durable mutation, which disarmed the deadline before the unbounded registration inventory. The write precedes publication now. The deadline gets its own knob (RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS, default 30 s, never below the lock-acquire timeout) instead of reusing the 5 s lock timeout, and the ownership inventory reads entries concurrently. - Directory candidates are written only while a directory exists; the backfill covers earlier entries. A directory on the migration-fence path is reported as a conflict instead of a parse error. - Dead test hooks, a stale comment and the support-matrix text that did not match the code are removed or corrected. * fix(select): preserve complete samples and Parquet batch limits * fix(s3,ecstore): close the round-2 adversarial review findings DeleteObject resolved x-rustfs-source-version-id for authorization on every request while storage consumes it only for a REPLICA request, so a non-replica caller with s3:DeleteObjectVersion could trigger a plain delete of the current version. The hook now resolves the header under the same predicate as storage, and the regression test covers the non-replica case. HEAD no longer skips its permission probes for the bucket owner; an explicit bucket-policy Deny on the metadata actions binds the owner too. Multipart index pages verify each root against the staging area and prune markers whose staging root is gone, under the marker's own lease with a re-check, so the per-drive index stays proportional to live uploads instead of every key ever multipart-initiated on the drive. SigV2 configuration PUTs that declare an empty body need no Content-MD5: there is no body to replay. The minimum-progress floor stays independent of RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT; the knob's documentation now says so instead of claiming 0 disables it. The quota bootstrap window (503 until the background scan publishes the ledger) is kept, because ledger admission must not come from scanner reporting; the compatibility matrix now documents it. The S3 compatibility matrix gains a section for the client-visible validation bounds, and the drive-timeout tuning guide tabulates RUSTFS_GET_DISK_READ_IDLE_TIMEOUT_SECS. * fix(lock): queue table-bucket publication writers fairly Table commits take the bucket publication fence as a write lock; under a stream of commits the lock's try-and-retry acquisition let a new commit jump ahead of object writers that had been waiting longer, which is part of what starved them (backlog#3111). Requests for the publication lock path now take a FIFO ticket per shard: a waiter is admitted only when it is at the head of the queue, cancelled waiters release their turn, and batch acquisitions join the same queue. Other lock paths are unchanged. This moved here from the request-layer batch, where no cited issue covered it; it serves the table-catalog issue this PR closes. * fix(table-catalog): close the round-2 adversarial review findings Parquet page-header walker: read the size fields by field id as parquet does, accepting I16/I32/I64 wire kinds and rejecting any other kind, so a header encoding uncompressed_page_size as I64 can no longer bypass the budget while parquet allocates it. Bound every list, set and map count by the bytes remaining in the chunk, and reject bool collection elements (parquet skips them with zero width, so either width would desynchronize the two parsers); this removes the 2^63-iteration spin on LIST<BOOL>. Start each column-chunk walk at the dictionary page offset whenever the footer declares one, exactly as parquet's byte_range does, instead of min(dictionary, data). Footer pre-walk: before parquet decodes a footer, walk FileMetaData with the hardened cursor and charge the declared row_groups count times the RowGroupMetaData size to the decode budget, because parquet 59 reserves that Vec before reading one element. Preparation deadline: clamp the RUSTFS_TABLE_CATALOG_PUBLICATION_PREPARATION_TIMEOUT_SECS knob to 86,400 seconds so Instant + Duration cannot overflow. Index-miss scan: after each listing page, jump the cursor past the maintenance subtree the page ended in, and allow one extra page per table entry already listed, so job-report history scales with the catalog while a report flood under one table stays bounded. Migration fence: read the fence-path object once and decide from its stored kind whether it is directory state or a fence. Expired or failed create: remove the initial metadata file when the registration fails before any catalog write may have begun; nothing else references it. Correct the comment and support matrix that claimed maintenance reclaims it. Tests cover each walker divergence, the footer row-group reservation, the deadline ceiling, the paging budget for honest catalogs and for a report flood, and the removal of the unpublished metadata file. * test(admin): register the catalog directory routes in the route matrix The table-catalog batch added GET, POST and DELETE {warehouse}/catalog/directory on both the Iceberg and the compat admin prefixes without listing them in the admin route matrix, so test_admin_route_matrix_matches_registered_routes failed on CI. * fix(s3,ecstore): close the round-3 adversarial review findings Multipart index pages now treat the cursor as the last examined marker rather than the last root, so a page that pruned a stale marker still validates and ListMultipartUploads keeps using the index. A marker is written before its staging root is published, so pruning spares markers younger than a five-minute grace period and an existing marker is touched on every new upload; the prune re-checks both under the marker lease. Bounded delete-recovery inspection rides the durable-mutation lane like the unbounded path instead of the one-worker foreground lane. The on-demand-migration inline body budget scales with the object size (at least 64 KiB/s) instead of a flat 120 s. The control-body digest service honors RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT=0 as "no inactivity timeout" like every other reader of that knob. Quota reconciliation throttle objects are deleted together with a retired generation's ledger, so bucket turnover leaves no residue. The PUT guard test shim is gone; tests call the shared guard. Bounded DiskAPI defaults answer MethodNotAllowed like their siblings. Stale budget docs, an unreachable CSV arm, an always-Some limit parameter, commented-out constants and a misplaced doc comment are cleaned up; weak test assertions name the expected error; lifecycle rule-ID control characters gain a test. The compatibility matrix records the CopyObject tag-copy permission, the exact SigV2 empty-body predicate, replica and migration writes of control-character keys, and the SigV2 DeleteObjects digest rule. * fix(table-catalog,heal): close the round-3 review findings A. Pack compaction rewrite groups to min(max-rewrite-bytes-per-job, TABLE_COMPACTION_MAX_STORED_BYTES), reject an over-ceiling group before any output is written, and name the ceiling in the commit-path errors; document the effective cap in the support matrix and scheduler runbook. B. Keep TaskCancelled/TaskTimeout classification on replacement passes: the namespace verification runs only for completed or storage-failed passes, in the erasure healer and the task bucket prepass. C. Compare the replacement bucket pass per bucket: a recorded identity that is gone or replaced still errors or restarts the pass, buckets created mid-pass are appended to the identities, pending_buckets and a persisted positional order without rewinding progress or the checkpoint; verification ignores additions. D. Read the migration-fence object once in the object-backed write permit and dispatch on its kind. E. Drop the strong store's copy of maintenance_cleanup_roots. F. Remove the unused test hooks fail_put_prefix, corrupt_put_prefix and pause_after_metadata; make compact_parquet_data_files and PublicationTicket.state private. G. Document the body-revision identity, the legacy revalidation budget and its HTTP 503 in the support matrix. H. Define HEAL_REPLACEMENT_RECOVERY_DIR once in ecstore and consume it from heal through the storage_api facade. * fix(ecstore): cap the index pages one disk may answer Pruned multipart index pages may carry few roots, so the page cursor no longer bounds how many pages a peer can make a listing fetch. Cut a disk off after 4096 pages (about one million examined markers) so it contributes no coverage and the listing falls back to the staging scan. * fix(ecstore,table-catalog): close the round-4 adversarial review findings Multipart index markers dated in the future (a backwards clock step between recording a marker and publishing its staging root) were judged old and could be pruned while their upload was in flight; a future-dated marker is now kept. The marker touch on EEXIST is best effort, so a filesystem that refuses to set times no longer fails the upload commit. Pruning runs on the durable-mutation lane instead of the one-worker foreground lane. Quota reconciliation keeps one throttle object per bucket incarnation (a quota revision change replaces it instead of orphaning the previous revision's object), the retirement cleanup deletes it ahead of the ledger and also when the ledger is already gone, and a scan the retirement already refused writes nothing back. Bounded DiskAPI defaults answer UnsupportedDisk, the verdict the aggregate gives a peer without the protocol. The compaction ceiling pre-check runs over every rewrite group before any output is written, and the worker's ceiling error no longer names a placeholder group. Docs: the inline on-demand-migration client budget, the protocol-2 downgrade conflict of replacement passes, the exact footprint numbers. |
||
|
cxymds
|
8a5a2c9c93 |
fix(s3): treat duplicate CreateBucket as owned by the requester (#9167)
* fix(s3): treat duplicate CreateBucket as owned by the requester
The duplicate-bucket response was decided from the requester's root-owner
flag, so every IAM user recreating a bucket received 409
BucketAlreadyExists, including a user recreating a bucket it created.
BucketAlreadyExists means the bucket belongs to another account; a
deployment owns a single namespace, so the answer is purely region-based:
the default region returns the legacy 200 OK, any other region returns
BucketAlreadyOwnedByYou.
Both the existing-bucket fast path and the BucketExists path now share one
helper. Authorization still runs first, so a requester without
s3:CreateBucket gets AccessDenied and cannot probe bucket existence.
Refs #8493
* test(ci): refresh e2e selections for duplicate CreateBucket
The two new create_bucket_region cases join the e2e-smoke and e2e-full
selections, which main also extended with multipart_upload_id tests.
smoke is measured from a local nextest listing: removing the two new cases
reproduces main's committed digest exactly. e2e-full is derived the same way
from main's listing artifact (run 41933823399,
|
||
|
Jose Meira
|
736bf777b4 |
feat(container): add declarative bucket bootstrap (#9162)
* feat(container): add declarative bucket bootstrap * fix(container): handle false bucket versioning setting * Update Rust base image to version 1.99.1-trixie Signed-off-by: Hauser Zhi <housemecn@gmail.com> * Update Rust base image version to 1.99.1 Signed-off-by: Hauser Zhi <housemecn@gmail.com> --------- Signed-off-by: Hauser Zhi <housemecn@gmail.com> Co-authored-by: Hauser Zhi <housemecn@gmail.com> |
||
|
Hauser Zhi
|
41b13aeb8b |
fix(tls): unify FIPS provider policy across clients (#9125)
* fix(tls): enforce active FIPS provider at startup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci: provision FIPS validation runner Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): validate FIPS before observability startup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(tls): reuse the installed crypto provider Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): gate Linux-only commit helpers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(tls): centralize FIPS provider policy Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: Chris An <anzhengchao@gmail.com> |
||
|
hector
|
8c8b5da63f |
ci(nightly): dispatch nightly-candidate-ready after asset publish (#9131)
After the nightly packages are on the auto-testing nightly release and the schema 2 R2 candidate exists, post repository_dispatch to rustfs/auto-testing. Vault lanes stay parallel, and the workflow_run functional chain is left in place. Co-authored-by: hector <majinghe@users.noreply.github.com> |
||
|
Chris An
|
2d63931bba | fix(test): use available Docker privileges in OIDC live gate (#9109) | ||
|
hector
|
e494dc2b4b |
ci(validation): sweep the migrated canonical validation cases nightly (#9107)
* ci(validation): sweep the migrated canonical validation cases nightly The canonical release-validation cases migrated into auto-testing by rustfs/auto-testing#113 (66 implemented, all non-heal) currently run only through the manual validation flow; the last fleet executions stopped on 2026-10-27 and nothing schedules them. Add a scheduled validation sweep that closes the gap: - nightly at 19:30 UTC (after the functional chain settles), rotating windows of the implemented catalog cases so the whole catalog is covered roughly weekly, with a bounded batch per night - workflow_dispatch for a single case (--case), one module (--module), or an explicit batch (--limit), mirroring how the lanes are testable standalone - fleet admission through functional_environment enter/finish with the same quarantine discipline as the lanes, inter-case node cleanup, per-case logs + reports retained as a sweep artifact - a markdown sweep report uploaded to rustfs/dashboard under functional-reports/validation/ so the cases appear next to the lane reports, plus an alert-on-failure job for scheduled runs - heal.* cases stay on the interactive validation flow: many need the reviewed native/tool collectors built by separate validation-tools runs before fleet execution The registry gains the workflow responsibility (monitor: workflow, max-age 60h so the watchdog notices a dead sweep) and the wiring test baseline count moves with it. Verified locally: check_wiring, check_test_wiring, test_validation_registry (9/9), test_functional_chain + test_functional_chain_health + test_security_workflow (99/99). * ci(validation): group the sweep report by catalog module The validation cases cover distinct areas (pool lifecycle, ILM expiration, bucket/site replication, kms, data services, identity, s3 compatibility), so a flat case table hides the category breakdown. Render one section per catalog module with its own case table and a Subtotal line, and keep the single grand - Total:/- PASS:/- FAIL: block for the dashboard overview parser. Subtotals deliberately use a different bullet prefix so the overview regexes cannot mistake a module subtotal for the grand total. * ci(validation): sweep the eight planned cases whose executors exist The planned catalog entries are not all unimplemented: all eight (kms.sse-kms-round-trip, tiering.transition-and-restore, events.delivery, audit.request-correlation, iam.policy-enforcement, oidc.login-and-session, s3.basic-object-round-trip and scanner.usage-convergence) have dedicated executors routed in run_validation_case.sh and live PASS evidence recorded in the migration map. Widen the sweep filter to include them: the rotating nightly batch now covers 74 cases (66 implemented + these 8), and the fresh evidence each sweep produces feeds their acceptance review. heal.* procedure-ready cases remain excluded - their executors need the reviewed native/tool collectors built by separate validation-tools runs before fleet execution. --------- Co-authored-by: Hauser Zhi <housemecn@gmail.com> |
||
|
Chris An
|
1cb4c0ebe2 |
ci: stamp the plan contract and rename the workspace lint-and-build key (#9095)
* ci: gate the plan on a contract marker A base planner predating the NIGHTLY_JOBS split still emits the old required set, and the strict verifier would reject it. Select now stamps plan_contract=2 and the workflow refuses any plan without the current marker, falling back to the full matrix, so a stale base planner can neither authorize the new narrower selection nor force a job the new workflow no longer schedules. * ci: rename the workspace lint-and-build job key and evidence path The job key test-and-lint named a job that runs no tests: it does Clippy, the test-binary archive, doctests and the log-rule anchors, while the tests live in workspace-tests. Anyone reading needs: [test-and-lint] would conclude tests had already gated that dependency. The key becomes workspace-lint-and-build to match its display name, and the shared evidence directory becomes artifacts/workspace so the path stops asserting the same thing. The aggregate's display name stays Test and Lint because that is the live ruleset check context. ci-gates.md also stops attributing the ECStore test run to the wrong job. |
||
|
Chris An
|
e5563e6004 |
feat(release): gate publication on candidate certification (R01) (#9090)
Add a fail-closed release certification path so incomplete or substituted evidence blocks every official publication exit. scripts/release/release_gate.py validates the versioned release-validation plan, the candidate identity and the G01 evidence-envelope set, then re-checks promoted bytes against the passed list. build/docker/package/helm publication jobs require that certification and stage artifacts before promoting identical bytes. |
||
|
Chris An
|
51275cb7d8 |
ci: move specialized lanes off the pull-request path (#9093)
Connect Feature Boundaries, real io_uring and the rio-v2, Swift, SFTP and HTTP/3 feature lanes no longer gate a merge. They run only in the scheduled and manual full-validation runs, which report a regression within a day; on the pull-request path they bought no earlier signal worth the runner occupancy (Connect alone measured 53.5 median minutes and occasionally set the wall). Workspace lint and tests, the short core contract, the debug build, E2E smoke and the S3 lanes stay required on PRs and merge groups. |
||
|
Chris An
|
3bf270b49f |
ci: record CI01 measurements, drill evidence and fix the OIDC live gate (#9091)
* ci: retain CI01 timing sample and fix the OIDC live gate tool Record the measured PR feedback and runner occupancy for the CI plan's CI01 criterion, including the honest gap between the experimental targets and the sampled medians, and replace the awscurl dependency in the live Keycloak gate with the AWS CLI so the security suite runs on PEP 668 managed fleets. * docs(testing): attach live run evidence to the event-drill index |
||
|
Chris An
|
0e5fe8c658 |
feat(storage): share physical admission across scanner and heal (#8911)
* feat(scanner): share node admission with background heal * fix(scanner): drain candidates after listing completion * fix(scanner): yield cache leases for shared node admission * test(scanner): attest adaptive admission in process receipts * fix(concurrency): declare Tokio time dependency * feat(concurrency): retain bounded physical resource subsets * fix(scanner): retain resource ownership in native filesystem jobs * feat(concurrency): acquire physical groups before node work * feat(scanner): gate background reads by physical device * feat(concurrency): combine preallocated resource selections * feat(scanner): share physical topology across disk plans * fix(storage): retain only native disk resource subsets * feat(storage): admit auxiliary physical resources without waiting * feat(heal): admit physical resources before object repair * fix(storage): retain resources during native format probes * fix(storage): retain resources through direct writer I/O * fix(storage): retain resources through buffered writer I/O * fix(storage): retain physical resources through streamed reads * fix(storage): retain resources through native filesystem jobs * fix(storage): retain physical ownership during bitrot reads * fix(ecstore): retain resources during positioned reads * fix(ecstore): retain staged metadata write ownership * fix(ecstore): retain native io_uring read ownership * fix(ecstore): enforce receiver-side background admission * fix(ecstore): retain resources through whole-file native I/O * fix(ecstore): retain physical ownership during durable rename sync * fix(storage): account for retained background stream buffers * fix(storage): retain memory charges for background read results * fix(storage): account for native io_uring read allocations * fix(storage): retain memory charges for direct-write buffers * fix(ecstore): account for heal and RPC reader buffers * test(scanner): verify samples and clean concurrent collectors * fix(storage): retain inline and HTTP writer memory reservations * perf(rio): reuse reserved HTTP writer storage * fix(rpc): reserve background scanner pipe storage * perf(rpc): reuse reserved response reader storage * feat(scanner): measure cache copy allocation requests * fix(rpc): budget namespace scanner responses * fix(scanner): retain the enabled supervisor task handle * fix(scanner): budget namespace frame read staging * fix(scanner): account namespace envelope decode storage * fix(heal): retain and join the MRF consumer on shutdown * test(heal): isolate process-global MRF consumer fixture * fix(heal): join manager workers during shutdown * test(heal): verify shutdown at native read boundaries * fix(heal): drain channel commands before stopping MRF * fix(heal): admit resume scans and retain replacement workers * build: pin profiling sources and trust fixed ownership dependencies * fix(heal): yield node admission while accepted native work drains * fix(startup): retain shutdown work and preserve deadline failures * fix(heal): admit inventory probes and retain replacement work * fix(shutdown): preserve maintenance and frontend join failures * fix(shutdown): retain HTTP child tasks through real retirement * fix(shutdown): join actual TLS reload work before retirement * test(multipart): preserve failures before fault barriers * fix: keep physical admission inside repository guards Route the multipart lock diagnostics filter through the e2e storage compatibility module, reuse the parent module's s3s body aliases in the namespace memory test, and share one Reed-Solomon verify error mapping across the reserved-buffer and default paths. * fix: reconcile physical admission with main's new call sites Main added a source-admission metadata write layer, a CAS write fence, fixture writers and scanner handoff tests against the old borrowed-slice and node-permit shapes. Pass owned Bytes through the new metadata layer, answer the scanner handoff with ScannerNodePermit, gate the test-only disk lookup re-export, and clear the lint findings that stable 1.99 reports for the shutdown and admission code without suppressions. * test(concurrency): name reversed attachment groups clearly * fix(ecstore): box disk operation futures below admission layers Every LocalDiskWrapper operation reaches the LocalDisk future through the health, admission and timeout layers (track_cleanup_mutation, the track_disk_health family, the io-permit and native-resource admission split, and track_admitted_disk_operation). Each layer embedded that future, and unoptimized builds copy it into every awaiting frame, so one local delete cost about 1.3 MiB of stack and 16 tests overflowed libtest's default 2 MiB stack. Box the operation future once where track_admitted_disk_operation creates it, through a non-async helper so the temporary never stays in a polled frame. The layers above now hold only a pointer; the cost is one allocation per local disk operation. * fix(heal): box object heal futures below admission The admission wrappers took the object heal future by value, so unoptimized builds copied it into three awaiting frames, about 650 KiB per heal. Create it through a non-async helper instead, leaving only a pointer in those frames. * test(ecstore): count the I/O backend's root FD owner |
||
|
Chris An
|
7d45685d70 |
feat: classify scanner metadata calls by diagnostic prefix (#9052)
* feat: classify scanner metadata calls by diagnostic prefix * fix: export zero scanner visit counts for diagnostic scope |
||
|
Chris An
|
76bab81aa6 |
fix: reject unverified scanner performance evidence (#9049)
* fix: reject unverified scanner performance evidence * fix: keep unverified G14 descriptors unpublished * test: release scanner fixtures without signaling stale pids |
||
|
Chris An
|
bc0706b2e3 | fix(ci): accept binary target IDs in workspace shards (#9075) | ||
|
hector
|
589b499e3b |
ci(ft,heal,chain): FT gate 54, heal budget, and a temporary performance lane disable (#9057)
* ci(ft): update the lane gate to the 54-case manifest with scenario F auto-testing #118/#119 added FT scenario F and case B5, so the lane now runs 54 cases over scenarios A,B,C,C2,D,E,F and passes 54/54 — but the workflow gate still pinned the 44-case A-E manifest and failed the report step. Align the gate and the contract-test fixture with the 54-case manifest. * ci(heal): give heal a 90-minute run budget inside a 120-minute job The heal task was still progressing (scanned 494 -> 571+, failed=0) when the 45-minute step cap killed it, so the verification steps never ran and the lane produced invalid evidence (chain 40796474356). The step budget is now 90 minutes, and the job-level budget rises from 60 to 120 minutes so the enclosing job cannot cancel the raised step budget. The timeout contract now records heal as the exception to the shared 60/45 baseline. * ci(chain): force the performance probe offline while pf-testing is broken The pf-testing environment is temporarily broken, so every nightly chain would end red in the performance lane. performance is required=false in scheduled-validations.json, so forcing the probe's performance_ready output to offline takes the chain through its designed skip path: the lane job is skipped, aggregate receives --allow-skipped performance, and the report records the optional-lane skip with its reason. Marked TEMPORARY; restore the probe once the environment is healthy again. * test(table): drop a stale workflow-comment pin left by #9059 #9059 rewrote the table suite step comments but kept the assertion for the old '# Red only for harness/environment breakdowns' line, so test_security_workflow.py fails against current main. The semantics stay pinned by the continue-on-error and HARNESS_OK assertions. |
||
|
Chris An
|
191c95a469 |
ci: enforce affected validation and rehearse bounded shards (#9070)
* ci: enforce affected validation and rehearse bounded shards * test(ci): keep regression output stable under forced color |
||
|
Chris An
|
dacb003856 |
ci: let failed-job reruns reuse the producer's test archives (#9071)
Since #9059 the shared test archives and debug binaries are named with the run attempt, and the archive identity binds the attempt. A "Re-run failed jobs" attempt does not rerun the successful producer, so its consumers look for <name>-2, fail with "Artifact not found", and every flaky test forces a full rerun. Give those artifacts one name per run, overwrite them when a producer does rerun, and bind the archive identity to the run instead of the attempt. Source, tree, lock, toolchain and environment stay bound, and an archive from another run is still rejected. |
||
|
Chris An
|
c67f9fc377 |
ci: qualify workspace packages that share a name with a dependency (#9065)
The planned package list is passed to cargo as bare -p names. The zstd compat crate is named zstd so it can patch crates.io, and with zstd 0.14.3 also locked, cargo rejects -p zstd as ambiguous, failing Workspace Lint and Build on main since #9059. Qualify such names with the local version from Cargo.lock, and fail loudly if even that cannot disambiguate. |
||
|
Chris An
|
4fcbf60f10 |
ci: unify validation planning, artifacts and suite orchestration (#9059)
* ci: unify validation planning, artifacts and suite orchestration * fix(ci): preserve JSON diagnostics for release checks * fix(ci): keep manual attempts without plans visible * fix(ci): distinguish nextest setup scripts from test results * ci: reuse extracted archives and consolidate duplicate checks |
||
|
Chris An
|
b7ff09e357 | chore: strengthen agent delivery standards and validation (#9058) | ||
|
Hauser
|
e0444b519d |
feat(ecstore): add sampled batch CPU diagnostics (#9019)
* feat(ecstore): add sampled batch CPU diagnostics Keep timing and request correlation in a separate default-off feature. Preserve the existing positioned-read backend, byte and error contracts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route batch diagnostics through storage facade Keep the diagnostic implementation private to its owner and expose explicit feature-gated helpers through the existing storage boundary. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retain diagnostic IDs across GET producer tasks Capture request context before spawning the legacy producer. Keep the existing producer, cancellation and read-lock lifetime intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): avoid unused batch read index Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(obs): isolate local batch diagnostic JSON events Omit span metadata and inferred parent IDs for diagnostic root events. Verify per-shard index order and document existing production metric timing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): accelerate receipt fixture in CI Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): allocate stack for bucket recovery fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(s3): allocate stack for conditional GET fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
4a63e53161 |
test(validation): run canonical cases and build pinned Heal tools (#8455)
* test(validation): add pinned auto-testing manual lane * Install pinned ILM client for canonical validation lane * Prepare Python client for pool validation cases * Install validation client for replication cases * test(validation): provide the Python client for Heal cases * test(validation): build the candidate-bound Heal inspector artifact * test(validation): expose tool artifacts through the existing manual lane * test(validation): verify pinned Heal tools before fleet execution * test(validation): verify Heal tools against released shard captures * ci: audit retained validation environments before finalization * test(validation): add optional native Heal regression tooling * test(validation): route native tools through the existing driver * Run validation tool resolution on hosted Linux without acquiring the fleet lock * test(validation): bind C07 to pinned native authority evidence * test(validation): build optional C10 fixture codec on hosted Linux * test(ci): bind C10 execution to the verified shared codec * feat(ci): expose read-only validation fleet inventory * docs(ci): describe supported validation recovery * test(ci): add isolated manager native validation builds * test(ci): expose manager regressions through tools-only dispatch * ci(validation): consume pinned manager native prerequisites * fix(heal): preserve typed bucket errors for bounded retries * fix(heal): preserve typed errors and SSE-C response headers * ci(validation): publish isolated immutable candidate packages * test(validation): enforce private candidate manifest schema * test(validation): verify isolated package publication failures * ci(validation): dispatch isolated packages through the driver * fix(heal): deeply verify read repair payload failures * test(ecstore): verify incarnation fence at repair commit * ci(validation): verify C11 native storage evidence * fix(ci): isolate validation package publication credentials * test(ci): archive scoped F07 directory fsync evidence * fix(ecstore): retain typed quorum errors for missing parts * chore(ecstore): shrink typed-error ratchet baseline * fix(ecstore): retain below-quorum data during no-remove heal * fix(ci): allow recovery of completed cancelled validation runs * fix(heal): retain exhausted admin batch repair receipts * fix(observability): retain typed object error diagnostics * fix(s3): map typed RPC unavailability to service refusal * fix(filemeta): classify supported inline framing damage as corrupt * test(ecstore): exercise heal during committed overwrite cleanup * ci(validation): admit source-bound G01 native evidence before fleet execution * test(ecstore): fence version races and bind cleanup slots * ci(validation): require complete race native runtime preflight * test(ecstore): fence marker and batch deletes during heal * ci(validation): require G04 native and runtime admission * test(ecstore): fence multipart continue complete and abort during heal * test(ecstore): cover default heal owners and copy metadata races * ci(validation): admit source-bound owner and copy race proofs * test(scanner): assert typed corruption after metadata truncation * fix(ci): require complete native admission for multipart races * test(validation): wire I05 queued-successor native prerequisites * fix(scanner): bind object heals to the scanning disk scope * test(scanner): reuse storage boundary for scope regression * style: format RPC digest precedence assertion * fix(heal): query automatic tasks on their owning node * test(admin): reuse error facade in heal query regression * fix(heal): retain cancellation after successful bucket storage I/O * fix(ci): retain validation evidence outside runner temp * fix(heal): retain resolved object identity on terminal failures * test(scanner): adapt scope regression to admission receipts * ci(validation): inspect shared runner without fleet access * ci(validation): prepare Python before creating client environments * ci(validation): isolate each client virtual environment * ci(validation): retain interrupted evidence without fleet recovery * ci(validation): bind recovery to retained runner evidence * ci(validation): wire persistence and lock source plans * fix(validation): align read-only dispatch defaults --------- Co-authored-by: Chris <anzhengchao@gmail.com> |
||
|
Hauser
|
9601a7c4bc |
fix(admin): protect notification target edits (#9048)
* fix(admin): protect notification target edits Use a durable configuration revision and conditional mutation to preserve concurrent changes and inherited target settings. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): use the gateway error boundary Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(admin): limit target detail health probes Reuse normalized source information and probe only runtime targets matching the requested instance. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(admin): streamline target config validation Preserve duplicate-key order when comparing snapshots, avoid copying secret values, and reuse the validated target configuration. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
3e056ffb4d |
fix: keep incomplete scanner measurements unknown (#9044)
* fix: keep missing scanner error rates unknown * fix: keep incomplete pressure samples unknown |
||
|
Chris An
|
c232f1552d | fix(metrics): reject missing histogram counts (#9039) | ||
|
Chris An
|
8e88419f3a |
feat(scanner): share node admission with background heal (#8903)
* feat(scanner): share node admission with background heal * fix(scanner): drain candidates after listing completion * fix(scanner): yield cache leases for shared node admission * test(scanner): attest adaptive admission in process receipts * fix(concurrency): declare Tokio time dependency * fix(scanner): route test disk access through owner facade * test(scanner): fix lint warnings |
||
|
Chris An
|
f879946ead | chore: remove redundant review skills (#9022) | ||
|
Chris An
|
54ac37ec0a |
test(decommission): cover held target lock and partial-copy resume (#9016)
* test(decommission): cover held target lock and partial-copy resume Port two tests from the closed lock-recovery branch onto the deferral that defers any typed namespace lock timeout on a version copy. - A version commits, then every attempt at the next one times out on the target lock: the entry is deferred as a lock-timeout deferral, no failed item is counted, both source versions stay, and the resumed entry moves both versions with their exact bytes. - A real write lock held on the target key makes target selection return a typed object-lock timeout, the entry defers instead of failing, and once the lock is released selection recovers and the resumed entry completes the move. * chore(ci): shrink the error(format!) ratchet baseline for rebalance The rebalance control and meta modules on main now have fewer Error::other(format!) call sites than the baseline records, which fails the shrink-only ratchet in Quick Checks. Record the smaller counts. |
||
|
Hauser
|
b9f01162b2 |
feat(tools): inspect BTF scalar member layouts (#9003)
Report selected integer offsets and widths with strict member and payload validation. Cross-check compiler fixtures with bpftool and independent C offsetof/sizeof results. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
be485e6cab |
test(scanner): add memory acceptance runner and budget probes (#8988)
* test(scanner): add memory acceptance runner and budget probes Add owned S3 workload phases with process, endpoint, cgroup and binary identity checks for scanner memory and scanner-disabled request bursts. Keep scale, restart, heap-attribution and overall-acceptance limits explicit. Characterize healthy and seeded coverage-gap cache retention through the real folder walker and production cache codec without changing runtime code. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): bind acceptance capacity to effective storage Validate the serving process's complete local disk inventory before load writes and during runtime sampling. Resolve each disk inside the owned storage root, reject nested mounts and filesystem mismatches, and bind storage observations to phase identity for container and native runs. Cover the unused-mount bypass, CLI-overridden environment, native paths, symlink escapes, named volumes and stable multi-disk inventory ordering. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): fence status observations before workload writes Close each status API window with a second storage-bound process sample before Phase entry can authorize bucket creation or PUT requests. Add a fail-before regression for a process change during the initial status request, asserting no ownership or workload writes occur. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
254592717a |
feat(tools): add read-only BTF prototype inspection (#8985)
* feat(tools): add read-only BTF prototype inspection Expose bounded declared prototype headers through a standard-library CLI, preserve explicit void and failure semantics, and wire local contracts plus an independent compiler/bpftool reference into CI. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(tools): retain native reference failure diagnostics Clear checkout credentials and capture bounded native stderr alongside stdout, preserving exact exit status and owned child cleanup. Keep the first reference failure unresolved rather than guessing its missing diagnostics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci(tools): select a concrete bpftool file parser The hosted PATH wrapper can exist without tools for its Azure kernel. Select an executable ELF by availability/version before comparing the fixed fixture, installing generic tools only when needed. Data comparison errors remain fatal and never trigger tool switching. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
hector
|
26d55bb04e | ci: remove never-green Performance A/B workflow (#8979) | ||
|
Chris An
|
4a96f06983 |
refactor(listing): remove env-gated ListObjects index provider (#8948)
* refactor(listing): keep paginated listing on ECStore only `Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`, `list_objects_generic`, `inner_list_object_versions`, `list_path` and `list_merged`, plus a third copy of the task that converts merged raw entries into the walk result stream. Only `ECStore` can answer a paginated listing, since a page merges every set of every pool; the copies drifted (the `Sets` copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and were reachable only from tests and one decommission helper. Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the duplicated pagination code. `SetDisks::walk` stays as an inherent method for set-scoped callers and now honors `versions_sort` like the store-wide walk; `SetDisks::list_path_result` stays for the recursive delete lister. The walk entry conversion moves into one `spawn_walk_entry_forwarder` shared by both walks. The decommission receipt lister walks its pool's sets instead of paginating through the removed `Sets` listing. Tests that exercised the removed layers now exercise `ECStore` for pagination or `SetDisks::walk` for per-pool version inspection. The architecture and logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry points are dropped, and the error-other ratchet baseline shrinks by one. * fix(listing): exclude synthetic directories from receipt walks * refactor(listing): remove env-gated ListObjects index provider Delete the experimental, default-off key-only / verified-page / metadata-fast index provider from `ECStore::list_objects_generic` together with everything only it drove: the `RUSTFS_LIST_OBJECTS_INDEX_*`, `RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_*` and `RUSTFS_LIST_OBJECTS_METADATA_FAST_*` readers, the persistent key-only index and its on-disk format, the namespace mutation journal (state, chaos injector, system/local backends, degraded set, global lock), the index lifecycle/health/source-selection types, the verified-index candidate pager, the `list-chaos` cargo feature, the `rustfs_s3_list_objects_index_*` metrics and the archived bench script. `list_objects_generic` now always walks. The write-path hooks `observe_list_objects_mutation(s)` are gone; their callers bump the scanner namespace generation directly through `observe_scanner_namespace_mutations`, which is the sub-step the old hook performed before taking the journal lock, so the generation observed by the scanner, the node RPC service and the data-movement lease checks is unchanged while PUT/DELETE no longer take a global tokio RwLock or read environment variables. `ListSourceMode::{metadata_authority, requires_live_metadata_verification, can_satisfy_strong_listing}` and `ListMetadataAuthority` were only reachable from the provider and are removed as well; the enum, its variants, the cursor encoding and `ListContinuationV2` are left untouched. * docs(listing): define index retirement upgrade prerequisites * fix(ci): retire the removed listing index test requirement |
||
|
Chris An
|
b7b26a6a49 |
refactor(listing): keep paginated listing on ECStore only (#8936)
* refactor(listing): keep paginated listing on ECStore only `Sets` and `SetDisks` carried their own copies of `inner_list_objects_v2`, `list_objects_generic`, `inner_list_object_versions`, `list_path` and `list_merged`, plus a third copy of the task that converts merged raw entries into the walk result stream. Only `ECStore` can answer a paginated listing, since a page merges every set of every pool; the copies drifted (the `Sets` copy still rejected `//` prefixes that #8605 made literal on `ECStore`) and were reachable only from tests and one decommission helper. Remove the `ListOperations` impls for `Sets` and `SetDisks` together with the duplicated pagination code. `SetDisks::walk` stays as an inherent method for set-scoped callers and now honors `versions_sort` like the store-wide walk; `SetDisks::list_path_result` stays for the recursive delete lister. The walk entry conversion moves into one `spawn_walk_entry_forwarder` shared by both walks. The decommission receipt lister walks its pool's sets instead of paginating through the removed `Sets` listing. Tests that exercised the removed layers now exercise `ECStore` for pagination or `SetDisks::walk` for per-pool version inspection. The architecture and logging guard rows that pinned the removed `SetDisks`/`Sets` listing entry points are dropped, and the error-other ratchet baseline shrinks by one. * fix(listing): exclude synthetic directories from receipt walks * fix(listing): stop abandoned walks and filter directory entries |
||
|
Chris An
|
7b29e9d65f |
test(scanner): verify checkpoint debt across process restarts (#8938)
test(scanner): preserve checkpoint debt across process restarts |
||
|
Hauser
|
2ccc8d8647 |
fix(admin): decouple target inventory from runtime convergence (#8921)
* fix(admin): list destinations without runtime reconciliation Read configured notification destinations directly from persisted config and merge existing runtime health without starting target reconciliation. Cover blocked reconciliation, absent and live-only runtimes, corrupt config, and authorization precedence through the real admin handler. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(admin): unify passive target inventory reads Read one durable configuration alongside existing runtime observations for event, audit, plugin, and extension inventories. Share the configuration across plugin domains and retain strict mutation readiness separately. Propagate module-state read failures and cover fresh configuration, secret redaction, concurrent reads, cancellation, and runtime independence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): route inventory errors through the S3 facade Keep the shared target reader behind the existing admin protocol boundary and reuse its error factory without changing error codes or messages. Tighten the error-macro ratchet to the actual reduced footprint. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
af4ce32528 |
perf(ecstore): reduce scanner metadata path allocation overhead (#8899)
* fix(scripts): collect scanner host telemetry on macOS Use platform-supported iostat arguments and expose optional collector failures without interrupting scanner-status collection. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): reuse validated metadata read path buffers Share the checked bucket/object tuple across metadata read consumers and borrow clean normalization inputs without skipping filesystem validation. Keep Windows reconstruction and the second bucket check intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scripts): finalize scanner telemetry for failed runs Initialize host telemetry artifacts before external captures so reused output directories cannot retain prior successes. Share normal and error-path finalization, reap collectors on request failure, and preserve the primary exit status even if status recording fails. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): distinguish unreadable recovery inventories Treat only the exact system-prefix read-quorum result as incomplete during bounded polling. Preserve strict final reads and the shutdown/drain assertions, record failed observations, and cover interrupted empty streaks and unrelated errors. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
7130f25af5 |
fix(ecstore): unblock delete replay and parallelize preparation (#8896)
* perf(ecstore): avoid head-of-line blocking in delete recovery Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): bound durable delete preparation and bootstrap recovery Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): verify clean exits in recovery acceptance Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): distinguish prepared and decided batch crashes Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): move final delete recovery options Move the options value into the batch-ordering regression's final case instead of cloning it. Keep the earlier clone needed for reuse. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): replay delete receipts after interrupted proof GC Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
c8f14dc337 |
test(scanner): attest running benchmark process identity (#8876)
* test(scanner): attest running benchmark process identity * fix(scanner): reject aliased process receipt data roots * fix(scanner): attest scheduled scan configuration |
||
|
Chris
|
334e8d9d3e |
test(scanner): verify retained versions with an independent oracle (#8870)
* test(scanner): add independent version content oracle * test(ecstore): drain guarded delete fixture rename tails * docs(scanner): record SDK seeding requirements |
||
|
Hauser
|
911ea265c5 |
fix(storage): apply and index explicit delete-marker removals (#8854)
* fix(ecstore): purge every explicit delete-marker version Keep explicit marker removals on the canonical version-delete path so same-key batches are not truncated and pending replication does not retain source markers. Preserve marker identity and timestamps in the response. Add real four-disk regressions for mixed version cleanup, duplicate marker IDs, pending replication, and null/UUID marker removal. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(filemeta): index explicit delete-marker batches Reuse the batch index and final compaction for ordinary marker removals. Keep small marker-only histories on the scalar path and preserve fallback for free versions and complex mutations. Add scalar-equivalence, boundary, corruption and deterministic scan-count coverage, a real-disk mixed-history regression, and reproducible marker benchmark cases. Preserve existing benchmark defaults while honoring CLI sampling overrides. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(filemeta): remove redundant marker fixture clone Move the original metadata into its final test fixture after the last borrow, avoiding the redundant clone reported by strict Clippy checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): validate explicit marker deletion and recovery Exercise UUID/null and duplicate explicit marker removals over real HTTP with retries disabled, including immediate four-node visibility and exact retained payloads. Cover all-drive PREPARED/APPLIED/DECISION process termination, rollback/commit recovery, and old-ID retries across a second restart without deleting a fresh marker. Use flat sampling and fixed per-case fixture batches in the metadata benchmark. Update profile selection digests for the added regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): synchronize crash fixture startup and flush readiness Give disk setup and the durable flush boundary separate readiness gates, keeping each phase bounded to ten seconds. Preserve the real SIGKILL and restart assertions while reporting early child exits and retaining output. Reap fixture children on assertion failure and cover early-exit diagnostics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
8c7b3996cb |
feat(scanner): extract OTLP histogram and counter windows (#8862)
* test(scanner): validate OTLP histogram measurement windows * feat(scanner): extract exact integer counter windows * test(ecstore): drain guarded delete fixture rename tails * test(ilm): retain transition timeout evidence |
||
|
Chris
|
7f1795f275 | test(scanner): add recoverable benchmark service control (#8868) | ||
|
Chris
|
b5f275ca3f |
fix(scanner): collect benchmark metrics from realtime route (#8859)
* fix(scanner): collect benchmark metrics from realtime route * docs(scanner): align benchmark help with realtime route * test(e2e): bound parallel seeding for scanner usage checks |
||
|
Chris
|
b39fcb2e85 |
test(heal): cover rc.5 upgrades and refresh ctutils (#8847)
* test(e2e): use the resumed decommission completion budget * test(heal): exercise checkpoints written by rc.5 * test(heal): route upgrade fixtures through the storage facade * fix(deps): update ctutils to address constant-time advisory * test(lifecycle): gate recovery I/O while advancing test time |
||
|
Chris
|
abe14e2f97 |
feat(scanner): configure startup recovery backoff cap (#8845)
* feat(scanner): configure startup recovery backoff cap * test(scanner): compare recovery retry count without narrowing * docs(scanner): document recovery backoff cap and expose it in Helm List scanner.recovery_backoff_cap in the runtime controls table, drop the fixed ten-minute wording from the follower recovery section, and map config.rustfs.scanner.recovery_backoff_cap_secs to RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart. |
||
|
Chris
|
623601d2f9 | fix(test): select the current G09 upgrade case (#8841) | ||
|
Hauser
|
44a73b0bef |
fix(s3): preserve typed upload errors across body readers (#8816)
* fix(s3): report payload SHA256 mismatches accurately Distinguish signed payload SHA256 failures from MD5 and additional checksum failures at the shared API error boundary. Preserve the payload error through archive extraction and use typed s3s errors. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(s3): classify body failures in one bounded pass Share typed body-error classification across direct and storage I/O. Retain marker precedence while replacing repeated source-chain scans, and map storage-wrapped short HashReader bodies to IncompleteBody. Cover nested priorities, real short reads, and one traversal budget. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(s3): route upload stream errors through the app facade Keep the real stream-error fixtures behind the existing test-only S3 boundary without increasing the direct s3s import footprint. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): preserve typed body failures across archive reads Route TAR entry failures through the shared archive error adapter and preserve every classified body error. Keep generic archive parsing as InvalidArgument without formatting discarded transport diagnostics. Use the existing timeout marker for PUT and Snowball body guards. Add regressions for real stream verification after sniffing and timeout responses, and tighten the reduced S3 macro footprint baseline. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
d2eda195e6 |
fix(decommission): defer source cleanup quorum failures (#8760)
* fix(decommission): defer source cleanup quorum failures A quorum failure while deleting the source of an already copied entry was fatal and marked the whole pool failed, although the copied source is still in place and an uncommitted delete transaction is settled by delete recovery. Carry the cleanup error typed and resolve a quorum failure as Deferred: the entry is replayed at the end of the round, and a deferral that persists takes the existing capacity-blocked pause path instead of a terminal failure. Other cleanup errors and quorum errors from other stages stay terminal. Add set-level regressions for parent and child keys (`a/b`, `a/b/c`): parent cleanup leaves every child file untouched, and parent cleanup reaches quorum while the child is cleaned up or rewritten concurrently. * test(ecstore): keep source cleanup errors typed in prefix cleanup tests Return SourceCleanupError from the test helper instead of formatting it into Error::other, and shrink the error(format!) ratchet baseline for core/pools.rs, which lost one call site in the cleanup failure change. * fix(decommission): retry cleanup quorum failures and count replays once Retry a source cleanup quorum failure in place, up to three attempts with the existing source-cleanup backoff, before deferring the entry. Carry the versions an entry already counted on its deferral, so the end-of-round replay does not add a fully counted entry to items_decommissioned and bytes_done again. A deferral that persists pauses the pool through the existing deferred drain path. When source cleanup caused it, persist a reason led by "source cleanup blocked" without the carrier error's free-drive wording, and report waitingReason "source_cleanup" instead of "capacity". Add a source_cleanup test fault stage and a routine test that drives a cleanup quorum fault through deferral and replay to completion. * fix(decommission): wait seconds between cleanup quorum retries Give the in-place source cleanup quorum retry a fixed floor of one and then two seconds, with the existing jitter on top, so it can ride out a drive reconnect or health-check blip instead of retrying within a few hundred milliseconds. The wait stays bounded and cancel-safe. Find a source cleanup pause reason through stage wrappers, as the capacity-blocked check does, and add a routine test where cleanup keeps losing quorum: the decommission pauses with a source cleanup reason, the bucket stays queued and the source stays in place. Document the additive waitingReason value source_cleanup and its compatibility with older nodes reading the same pool.bin. * fix(decommission): never retry a lost source cleanup fence The cleanup retry classifier used is_quorum_error(), which also matches NamespaceLockQuorumUnavailable. The delete commit fence reports a lost namespace lock or mutation fence with that variant, so a lost fence was retried in place and only stopped later at the pre-cleanup fence check. A lost fence means this worker no longer owns the key; it must stop without retrying. Retry and defer only drive quorum failures: ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and InsufficientWriteQuorum. --------- Co-authored-by: Hauser <housemecn@gmail.com> |