From 2bdec2fdcdcdd0bbc8ea6dced371ad836b2b4f6f Mon Sep 17 00:00:00 2001 From: Chris Date: Tue, 15 Sep 2026 20:56:13 +0800 Subject: [PATCH] chore(release): merge main into release (#7913) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593) feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592) Replace the assets-branch scheme with a proper GitHub Release on rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets are replaced in place on every build. This is the standard channel — visible on the repo's Releases page, stable download URLs, no git history growth (release assets live outside the repository). - New scripts/release/publish_nightly_assets.sh: resolves-or-creates the 'nightly' release via the REST API, deletes same-name assets, uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release body with the build provenance (ref@sha, run link, sizes, SHA256). Plain curl + python3, no gh CLI (the build fleet has none — #7586). - The workflow step shrinks to invoking the script; full flow exercised end-to-end against the real release with probe files (create / upload / overwrite / download round-trip / body update). * fix(ecstore): stop pruning at nonempty directories (#7616) * fix(ecstore): stop pruning at nonempty directories * test(ecstore): release pruning fixtures before temp cleanup * fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626) * fix(replication): total-order rule sort and honor V1 top-level Prefix Rule matching had two defects from the pre-GA replication audit (rustfs/backlog#2367 C-1 and C-2): - The actionable-rule sort compared same-destination rules by priority but answered Equal for any other pair, which is not a total order; the standard library sort panics on such comparators once a slice exceeds the insertion-sort threshold, so an object matching more than 20 enabled rules across two or more targets could panic the PUT or DELETE task. Rules now sort by priority descending with destination and id as tie-breakers, and filter_target_arns preserves that order instead of draining a HashSet. - A V1 rule written without a carries its prefix at the top level; that field was never read, so logs/ matched every object. ReplicationRuleExt::prefix now falls back to it, with a keeping precedence. The existing prefix fixtures were built this way and had been asserting nothing. * fix(admin): advertise data-usage and listen capabilities to rc The rc client gated `rc du` and `rc watch` on a pinned contract that matched server versions by the string prefix `1.0.0-rc.`; a server that reports `1.0.0` no longer matches, and the dynamic `advertised` list did not carry either name, so `rc du` against a GA server fails with an unsupported-capability error (rustfs/backlog#2367 E-2). Advertise `admin.data-usage` from the admin route inventory like the IAM entries, and `listen_notification` for the bucket `?events=` extension route the admin router dispatches. The client merges advertised entries ahead of its pinned contract, so no version sniffing is needed. * fix(site-replication): stop notifying the local site on remove and rotate The pending-remove and pending-rotation notification loops skipped the local site by endpoint only, while finalization identifies it by deployment id or endpoint. The reconcile tick resolves the local peer from the node's own listen address (and a handler from the request Host), so `remove --all` dialed the site's registered endpoint, waited out the request timeout against the lifecycle lock it was holding, and answered `Partial: failed to notify 1 peer(s)` for a removal that had succeeded (rustfs/backlog#2367 A-4, backlog#2195 item 3). Both loops now iterate the peers still awaiting notification through one helper that applies the finalization identity. * fix(site-replication): promote and settle IAM retries without a tick of slack Two retry-queue behaviours kept an IAM change from converging for ten to twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3, backlog#2305): - The lightweight 30-second pass filtered its reachability probe to bucket ops, so a backed-off IAM or bucket-metadata snapshot waited for the 600-second tick to notice the peer. It now probes every backed-off class and still replays only bounded bucket ops; promotion is a state flip the heavyweight tick acts on. - Backoffs are multiples of the tick interval, so a failure stamped δ seconds after a tick was 600 − δ old at the next tick and slipped a whole extra interval. The heavyweight drain now evaluates backoff halfway to its next tick. - An IAM entry first created by a non-deletion failure (the add bootstrap's snapshot send, the drain's own replay, an import-iam schedule) was never stamped `deletions_recorded`, so a later recorded deletion could not settle it and it escalated to the marker only `replicate repair` clears. Entries created by this binary now start recorded; a row persisted by an older binary keeps the escalation semantics. * fix(site-replication): reload peer node caches after bucket wiring writes Every S3 bucket-config write ends by asking the other nodes of the cluster to reload the bucket's metadata; the site-replication writers never did. On a multi-node site the node that ran the pairing (or applied a peer's bucket-meta item) rewrote the bucket targets and the derived replication rules on disk, while every other node kept serving its cached copy for up to the 15-minute refresh. A `resync start` routed to such a node reported every freshly wired bucket as `Config not found` and a bucket whose operator target the pairing had replaced as `recorded remote target no longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional SITE-105). Add one best-effort reload helper in the site-replication hooks and call it after the bucket setup, versioning, peer bucket-meta apply, removed-peer cleanup, make-with-versioning, and endpoint-refresh writes; the ensure helpers now report whether they wrote so unchanged passes stay silent. The resync manifest and start now read the persisted wiring instead of the node-local cache, matching the target read the start path already did. The new four-node e2e pairs two clusters and starts a resync through a non-coordinator node right after pairing; it also covers an IAM user created on a non-coordinator node converging to the peer site. * test(e2e): cover delete-marker replication from a multi-node source The functional suite reported delete markers created on a 3-node source never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report was a probe defect, but the shape had no coverage: the existing delete-marker e2e runs a single-node source. Pin it against a four-node source replicating to a four-node peer and to a single-node target, with the write and the delete issued through different nodes. * ci(e2e): refresh the distributed selection for the new replication cases Four distributed cases were added (two site-replication, two delete-marker replication). The linux digest is derived from the last CI listing of the lane (34 cases, matching the previous pin) plus the four new names; the darwin digest is the local listing, which selects the same 38 cases. * feat(console): support a configurable console URL prefix (#7634) Co-authored-by: houseme * chore(release): merge release into main for rc.6 (#7638) * test(heal): cover MRF manifest CAS legacy transition (#7443) Co-authored-by: zhi22915 * fix(ci): enable release branch checks and repair test imports (#7441) * test(scanner): structure heal release evidence lanes (#7442) Co-authored-by: zhi22915 * fix(scanner): require segment producer identities (#7444) Co-authored-by: zhi22915 * fix(scanner): expose recovery intent identity (#7445) Co-authored-by: zhi22915 * test(scanner): summarize failed heal perf reports (#7446) Co-authored-by: zhi22915 * fix(common): retain unmatched MRF repair proofs (#7447) Add a recorded verified-repair consumer that discharges only exact retained anchors for the requested bucket while leaving unmatched proofs in the event ring. This gives the future durable successor writer a fail-closed primitive before any tombstone or GC path is enabled. Co-authored-by: zhi22915 * fix(ci): route replication read plan through boundary (#7448) * test(scanner): cover flat-bucket quantum fairness (#7449) Add a production-entry scanner cohort regression that combines a wide flat bucket with a small bucket under a fixed object budget. The test keeps partial budgeted rounds unpublished, verifies small buckets are only marked after real execution, and confirms a later unbudgeted round can publish the complete aggregate. Co-authored-by: zhi22915 * fix(heal): reuse same admission request id (#7451) Keep a retried heal start with the same request id from bypassing admission deduplication when the transport replay cache is unavailable but the manager still owns the task. Co-authored-by: zhi22915 * test(scanner): require perf summary evidence fields (#7453) Fail closed when measured passing Scanner/Heal ABBA summaries omit W10/W11 foreground pressure, lock wait, or attempt-cost evidence. Co-authored-by: zhi22915 * test(scanner): bind release evidence fields (#7452) Require explicit scanner/heal release evidence field contracts for the scoped ACK and mixed-version rollback gates. Co-authored-by: zhi22915 * fix(scanner): tag dirty usage producer identities (#7454) Record production-facing segment invalidation producer identities when existing object-level dirty usage hooks observe PUT, CopyObject, DeleteObject/DeleteMarker, and CompleteMultipartUpload mutations. Keep the data non-authoritative and process-local so segment reuse activation still requires durable generation-window proof. Co-authored-by: zhi22915 * test(scanner): gate release evidence bundles (#7457) Co-authored-by: zhi22915 * test(scanner): require hard evidence ABBA manifest (#7459) Co-authored-by: zhi22915 * feat(heal): add MRF committed snapshot writer (#7458) Co-authored-by: zhi22915 * test(heal): cover EC8+4 restart shard rebuild (#7461) Retry heal-control RPCs once after transport auth rejects a stale replay-scope epoch, and add a distributed EC8+4 restart heal evidence case that rebuilds a replaced drive with exact shard/body assertions. Co-authored-by: zhi22915 * test(scanner): echo measured release evidence in ABBA fake adapter (#7460) Keep the synthetic ABBA test adapter aligned with the measured release evidence contract so result validation covers release_evidence drift. Co-authored-by: zhi22915 * test(scanner): require complete ABBA summary matrix (#7462) Co-authored-by: zhi22915 * heal: retain MRF replay journal after accepted/merged replays Keep startup replay journal until a durable successor snapshot exists after Accepted/Merged admission. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * heal: reclaim superseded MRF snapshots after readback Add a committed snapshot predecessor cleanup primitive that deletes only an older slot after the successor is read back as the current committed snapshot. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * heal: require exact owner for object receipts Require object heal receipts to match the expected bucket incarnation before they can be recorded as positive repair evidence. This prevents stale or cross-incarnation receipts from clearing the wrong heal responsibility. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * heal: latch object receipt owner before repair Capture the expected bucket incarnation before invoking object repair so a post-repair owner change cannot rewrite the responsibility that a storage receipt is allowed to prove. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * heal: replay committed MRF checkpoints durably (#7465) Prefer committed MRF checkpoints during startup replay, retain accepted replay responsibilities until exact verified repair proofs arrive, and reclaim committed manifests only after discharge. Co-authored-by: zhi22915 * test(scanner): bind hard evidence bundle provenance (#7467) Require Scanner/Heal release bundle fields to carry source, run, window, timestamp, command, and artifact format provenance before a measured gate can pass. Keep EC8+4 and performance gate fields tied to a single measurement window so unrelated artifacts cannot be stitched into a release approval. Co-authored-by: zhi22915 * heal: reject dry-run object repair receipts Do not record positive storage repair receipts for dry-run object heal tasks, even if a producer accidentally returns a matching receipt. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * heal: reject cancelled object repair receipts Do not record positive storage repair receipts once an object heal task has been cancelled, even if the receipt still matches the requested owner and object identity. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * test(heal): reject failed object repair receipts Cover the receipt consumer path where a storage repair result carries both an error and a matching positive receipt. The failure may be recorded, but the receipt must not create repaired, healthy, or absent proof. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * test(heal): cover bucket object repair receipts (#7468) Cover bucket and root heal sweeps recording authoritative object outcomes only when storage receipts match the latched bucket incarnation. Verify unavailable or stale receipt ownership keeps object repair execution intact while leaving canonical outcome proof as Unknown. Co-authored-by: zhi22915 * test(scanner): add EC8+4 heal restart evidence (#7469) Register and wire Scanner/Heal background target restart and crash evidence cases for the 3x4 EC8+4 topology. Validate the observed data/parity geometry in scanner-heal evidence receipts so multi-drive runs cannot satisfy the gate without proving EC8+4 metadata. Co-authored-by: zhi22915 * heal: verify replacement pool metadata repair (#7471) Co-authored-by: zhi22915 * test(heal): cover quorum and mixed repair receipts Add focused oracles for transient quorum results that carry a matching receipt and for mixed grace plus repaired receipt batches. Only the repaired object may produce positive proof. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * heal: verify admin recreate pool metadata (#7474) Co-authored-by: zhi22915 * heal: preserve merged result for duplicate submits Keep same-request-id replay receipts accepted for the receipt API, but preserve the legacy submit_heal_request duplicate admission result as Merged. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * test(heal): preserve EC84 restart semantics (#7478) Keep the EC8+4 background restart lane on the graceful-stop path and assert clean-restart marker absence only for restart scenarios. This prevents the hard evidence gate from silently exercising the crash path when it claims restart coverage. Co-authored-by: zhi22915 * test(heal): cover MRF snapshot torn successor recovery Co-Authored-By: heihutu Co-Authored-By: zhi22915 * test(heal): write EC84 distributed restart oracle (#7481) Bind the distributed EC8+4 restart evidence lane to a scanner/heal oracle artifact so release validation can consume the real nextest run instead of accepting only a passing test. Require the registry to assert 8+4 erasure geometry for the three-node, four-drive case. Co-authored-by: zhi22915 * fix(heal): harden MRF replay boundaries (#7483) Reject journal records with unknown version-presence flags even when their CRC is valid, so rollback/future payloads cannot be accepted as known records. Gate committed checkpoint cleanup by the writer owner captured from the replay source, preserving retained manifests from other owners inside the same sequence window. Co-authored-by: zhi22915 * test(ecstore): bind MRF manifest CAS dirsync recovery (#7482) Cover the committed MRF manifest path through LocalDisk conditional CAS when the metadata directory fsync fails. The fixture proves the previous manifest anchor survives rollback, an unanchored first successor is removed, and the legacy MRF journal remains readable even while global durability is relaxed. Co-authored-by: zhi22915 * test(scanner): derive evidence runner profile from registry (#7484) Co-authored-by: zhi22915 * test(scanner): require mixed-version evidence roles (#7485) Tighten the Scanner/Heal release bundle checker so mixed-version, scoped-ACK, and rollback fields cannot reuse a generic versions list without proving the expected evidence role. Require version lists to use source revision identities and include the tested source revision. Also require profile evidence fields to name the core profiling artifacts before release approval. The release gate remains blocked until measured field evidence is present. Co-authored-by: zhi22915 * fix(heal): publish committed MRF runtime checkpoints (#7490) Co-authored-by: zhi22915 * test(scanner): bind profile artifacts in release evidence (#7487) Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks. Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test. Co-authored-by: zhi22915 * test(scanner): harden measured ABBA evidence claims (#7491) Reject measured Scanner/Heal ABBA manifests whose mixed-version evidence uses the same baseline and candidate source revision or binary hash. Require crash fault modes and profile artifact names to match the exact supported sets, rejecting missing, duplicate, and unknown values. Update harness fixtures and regression coverage for same-build mixed-version claims and exact-set release evidence fields. Co-authored-by: zhi22915 * fix(e2e): group scanner heal evidence payload (#7494) Group the EC8+4 Scanner/Heal evidence writer inputs into a typed payload so the distributed e2e crate stays within the clippy argument limit without weakening the lint. The evidence writer still validates the same S3 bodies, physical shard census, process restart PIDs, and node listings. Co-authored-by: zhi22915 * test(scanner): require MRF replay bundle fields (#7486) Bind Scanner/Heal release bundle evidence for MRF durable replay to replay counts, retained responsibility anchors, and successor snapshot publication evidence. Co-authored-by: zhi22915 * test(scanner): profile EC84 evidence case runs (#7495) Give Scanner/Heal evidence cases explicit runtime profiles so EC8+4 background restart and crash cases use their own object count, object size, and partial-progress timeout defaults instead of inheriting the legacy 4x1 case assumptions. Expose the runtime profile in plan-only output and cover every registry case in the script self-test. Co-authored-by: zhi22915 * test(scanner): reject empty release evidence artifacts (#7496) Require Scanner/Heal release bundle artifact paths to resolve to non-empty files before hashing them. Cover empty hard-gate artifacts in the existing release bundle checker self-test and keep profile artifact size checking on the shared artifact boundary. Co-authored-by: zhi22915 * test(scanner): require G14 same-window field coverage (#7489) Tighten the Scanner/Heal release bundle checker so G14 same-window evidence must name the EC8+4, multi-set, and multi-pool fields covered in that measurement window. Keep the release gate blocked when same-window evidence omits one of the required G14 fields, without changing production runtime behavior. Co-authored-by: zhi22915 * test(scanner): require two-hour measured ABBA windows (#7493) Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement. Co-authored-by: zhi22915 * test(heal): cover MRF idle checkpoint cleanup (#7497) Add a runtime cleanup regression test that publishes both retained replay and runtime committed checkpoints, writes scoped and legacy journals, and verifies idle cleanup removes every recovery anchor from the registered local disks. Co-authored-by: zhi22915 * fix(e2e): ignore untracked files in build identity (#7498) Align e2e_test build provenance with Scanner/Heal evidence receipts and server binary provenance by treating only tracked source changes as dirty. This prevents unrelated untracked worktrees or evidence directories from causing compiled test identity mismatches before real evidence cases can run. Co-authored-by: zhi22915 * test(scanner): require MRF disk-full evidence fields (#7499) Require the Scanner/Heal release registry and bundle checker to carry explicit G08 MRF capacity, disk-full, and replica-loss evidence fields before release approval can pass. Co-authored-by: zhi22915 * test(scanner): bind release JSON artifact provenance (#7500) Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run. Co-authored-by: zhi22915 * test(heal): cover MRF rollback mirror filtering (#7501) Add a regression oracle that keeps scoped-only MRF responsibilities in the authoritative runtime snapshot while omitting them from the v1 legacy rollback mirror. Co-authored-by: zhi22915 * test(scanner): require segment activation evidence (#7504) * test(scanner): require segment activation evidence Co-Authored-By: heihutu Co-Authored-By: zhi22915 * test(scanner): cover activation proof inputs Co-Authored-By: heihutu Co-Authored-By: zhi22915 --------- Co-authored-by: zhi22915 * test(scanner): require MRF crash matrix cases (#7505) Co-authored-by: zhi22915 * test(scanner): require release bundle domain evidence (#7502) Reject scanner/heal release bundles that omit field-level domain evidence for scoped ACK, durable intent, mixed-version, scheduler pressure, profile cost, and two-hour pressure lanes. Co-authored-by: zhi22915 * test(scanner): require MRF retention GC evidence (#7506) Require P4 retained-responsibility release bundle evidence to list the retained replay anchor and idle cleanup cases, prove a two-hour retention window, and record both idle cleanup and verified-proof discharge observations. Co-authored-by: zhi22915 * test(scanner): require MRF cleanup GC soak evidence (#7507) Require the P4 release bundle to carry measured cleanup/GC soak evidence for retained MRF replay responsibilities. The bundle now needs a two-hour cleanup window, exact cleanup case coverage, observed verified idle GC, and zero pending responsibilities or stale journals after GC. Co-authored-by: zhi22915 * test(heal): cover MRF disk-full commit anchors (#7509) Co-authored-by: zhi22915 * test(e2e): report port allocator bind errors (#7510) Surface the scanned port window, attempt count, and last bind error when the e2e port allocator cannot reserve a localhost port. This keeps Scanner/Heal evidence failures actionable when the environment blocks binds before business assertions run. Co-authored-by: zhi22915 * fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511) * fix(sse): resolve bucket default encryption per request PUT and the POST-object/extract path resolved a bucket's default encryption with a hard-coded "no explicit SSE-C" flag, so the default was layered onto a request that already carried an SSE-C header triple and then tripped that request's own mutual-exclusion check. Every bucket with default encryption refused SSE-C single PUTs with 400 InvalidArgument, while CreateMultipartUpload on the same bucket succeeded because it resolves SSE elsewhere. Both call sites now derive the flag from the request headers, as COPY already did. The bucket default's KMS key id was also inherited independently of the effective algorithm, so an explicit AES256 request against an aws:kms default bucket produced a self-contradictory algorithm/key-id pair and was rejected. The key id is now inherited only when the effective algorithm is aws:kms, matching the storage-layer resolver. Refs backlog#2368 B1, B2. * fix(sse): refuse SSE-KMS without a running KMS service A write requesting aws:kms on a node with no KMS service fell back to the node-local SSE-S3 provider: the data key was wrapped with RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded aws:kms and the requested KMS key id. The stored object claimed a KMS protection it never had, under a key that was never consulted, and no signal distinguished it from a genuine SSE-KMS object. The managed-encryption path now asks the resolved DEK provider whether it wraps with a node-local master key and refuses SSE-KMS in that case: InvalidRequest when KMS was never configured, ServiceUnavailable when a configured service is not running. The check sits after the per-key authorization gate so an unauthorized caller still receives AccessDenied whatever the KMS runtime state is, and asks the provider rather than a parallel availability signal because the provider is what actually wraps the key. A missing master key no longer answers an SSE-KMS request with an SSE-S3-worded configuration error. The SSE-S3 local fallback is unchanged. Refs backlog#2368 B4. * fix(ecstore): restore and archive tiers in stored coordinates Multipart restore addressed the remote tier in plaintext coordinates while the copy-back reads the stored representation. Each part received a misaligned slice of the remote object whose length still satisfied the range, the hash reader and the completion size check, so the restore reported success and silently replaced the object's bytes. Encrypted and compressed multipart objects were both affected. Restore now accumulates stored part sizes, passes the stored length to the hash reader alongside the plaintext length, and validates against the stored size. The copy-back digests stored bytes, so its computed MD5 is not the object's public ETag. Restore now preserves the object ETag on both the single-part and multipart paths, and gives each restored part its own recorded part ETag rather than the object-level value. Transition also handed the tier the object's SSE headers and its RustFS-wrapped data key as request headers. Any S3 target rejected an SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a second time under a key id it does not own, and the wrapped DEK left the cluster. The archive request now strips every SSE header and encryption marker with the predicate the replication path already uses; the local xl.meta keeps all of it, so read-through and restore are unaffected. Objects restored by an affected release are not detected or repaired retroactively and must be re-restored from the tier. Refs backlog#2368 B3, B5; backlog#2369 P7.1. * fix(rio): lock the v1 nonce layout within a segment Decrypting a v1 segment tried three historical nonce layouts per frame, independently for every frame. The last of them exists for streams written before 1.0.0-alpha.91, which reused a segment's part nonce for every block in it; because block zero's derived nonce equals that base nonce, a frame encrypted at index zero authenticated at any position. An attacker able to rewrite the underlying shards could replay it and have the forged plaintext returned with 200 and an unchanged length. Shard integrity uses a keyed-hash-free checksum, which such an attacker can recompute, so it is not a barrier. A segment now locks onto whichever layout decoded its first non-zero-index frame and rejects any later frame needing a different one. That leaves one residual shape: a stream built purely from repeats of frame zero has no later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK (default true, so pre-alpha.91 objects keep decrypting) drops the third layout entirely when set to false, which closes it. Turning it off refuses pre-alpha.91 objects, so migrate them first by rewriting in place. Refs backlog#2369 P2. * fix(kms): reload a service that failed to start POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted configuration matched the in-memory one byte for byte. A node whose KMS failed to start keeps that configuration and sits in Error, so the documented recovery call returned "reloaded successfully" while leaving the node down. Peers reached the same path through the reload broadcast, so a cluster that lost Vault during a rolling restart had no working recovery route other than the node-local start endpoint. Reload now short-circuits only for a service that is actually running, and otherwise reconfigures, which starts a service that is not running. The AWS backend also advertised key-version enumeration through kms/status, which its own documentation says it cannot do; the capability and its golden snapshot now say false. Refs backlog#2369 P1, P7.3. * docs: record the SSE and KMS changes for 1.0.0 The Unreleased changelog section carried no entry for any encryption work merged since 1.0.0-rc.5, including three items with operational impact: the config-secret variable whose absence persists secrets in cleartext with only a warning, the v2 frame write switch and its rolling-upgrade constraint, and per-key authorization making a public bucket incompatible with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS refusal as a breaking change with both routes out. Also corrects four places where documentation contradicted the code: the cleanup register still called encrypted range seek opt-in after its default flipped, the Helm README claimed vault_mount_path only applies to Transit while the template also feeds the KV2 mount, the disaster-recovery drill listed bundle contents for backends whose export is refused with 501, and the Chinese README capability table predated most of the feature set. Documents the SSE-S3 local master key as a first-class operational mode with its rotation dead end, and what the v1 frame layout does and does not authenticate. Refs backlog#2369 P5. * fix(kms): classify data-path KMS failures by what the caller can do Only "key not found" and a backend outage were classified; every other KMS failure that reached the S3 data path fell through to 500 InternalError with a generic message. A disabled or pending-deletion key, a denied KMS grant, an encryption-context mismatch, an unsupported algorithm, a credential or timeout failure, and a capability the configured backend does not have all looked identical to a server fault. SDKs therefore applied exponential backoff to configuration errors no retry can fix, and monitoring counted every one of them against the server's own error rate. Unusable-key and request-side failures now answer 400, a denied grant 403, transient backend failures 503, and a missing backend capability 501. Damaged, unreadable, or unknown-format key material keeps its 500: it is a server-side integrity fault, and existing tests pin it. The classifier is deliberately separate from the admin lifecycle mapping, which answers 404 for a missing key because there a key id is the resource being addressed; on the data path it arrives inside a request header or a bucket default. Messages either name what the caller asked for or stay generic, with deployment-side detail left on the error source the way the storage-IO mapping already does. Refs backlog#2368 B6. * fix(kms): track and renew static Vault tokens Token authentication hard-coded "this token carries no lease", so the renewal task never started, the remaining-TTL gauge was never published, and nothing looked wrong. `vault token create` grants a 768-hour TTL by default, so a cluster that had been healthy for a month turned every KMS call into a 403 and could not recover without a restart or a reconfigure. Production configuration validation only rejects the literal dev-token, so an ordinary expiring token reaches a whole cluster. The source now reads `auth/token/lookup-self` at login and adopts what Vault reports. A token with no expiry behaves exactly as before. An expiring renewable one is picked up by the existing renewal loop and renewed at half TTL like every other auth method. An expiring non-renewable one warns with its remaining lifetime and publishes the gauge, so the fail-closed window is visible before it arrives. The probe never fails the login: a policy that omits lookup-self, or a Vault that is briefly unreachable, warns and falls back to exactly the previous behaviour rather than taking down a deployment that works today. The scripted Vault test double answers the lookup out of band so existing scripts keep describing only the protocol under test. Refs backlog#2369 P3. * feat(sse): report SSE-C requests that arrive without TLS An SSE-C request carries the customer's AES key in a request header, so AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS accepted them on any transport: a plaintext hop hands the key to anyone on the path, and since the object cannot be read without that same key, the exposure lasts as long as the object does. Refusing outright is the correct end state but not a safe default to adopt inside a release window, because the project's own s3-tests and e2e lanes and most staging deployments speak plain HTTP. This release reports instead: each such request increments rustfs_ssec_plaintext_requests_total and logs one warning per process, so an operator can confirm nothing would break before the default flips. RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now. The verdict is per connection rather than per deployment: the layer is built with whether this listener terminated TLS, and additionally accepts an https protocol forwarded by a proxy the trusted-proxy configuration already vetted. It sits beside the rate limiter, after the layer that makes a forwarded protocol trustworthy and after the request context, so a rejection can echo the request id. Refs backlog#2369 P7.2. * fix(kms): say what a node-local backend means for a cluster The Local backend keeps key material on each node's own disk and generates its Argon2id salt per node, so two nodes derive different keys from the same master_key and an object encrypted on one node cannot be decrypted on another. Behind a load balancer that surfaces as intermittent 500s on reads that succeeded moments earlier, with nothing tying the symptom to the cause: the only signal was a generic "development, testing and demos only" positioning warning that says nothing about what actually breaks. Configuring or reconfiguring Local while the deployment is distributed now logs a dedicated event and appends the consequence to the configure response, so the operator who made the change sees it. The product decision to warn rather than refuse is unchanged. Refs backlog#2369 P7.4. * docs: record the remaining SSE and KMS changes for 1.0.0 Adds changelog entries for the KMS data-path status classification, the Vault static-token lease probe, the SSE-C plaintext-transport report and its switch, and the node-local backend warning. Documents two things the backend security guide never stated: that SSE-C belongs on a secure transport, with the counter and switch to plan the change around, and that the Local backend cannot be shared by a multi-node deployment because each node derives different keys from the same master key. Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4. * fix(kms): report an unreadable key store as an outage on the S3 path A backend now distinguishes a key store it could not read from a key that is genuinely absent, but the S3 boundary collapsed the first one back onto 500 InternalError through the fallthrough for integrity faults. The distinction was therefore invisible to the client: a temporary key-directory outage looked exactly like a permanently damaged key record, and neither the status nor the metric said the request was worth retrying. An unreadable key store joins the retryable class and answers 503, next to a backend error and a credential failure. Damaged, unreadable or unknown-format key material keeps its 500. Refs backlog#2368 B6; builds on rustfs/rustfs#7470. * test(storage): cover MRF cleanup delete ENOSPC anchor (#7513) Co-authored-by: zhi22915 * test(scanner): require heal retry stats evidence (#7514) Co-authored-by: zhi22915 * test(scanner): add release bundle dry-run fixture (#7515) Co-authored-by: zhi22915 * fix(scanner): confirm recovery intent accept readback (#7516) Co-authored-by: zhi22915 * test(scanner): require MRF cleanup delete ENOSPC evidence (#7518) Co-authored-by: zhi22915 * test(scanner): require bounded retry window evidence (#7517) Co-authored-by: zhi22915 * test(e2e): emit scanner heal G09 upgrade evidence (#7519) Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided. Co-authored-by: zhi22915 * test(heal): persist MRF rollback mirror boundary (#7520) Co-authored-by: zhi22915 * test(scanner): add G09 upgrade evidence runner (#7522) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * test(scanner): add G09 upgrade evidence runner Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes. The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist. Co-Authored-By: heihutu Co-Authored-By: zhi22915 --------- Co-authored-by: 唐小鸭 Co-authored-by: Zhengchao An Co-authored-by: zhi22915 * feat(scanner): implement V2 evidence preflights (#7523) * feat(scanner): wire dirty usage producer identities Co-Authored-By: heihutu Co-Authored-By: zhi22915 * test(scanner): add segment activation preflight proof Keep scanner segment reuse behind a structured activation preflight so release evidence can prove the production gate remains disabled until every producer, generation, overflow, cold-oracle, and distributed invalidation check is satisfied. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * feat(scanner): expose distributed invalidation evidence Record an explicit distributed segment invalidation evidence summary when remote dirty usage snapshots are bound to the current activity window and the authenticated scoped ACK capability probe succeeds. Reject peer dirty usage snapshots that contradict the peer activity pending bit so scoped ACKs fail closed instead of clearing an unadvertised remote mutation. Co-Authored-By: heihutu Co-Authored-By: zhi22915 --------- Co-authored-by: zhi22915 * test(scanner): assemble release evidence bundles (#7526) Co-authored-by: zhi22915 * test(scanner): add G09 upgrade evidence runner (#7524) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * test(scanner): add G09 upgrade evidence runner Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate. Co-Authored-By: heihutu Co-Authored-By: zhi22915 --------- Co-authored-by: 唐小鸭 Co-authored-by: Zhengchao An Co-authored-by: zhi22915 * fix(scanner): replay recovery intents while disabled (#7521) Co-authored-by: zhi22915 * test(scanner): preflight G09 evidence disk space (#7527) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it. * fix(ci): bind performance runs to selected inputs (#7512) * fix(targets): reject trailing batch items (#7508) * test(scanner): preflight G09 evidence disk space Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass. Co-Authored-By: heihutu Co-Authored-By: zhi22915 --------- Co-authored-by: 唐小鸭 Co-authored-by: Zhengchao An Co-authored-by: cui fliter Co-authored-by: zhi22915 * fix(scanner): respect cargo target dir in G09 runner (#7528) * fix(replication): close the GA blocker set from backlog#2366 (#7503) * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the… * feat: configure the console base path at build time (#7636) Embed RUSTFS_CONSOLE_BASE_PATH into console routes, browser redirects, and startup URLs so OEM builds can use their own namespace. Preserve the default for an unset or empty build variable and keep route tests prefix-aware. * chore(release): prepare 1.0.0-rc.6 (#7641) * chore(release): prepare 1.0.0-rc.6 * ci: run offline enrollment boundary checks independently * fix(heal): preserve retryable batch failures during recovery (#7642) * fix(heal): preserve retryable batch failures during recovery * test(heal): pin prebuilt hooks binaries in ci * fix(s3): reject oversize single PUT early and map body errors to 4xx (#7635) * fix(s3): reject oversize single PUT early and map body errors to 4xx A single PutObject above the 5 GiB single-request ceiling was only rejected after the client had streamed 5 GiB into s3s's read-time body budget, and the resulting BodySizeLimitExceeded surfaced from the erasure writer as 500 InternalError. A body whose connection hit EOF before Content-Length bytes arrived (hyper's IncompleteBody) was also a 500. SDKs retry 500s, so one oversize upload was resent from offset 0 five times. - PutObject and UploadPart reject a declared length above MAX_SINGLE_PUT_OBJECT_SIZE with 400 EntityTooLarge before reading the body; the constant moves to rustfs_config so the s3s limit and the admission check share one value. - ApiError maps BodySizeLimitExceeded to EntityTooLarge and a hyper body EOF to IncompleteBody across both io::Error conversions. Fixes #7596. * test(s3): cover UploadPart admission, aws-chunked length, real s3s limit - Poll-counting test body proves PutObject and UploadPart reject a declared size above the ceiling with zero body polls; exact-cap and zero-length parts pass admission. - A STREAMING-* aws-chunked PUT whose framed Content-Length exceeds the cap is admitted when the decoded length is within it and rejected when the decoded length is over it. - The display-based BodySizeLimitExceeded matcher is checked against the real error produced by the pinned s3s Body budget. * fix(ecstore): make directory mtime fixture portable (#7623) * fix(ecstore): make directory mtime fixture portable * style(ecstore): format mtime fixture assertion --------- Co-authored-by: houseme Co-authored-by: Zhengchao An * fix(storage): prevent readiness after native migration failures (#7652) * fix(storage): prevent readiness after native migration failures * fix(storage): skip unsupported IAM records before reading * fix(storage): use stable typed migration metadata errors * fix(storage): include migration record in startup errors * test(storage): cover native migration startup failures * test(storage): use array chunks in migration fixture --------- Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com> Co-authored-by: cxymds * fix(admin): expose OIDC account display fields (#7654) Expose verified OIDC username and email claims as display-only metadata on self-account responses while preserving the virtual parent as the authorization identity.\n\nKeep rustfs-madmin public response structs unchanged by adding the optional wire fields through private handler response wrappers. * fix(replication): correct peer joins and remote-state reporting (#7650) * fix(replication): propagate verified peer deployment identities * fix(replication): report actual remote peer state * fix(replication): defer initial sync until all peers join * test(replication): shut down TLS fixtures cleanly --------- Co-authored-by: houseme * feat(ci): add fault-tolerance degradation suite to the functional chain (#7663) Adds a fault-tolerance suite that verifies read/write behavior under drive and node loss against the erasure-coding contract and snapshots health-endpoint responses at every degradation tier. Scenarios derived from product source (default_parity_count, erasure set sizing): - A: single-node 4 drives (EC:2, read quorum 2): hide 1/2/3 drives - B: multi-node 4x1 (one set of 4, EC:2): stop 1/2/3 nodes - C: multi-node 4x4 (one set of 16, EC:4, read quorum 12): 1 node down lands exactly on the read-quorum boundary; 2 nodes down breaks it - C2: multi-node 4x4 with RUSTFS_STORAGE_CLASS_STANDARD=EC:8 (read quorum 8, write quorum 9, lock majority 9): 2 nodes down puts reads inside the reported divergence window (read quorum met while the lock majority is broken) By default a reads-refused-despite-met-read-quorum observation is reported as known-divergence without failing the suite; the strict input escalates it. Chain order becomes: upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security -> replication -> fault-tolerance -> performance. * fix(ci): repair functional defaults and chain regression checks (#7664) * fix(ci): default functional suites to nightly packages * test(ci): follow the fault-tolerance chain handoff * "feat(ci): add fault-tolerance degradation suite to the functional chain" (#7667) Revert "feat(ci): add fault-tolerance degradation suite to the functional chain (#7663)" This reverts commit 7c47c48e8523515bdb4d0dedee96d503d1758a69. * fix(ci): align security workflow tests with chain (#7679) * test(kms): cover non-default Vault Transit paths (#7657) * test(kms): cover custom Vault Transit metadata locations * test(kms): include a non-default Transit mount in live coverage * test(ecstore): tolerate cleanup quorum transition * test(ci): restore ten-suite chain expectations * test(ecstore): tolerate cleanup quorum transition in both causal polls The unversioned duplicate-delete poll in batch_transitioned_delete_uses_free_version_per_item reads exact metadata while the same asynchronous free-version cleanup removes per-disk copies, so it can hit the same transient InsufficientReadQuorum as the versioned poll. Share one helper between both polls; every other error still fails immediately. --------- Co-authored-by: overtrue * fix(s3): bound stalled UploadPart request bodies (#7659) * fix(s3): bound stalled UploadPart request bodies * fix(ci): preserve the S3S footprint ratchet * fix: enforce S3 permissions for recursive force deletion (#7661) * fix: enforce S3 authorization for recursive deletion * fix: satisfy the s3s footprint guard * fix: restore list versions policy compatibility (#7686) * fix(tables): reject reserved warehouse locations (#7671) Co-authored-by: cxymds * fix(e2e): require a verified server binary for every e2e run (#7687) * fix(ci): share quick checks and lint workflows * fix(ci): install actionlint from its verified release * fix(ci): reject dependencies on required quick checks * feat(test): verify the E2E server build and source identity * test(e2e): register verified Darwin test membership * test(e2e): record verified Linux receipt test membership * test(e2e): record compiled Darwin receipt test membership * test(e2e): record compiled Linux receipt test membership * test(e2e): record Darwin e2e-full membership after merging main * fix(test): route scanner/heal evidence E2E runs through the verified server binary The evidence runners built rustfs with plain cargo and then ran e2e_test directly, which now fails without a run receipt. They build through scripts/e2e_binary.py and run the e2e_test invocations under e2e_binary.py run; the obsolete rustfs.features stamp is removed. * docs(e2e): run server-backed e2e commands through the verified binary wrapper * test(e2e): record Linux e2e-full membership from the branch CI listing * fix(ci): bind nightly lanes to one resolved source (#7688) * feat(ci): measure queue and execution time by run attempt (#7689) * feat(ci): measure queue and execution time by run attempt * fix(ci): bound timing samples and recognize the old workspace lane * fix(ci): sample completed runs for stable timing comparisons * fix(ci): verify complete functional chain evidence (#7690) * fix(ci): bind nightly lanes to one resolved source * fix(ci): verify complete functional chain evidence * fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697) * fix(sse): classify bare SSE-KMS writes when no KMS is available A `aws:kms` request without a key id, on a bucket without a default key, returned `500 InternalError` whenever no KMS service was running: the "no KMS key available" branch exited with an untyped storage error before the availability classification that the keyed form already received. Route that branch through the same split: `503 ServiceUnavailable` while a configured KMS is stopped, `400 InvalidRequest` when KMS was never configured, and `400 InvalidRequest` naming the missing key id when a running KMS has no default key. `CreateMultipartUpload` shares the path. Adds a unit test for the bare form and an e2e module that stops KMS through the admin API, runs a master-key-only node, and runs a Local KMS without a default key; refreshes the e2e-full selection digests. (cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8) * fix(sse): keep KMS error classes on the encrypted read path GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key no longer exists answered `500 InternalError` ("KMS key not found") while PutObject under the same key already answered `400 KMS.NotFoundException`. The read path carries its classification through ecstore's `EncryptionResolutionErrorKind`, which had no kind for a missing key, a denied KMS grant or a missing backend capability, so all three folded onto `DecryptionFailed` and the S3 layer reported an internal fault. Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on both sides of the boundary, and give an envelope the configured backend cannot unwrap a diagnosable message while keeping its `500`. Unit tests cover the kind round trip and the reader wrapping; a new e2e test deletes a key immediately and checks GET/Copy return 400 with `KMS.NotFoundException` while HEAD stays 200. The e2e-full selection digests are refreshed from the current listing (the previous digests predated the delete-authorization tests) and the e2e `create_default_key` helper is updated to the accepted `EncryptDecrypt` spelling. (cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2) * fix(kms): classify key-management errors on the admin routes `POST /kms/keys`, the legacy `create-key` alias and `generate-data-key` reported every backend refusal as `500`: a blank key name (which each backend failed on differently, the Local backend by writing a key file with an empty stem), a name already taken, an unknown key, a disabled key and a capability the backend lacks. `delete` and the lifecycle routes already classified the same errors. Refuse a blank or whitespace name in `KmsManager::create_key` before any backend sees it, and share one `KmsError` to status mapping across create, delete and generate-data-key (400 for validation and key state, 404 for an unknown key, 409 for a taken name, 501 for a missing capability, 500 only for damaged material). The XML-error routes carry the same status explicitly since s3s derives none for a custom code. The read-only Static backend now reports create, delete and cancel-deletion as `UnsupportedCapability`, matching its rotate and enable/disable answers, so the admin API returns 501 for all of them. (cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b) * fix(sse): stop SSE-S3 responses from naming the wrapping KMS key `x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms` objects only, but PutObject, CopyObject, CreateMultipartUpload and GetObject returned it for `AES256` objects too, carrying the KMS key that wraps the SSE-S3 data key (the service default, or the literal `default` on a node without KMS). The write paths copied `kms_key_id` from the encryption material unconditionally, and the single-decrypt GET classification did the same after resolving the key for authorization. Add `EncryptionMaterial::response_kms_key_id`, which yields the id only for SSE-KMS, use it at the four write-response sites, and gate the GET classification the same way. CompleteMultipartUpload and HeadObject already omitted the header. Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy and CreateMultipartUpload for AES256 with an aws:kms control. The e2e-full selection digests are refreshed from the current listing. (cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb) * fix(s3): validate PutBucketEncryption rules before storing them A default-encryption rule naming an unknown `SSEAlgorithm` (for example `AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as written: the only algorithm check on the route decided whether to fill in the default KMS key. `GetBucketEncryption` then advertised that configuration while the write path encrypted header-less writes under its `AES256` fallback, so the bucket's declared and actual schemes disagreed. Two comments claimed the route already refused unknown algorithms. Validate the configuration before any of it is applied: `MalformedXML` for a malformed rule set or unknown algorithm, `InvalidArgument` for a key id on a non-KMS rule, and nothing stored on refusal. Correct the two comments to describe when the AES256 fallback is still reachable. Unit tests cover every refusal and the accepted shapes; an e2e test checks the refusals leave the previous configuration in place. The e2e-full selection digests are refreshed from the current listing. (cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb) * test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes * test: align two unit tests with the new KMS and bucket-encryption contracts `scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still expects the state error (`InvalidOperation`) for cancelling a key that is not pending deletion; only the Static backend's mutations moved to `UnsupportedCapability`. The uninitialized-store PutBucketEncryption test now sends a well-formed AES256 rule so it reaches the store lookup instead of the new configuration validation. * Collect bounded Connect environment inventory (#7710) feat(connect): collect environment inventory * Add Connect service license commands (#7711) feat: add Connect service license client * Add consent-bound diagnostic scheduling (#7713) feat(connect): schedule consent-bound diagnostics * feat(connect): advertise environment diagnostics (#7714) * feat(connect): add consent-bound profile exports (#7716) feat: add consent-bound profile exports * feat(connect): add bounded local log capture (#7718) * feat(connect): add bounded network performance producer (#7719) * Add bounded local telemetry export commands (#7720) feat(connect): add bounded telemetry producers * Add bounded drive performance command (#7721) feat(connect): add bounded drive performance producer * Add bounded top diagnostic captures (#7722) feat(connect): add bounded top diagnostic producers * Add client-to-deployment performance diagnostics (#7726) feat: add client performance diagnostics * Add a local environment inventory command (#7728) feat(connect): add local environment inventory command * feat(connect): add bounded object performance diagnostics (#7734) feat(connect): add object performance producer * Add authenticated inter-node network performance probes (#7739) * feat(connect): add authenticated inter-node network probes * fix(connect): respect storage facade boundary * fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over (#7709) * fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over Site replication wired each bucket by looking for an existing replication target "to the same peer" and rewriting the first match in place as its own same-name target. An operator's bucket-level target that happened to point at that site (different target bucket, operator credentials) was the first match whenever it pre-dated the join, and the reconciler repeats the pass every 600s, so the takeover also depended on target order afterwards. The operator's rule then named an ARN no target backed and their bucket replication stopped silently, while the inherited bucket-level reset id made every site resync report the bucket as owned by another resync (rustfs/backlog#2479, rustfs/backlog#2489). Follow MinIO's `getRemoteARN` / `getRemoteARNForPeer` shape instead: - Wiring updates a target in place only under the same ARN, or when it is recognisably the site's own under an older ARN shape (same peer, same-name target bucket, site replication service account). Anything else gets the site target added next to it. - The site resync manifest takes the target the derived `site-repl-` rule names (same-name shape as fallback), so an operator target to the peer neither aborts the bucket as "multiple remote targets matched peer" nor gets resynced into. - Peer removal prunes only targets a pruned derived rule names or the same-name target bucket; operator targets stamped with the peer's deployment id survive together with their rules. Unit tests cover the three predicates. e2e `test_site_replication_keeps_operator_bucket_target_to_peer` runs a bucket-level replication plus `replication-reset` to the future peer, joins the sites, and requires the operator target untouched, both paths delivering, the site resync completing against the site target, and the operator target and rule surviving `replicate remove --all`; without the fix it fails at the join with the operator target gone. The repl-nightly selection digest is refreshed for the new case. * test(site-replication): drop a redundant clone flagged by clippy The reconcile unit test cloned the remote peer into the state map although the binding is not used afterwards; workspace clippy (-D warnings) rejects that as redundant_clone. * chore(deps): update flake.lock (#7733) * chore(deps): update flake.lock Flake lock file updates: • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/17de0b9' (2026-09-04) → 'github:NixOS/nixpkgs/aff8a0b' (2026-09-10) • Updated input 'rust-overlay': 'github:oxalica/rust-overlay/c361047' (2026-09-05) → 'github:oxalica/rust-overlay/228ecef' (2026-09-12) * fix: satisfy Rust 1.98 clippy lints Co-Authored-By: heihutu Co-Authored-By: zhi22915 --------- Co-authored-by: zhi22915 * chore(deps): update flake.lock (#7733) * chore(deps): update flake.lock Flake lock file updates: • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/17de0b9' (2026-09-04) → 'github:NixOS/nixpkgs/aff8a0b' (2026-09-10) • Updated input 'rust-overlay': 'github:oxalica/rust-overlay/c361047' (2026-09-05) → 'github:oxalica/rust-overlay/228ecef' (2026-09-12) * fix: satisfy Rust 1.98 clippy lints Co-Authored-By: heihutu Co-Authored-By: zhi22915 --------- Co-authored-by: zhi22915 * feat(connect): support explicit enterprise proxies (#7745) feat(connect): add explicit proxy transport * feat(connect): emit drive unavailable log events (#7747) * feat(connect): add local inspect export producer (#7750) * feat(connect): add local inspect export producer * fix(connect): use the storage inspection facade * test(connect): cover nested allocator profile stats (#7717) * test(connect): cover nested allocator profile stats * test(connect): use async profile test lock * test(security): allow synthetic redaction fixture --------- Co-authored-by: Hauser * docs(security): add presigned copy advisory lesson (#7754) * fix(ecstore): refuse RMW and reads of unreadable bucket configs (#7759) An XML bucket sub-config whose stored bytes cannot be parsed was still folded into "absent" on two paths: update_config_with handed it to mutate, which rebuilt it from nothing and overwrote the only copy of the bytes, and the tagging/lifecycle/CORS/website/logging/accelerate/request-payment getters reported ConfigNotFound. Add an explicit ConfigState (Absent / Valid / Unreadable) over the existing retained parse failure, refuse the read-modify-write of an unreadable target config before mutate runs (per config, other configs stay writable), and make those getters fail closed. GetBucketLifecycle now only maps ConfigNotFound to NoSuchLifecycleConfiguration. The Swift tagging rewrite recognizes the refusal by type instead of a string sentinel. Refs rustfs/backlog#1734 * refactor(filemeta): own persisted metadata key authority (#7760) Add rustfs_filemeta::metadata_keys as the single source of the nine xl.meta meta_user keys, byte-identical to current literals, and migrate filemeta consumers off s3s::header and rustfs_utils header constants. Pin the literals, cross-check historical sources, and prove old xl.meta bytes still decode via a captured fixture and a per-character mutation sweep. Refs rustfs/backlog#1735 * Sync network performance protocol fixtures (#7761) test(connect): sync diagnostic protocol fixtures * Sync site replication result fixtures (#7762) test(protocol): sync site replication result fixtures * Sync telemetry producer fixtures (#7763) test(protocol): sync telemetry producer vectors * refactor(rio): own trailer source and drop s3s dependency (#7764) Move the aws-chunked trailing checksum handle behind a RustFS-owned rustfs_rio::TrailerSource (Pending/Missing/Present lookup with a documented EOF timing contract) and adapt s3s::TrailingHeaders once in the application crate. rio no longer depends on s3s; behaviour is unchanged. Refs rustfs/backlog#1735 * Deliver diagnostic scheduler receipts to Connect (#7767) feat(connect): deliver diagnostic scheduler receipts * fix(ecstore): refuse writes on unreadable bucket configs with 503 (#7768) Close the remaining paths where a stored bucket sub-configuration whose bytes cannot be parsed still read as absent (rustfs/backlog#1734, slice 2): - Object writes resolve versioning through BucketVersioningSys::get_for_write; RUSTFS_BUCKET_CONFIG_PARSE_MODE=strict refuses them, the default permissive mode keeps the historical unversioned write and records it. - Versioning, Object Lock, encryption, public access block and notification getters and the delete-time versioning check return the typed UnreadableBucketConfig refusal, which ApiError maps to 503 naming the bucket, config and stored length. StorageError::clone keeps it typed. - Object Lock checks stay on when the lock config is unreadable. - Notification setup isolates an unreadable config to its bucket instead of clearing its rules; GetBucketNotificationConfiguration reports it. - Add rustfs_bucket_metadata_parse_failed_total and rustfs_bucket_metadata_unparsable_current, an invalid-mode startup check, and the s3gate-parse-strict compat register entry. * feat(connect): add site replication performance producer (#7769) feat(connect): add site replication performance core * refactor(ecstore): read persisted object-metadata keys from the filemeta authority (A3b) (#7770) Migrate ecstore and rustfs-lifecycle consumers of the persisted xl.meta meta_user keys (object lock, restore, replication status, storage class, SSE read) from s3s::header / rustfs_utils header constants to rustfs_filemeta::metadata_keys. HTTP header production is unchanged and every lookup keeps its previous exact / case-insensitive mode. - SSE key: persisted only as lowercase; the mixed-case spelling is outbound replication user metadata. Keep the case-insensitive read and pin it against the pre-A3a fixture. - warm tier: strip promoted keys case-insensitively so the persisted X-Amz-Replication-Status is no longer forwarded to the tier. - fix stale HashReader::add_checksum_from_s3s call in ecstore tests. Refs rustfs/backlog#1735 * feat(connect): add device license renewal client (#7771) * feat(connect): add approved artifact relay (#7772) feat(connect): add approved artifact relay core * fix(connect): export relay CLI types (#7773) * feat(connect): add device report upload client (#7774) * feat(connect): add typed telemetry trace source (#7775) * fix(connect): avoid shadowing replication URL helper (#7777) * fix(connect): restore site replication test nonce (#7778) * fix(connect): restore rustfs build after site replication perf producer (#7779) #7769 left rustfs/src/connect/diagnostics/perf_site_replication.rs failing to compile: a local binding named object_url shadowed the object_url() helper in the same scope (E0618), and a test put the consent nonce on the request instead of LocalSiteReplicationConsent (E0063/E0560). Rename the binding to source_url and move the test nonce into the consent. * fix(rustfs): restore test compilation (#7780) * feat(connect): sign site replication target pair (#7781) * refactor(filemeta): own RestoreStatus and drop the s3s dependency (A3c) (#7782) Replace s3s::dto::{RestoreStatus, Timestamp} in filemeta with a filemeta-owned RestoreStatus (same field names, OffsetDateTime expiry). RestoreStatusOps and parse_restore_obj_status keep their signatures and persisted rendering; the ecstore restore finalize / lifecycle restore writers and rustfs RestoreObject now build the filemeta type. - crates/filemeta: drop s3s; tokio "time" becomes a dev-dependency (the metacache tests got it through s3s feature unification). - metadata_keys: drop the s3s half of the historical-source cross-check; PINNED, the pre-A3a fixture and per-character mutation still pin keys. - new test re-renders the pre-A3a fixture restore value byte-for-byte. - s3s footprint baselines 210 -> 209 files, ecstore 37 -> 36. Refs rustfs/backlog#1735 * feat(connect): add service license relay import (#7784) * feat(admin): add a gateway key inventory for the S3 stack switch (#7785) GET /rustfs/admin/v3/gateway-key-inventory (admin:InspectData) lists every stored object key the RustFS S3 gateway would refuse on every operation, so an operator can copy those objects to a safe key before switching stacks. The rules mirror the gateway key floor; . and .. segments never appear because ecstore refuses them on every write. Refs rustfs/gateway#754 * feat(server): add RUSTFS_S3_STACK with a gateway GetBucketLocation path (#7786) * fix(connect): restore the site replication producer build The producer shadowed its object_url helper with a local of the same name, and its test placed the consent nonce on the request, so neither the rustfs library nor its tests compiled on main. * feat(server): add RUSTFS_S3_STACK with a gateway GetBucketLocation path RUSTFS_S3_STACK=legacy (default) keeps the s3s service as the whole S3 entry. RUSTFS_S3_STACK=gateway routes GetBucketLocation through the RustFS Gateway pipeline pinned at rustfs/gateway@90b83a20 and falls back to the same s3s service for every other request, decided before the body is read. Refs rustfs/backlog#1752. * revert: remove M1 gateway git dependency until CI can access rustfs/gateway (#7787) Revert "feat(server): add RUSTFS_S3_STACK with a gateway GetBucketLocation path (#7786)" This reverts commit b5c2e1da90cc74088cf9b208d51871db50caeb57. * fix: retry IAM migration quorum failures on startup (#7788) * fix: retry interrupted uploads through proxies (#7789) fix: retry proxied upload interruptions * fix(connect): capture telemetry from server runtime (#7790) * fix: classify explicit proxy failures (#7791) * fix: skip IAM migration when legacy volume is absent (#7792) * fix(release): include CLI in Linux artifacts (#7794) * fix(release): serialize Linux binary links (#7797) * feat(connect): expose live lock count snapshots (#7801) * feat(connect): capture live API activity (#7804) * feat(connect): collect native Linux thread states (#7806) Collect bounded aggregate thread states from Linux procfs while preserving explicit unsupported outcomes for unavailable scopes. * feat(connect): capture live RPC activity (#7807) * test(connect): verify top.disk release artifacts (#7808) * test(connect): require native top.disk runner (#7809) * feat(connect): capture bounded local CPU profiles (#7811) * ci: retain top disk acceptance archive (#7812) * fix(connect): observe host traffic in top net (#7814) * ci: enable CPU profiling in supported release builds (#7815) * test(connect): verify profile cpu release artifacts (#7818) * test(connect): retain CPU profile verification key (#7819) * test(connect): retain CPU profile import bundle (#7820) * feat(connect): expose object inspect CLI (#7821) * feat(connect): wrap diagnostic reports for upload (#7822) * fix(connect): parse inspect paths as paths (#7826) * feat(connect): verify typed diagnostic jobs (#7827) * fix(connect): accept KMS report upload authorization (#7828) * Run signed diagnostic jobs in the RustFS service (#7829) feat(connect): run diagnostic jobs in service * fix(connect): emit canonical report manifest time (#7830) * Upload diagnostic job results through support bundles (#7831) feat(connect): upload diagnostic job results * fix(connect): emit canonical top envelope time (#7836) * fix(connect): publish diagnostic producer capabilities (#7837) * fix(io-metrics): drop the rustfs-common edge by injecting the S3 telemetry observer (#7795) * fix(io-metrics): drop the rustfs-common edge by injecting the S3 telemetry observer dc700d366 (#7775) made the io-metrics leaf crate depend on rustfs-common to publish typed telemetry trace events from S3HttpRequestGuard, which breaks the backlog#1834 leaf-crate rule (io-metrics may only depend on rustfs-s3-ops). S3HttpRequestGuard now takes an optional fn-pointer completion observer (operation, latency, 2xx?) and knows nothing about the trace bus. The server builds guards through rustfs::server::s3_http_request_guard, which attaches the observer only while a telemetry subscriber exists and owns the S3Operation -> TelemetryTraceOperation mapping. * chore(guard): admit #7785 s3s ratchet growth (+1 file, +4 lines) #7785 landed the gateway key inventory admin handler after the baselines were verified, leaving check_s3s_footprint.sh red on main and every branch cut from it (measured 210 files / 1592 lines vs 209/1588 baselines). The handler follows the house admin convention whose Operation::call signature is s3s-typed (S3Request/S3Result), so it cannot route through a non-s3s seam until the s3gate admin migration replaces the admin router (rustfs/backlog#1677 F1); no local refactor can shed the file-level import. Record the measured growth with rationale: files 209 -> 210, s3_error! lines 1588 -> 1592. * fix * fix: resolve clippy gate failures in connect and CLI surfaces - Box the two large CommandResult performance variants and LicenseRenewalOutcome::Installed (clippy::large_enum_variant) - Return RelayError instead of () from RelayTransport::deliver and the test destination (clippy::result_unit_err) - Drop an unused mut on the protected relay file handle (unused_mut) - Pass the digest by value to base64 encoding and drop a redundant result_json clone (clippy::needless_borrows_for_generic_args, clippy::redundant_clone) * fix(connect): resolve the remaining site-replication clippy errors - Group the endpoint credential triple into SiteReplicationCredentials so SiteReplicationEndpoint::new takes 6 arguments (clippy::too_many_arguments) - Take the build-feature validator by &str (clippy::ptr_arg); the validator closure stays because &String needs the deref coercion * style: apply rustfmt to the site-replication credential grouping * fix(connect): keep the CPU profile fixture off the ecstore import scan #7811 used "rustfs_ecstore::disk::read_object" as the raw-symbol fixture of the profile redaction test, which the architecture guard counts as a direct rustfs_ecstore reference outside the compatibility boundaries (its scan deliberately includes inline test modules). The accumulator treats symbols opaquely, so use the internal rustfs::storage:: path that keeps the fixture realistic without matching the import scanner. * fix * fix * fix * fix --------- Co-authored-by: Hauser * fix(replication): count a bodiless 405 as a replicated delete marker (#7756) * fix(replication): accept a bodiless 405 as a replicated delete marker during resync A resync verifies each delete marker with `HEAD ?versionId=` on the target. S3 targets (RustFS, MinIO, AWS) answer that with 405 and no body, and the SDK only synthesizes an error code for 404, so the error arrived with `code() == None`. `is_retryable_delete_replication_head_error` then treated it as an ambiguous failure: every delete marker counted as a failed object with `target service error`, and a site resync over any bucket that holds a delete marker reported the whole bucket as failed (rustfs/backlog#2479, SITE-105 on rc.6 and nightly). Use the raw HTTP status the way the 404 path already does: a 405 without a code confirms the marker propagated. Ambiguous statuses still fail. - Unit tests drive `verify_resync_head_result` against a scripted target answering 405 (accepted) and 503 (still failed). - e2e `test_site_replication_resync_replicates_delete_marker` joins two sites, converges a live object and a delete marker, and requires the site resync to complete with zero failed objects; the repl-nightly selection digest is refreshed for the new case. * fix(replication): verify marker-version purges by absence during resync A delete-marker resync entry with a pending or failed version purge asks the target to remove the marker, so the bodiless 405 that proves a created marker propagated proves the purge did not happen. Accept the 405-as-success mapping only for marker creation (empty version_purge_status); a purge counts as replicated only when the target answers not found, and any other HEAD outcome stays failed. Unit tests drive both purge statuses against the 405 fixture and the 404 fixture. * fix(kms): refuse key ids that leave the key prefix on the Vault backends (#7727) Only the Local backend refused a key identifier containing a path separator. On Vault KV2 a create with the name bad/name succeeded and produced a nested KV2 path that the listing then reported as a directory rather than a key, and an identifier containing .. addressed a record outside the configured key prefix once the HTTP client normalised the URL; Vault Transit built its transit key name and its metadata path from the same unchecked identifier. Lift the Local backend's containment rule into a shared segment check (empty, /, backslash, NUL, . and ..) and apply it at the single point where each backend turns the identifier into a path or a Transit key name, so create, describe, encrypt, delete and the metadata writes all refuse with InvalidKey before any request reaches Vault. The admin API already maps that to 400. The AWS backend is untouched: it addresses keys by ARN and alias, both of which contain /. Refs rustfs/backlog#2474 (KMS-213 CreateNegatives on vault-kv2). Co-authored-by: Hauser * Preserve safe profile job failure reasons (#7860) fix(connect): preserve safe profile job failure reasons * Freeze the heartbeat producer capability registry (#7861) test(connect): freeze heartbeat producer capabilities * fix: resume GET body after peer short EOF (#7852) * fix: resume get body after short eof Treat mid-stream short EOF errors as resumable when the S3 GET body has an attached resume context. This lets peer-kill reads reopen the same object version at the emitted offset instead of aborting the 200 response body. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * fix(connect): surface TLS peer bootstrap failures Map Connect registration TLS peer validation failures to the dedicated bootstrap error so wrong CA inputs fail closed with the sanitized TLS variant instead of the generic exchange error. Co-Authored-By: heihutu Co-Authored-By: zhi22915 --------- Co-authored-by: zhi22915 * ci: file scheduled-failure issues in rustfs/backlog (#7847) Scheduled pipeline failures currently open [scheduled-failure] tracking issues in rustfs/rustfs itself, adding bot noise to the code repository's issue tracker. Route that alerting to rustfs/backlog instead. - schedule-failure-issue gains target-repository (default rustfs/backlog); dedupe lookup, comment appends and issue creation target that repository. - New source-token input (default ${{ github.token }}) reads the failed-job list from the reported run; the run still lives in and links to rustfs/rustfs. The issue body now carries a '- Repository:' line. - All consumer workflows pass secrets.BACKLOG_ISSUE_TOKEN; the workflow-scoped GITHUB_TOKEN cannot open issues cross-repo. Alert job permissions blocks are unchanged. * fix(admin): keep the gateway key inventory off the s3s surface (#7817) The gateway key inventory handler added in #7785 imported s3s directly and built four errors with s3_error!, pushing the s3s footprint ratchet to 210 files and 1592 s3_error! lines on main. Route its S3 types and error construction through the crate::admin::storage_api::s3 facade like sibling admin handlers; the codes, HTTP statuses and messages are unchanged, and a new unit test pins them. Co-authored-by: cxymds Co-authored-by: Hauser * fix(s3): honor presigned UploadPart checksum queries (#7748) * fix(s3): honor presigned UploadPart checksum queries * fix(s3): keep checksum errors within gateway boundary --------- Co-authored-by: Hauser * fix(auth): reject unsigned x-amz headers on header-signed SigV4 requests (#7796) * fix(auth): reject unsigned x-amz headers on header-signed SigV4 requests A SigV4 request authenticated with an Authorization header only binds the headers named in its SignedHeaders list, but RustFS acted on every x-amz-* header that arrived, so a replayed header-signed PutObject carrying an unsigned x-amz-copy-source became a CopyObject run as the signer that could copy any object the signer can read (GHSA-xm99-m3gq-83g8). The presigned form was already closed by GHSA-g8w9-qw9q-fghr. reject_unsigned_amz_headers_on_sigv4_request now guards S3Access::check and S3Router::check_access: every SigV4 signed-header list the request carries must cover every x-amz-* header, the Authorization header is parsed with the verifier's own s3s-sigv4 parser and compared case-insensitively, the algorithm token is pinned to AWS4-HMAC-SHA256 because the upstream header path accepts any token, and the exempt set mirrors the upstream s3s fix (x-amz-content-sha256, x-amz-decoded-content-length, x-amz-trailer, x-amz-checksum-algorithm) plus x-amz-cf-id. Adds ghsa_xm99 unit, router and e2e regressions, raises the security smoke floor to 28, and records the advisory in docs/testing/security-regressions.md and CHANGELOG.md. * chore(deps): switch s3s and s3s-sigv4 to crates.io 0.16.0 * fix(server): enforce the SigV4 header guard ahead of s3s dispatch s3s 0.16 verifies the claimed algorithm as the first step of its own signature flow, so a request whose Authorization header swaps the AWS4-HMAC-SHA256 token was answered with 501 NotImplemented before RustFS's access layer could rule on the unsigned x-amz-copy-source (GHSA-xm99-m3gq-83g8). Add the SigV4HeaderGuardLayer as the innermost external stack layer, running reject_unsigned_amz_headers_on_sigv4_request in front of s3s and serializing its rejections as the same AccessDenied S3 error document the access layer produces. --------- Co-authored-by: Hauser * fix(table-catalog): map transient lock failures to unavailable (#7841) * Verify profile service jobs on native Linux (#7862) ci: verify profile service jobs on native Linux * Authorize the private Connect acceptance checkout (#7863) ci: authorize private Connect harness checkout * Run profile service acceptance directly (#7864) ci: run profile service acceptance directly * Retain profile service failure evidence (#7868) ci: retain profile service failure evidence * fix: update dependencies and preserve transport errors (#7798) * chore(deps): bump ed25519-dalek in the dependencies group Bumps the dependencies group with 1 update: [ed25519-dalek](https://github.com/dalek-cryptography/curve25519-dalek). Updates `ed25519-dalek` from 2.2.0 to 3.0.0 - [Release notes](https://github.com/dalek-cryptography/curve25519-dalek/releases) - [Changelog](https://github.com/dalek-cryptography/curve25519-dalek/blob/3.0.0/CHANGELOG.md) - [Commits](https://github.com/dalek-cryptography/curve25519-dalek/compare/ed25519-2.2.0...3.0.0) --- updated-dependencies: - dependency-name: ed25519-dalek dependency-version: 3.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: dependencies ... Signed-off-by: dependabot[bot] * chore(deps): bump jiff, aws-sdk-s3, datafusion and smallvec Bump the dependencies group: - jiff 0.2.35 -> 0.2.37 - aws-sdk-s3 1.146.0 -> 1.146.1 - datafusion 55.0.0 -> 55.1.0 - smallvec 1.16.0 -> 1.16.1 Update Cargo.lock for the matching transitive bumps (datafusion subcrates, hickory, cc, crc32fast, tinyvec, toml_edit and others); tinyvec_macros is no longer required. * chore(deps): use s3s 0.16.0 release and bump rust-version to 1.98.1 * fix: preserve ready reader errors during erasure write failures When erasure writers fail after the producer has already surfaced a reader error, return the reader error instead of aborting the producer and masking it with the write-side failure. This keeps UploadPart body read timeouts classified as RequestTimeout under loaded pipeline timing. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * test: use current checksum API in multipart regressions * fix: stabilize connect dependency update tests Classify Connect proxy failures only when an explicit proxy is configured, preserve TLS classification across request error shapes, keep short object performance windows from spending too much budget on cleanup, and update multipart checksum tests for the current HashReader API. Co-Authored-By: heihutu Co-Authored-By: zhi22915 --------- Signed-off-by: dependabot[bot] Signed-off-by: Hauser Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hauser Co-authored-by: zhi22915 Co-authored-by: overtrue * ci: expect BACKLOG_ISSUE_TOKEN in scheduled alert wiring checks (#7867) #7847 routed schedule-failure alerting to rustfs/backlog by switching the alert jobs' token to secrets.BACKLOG_ISSUE_TOKEN, but check_test_wiring.py still asserts the literal 'github-token: ${{ secrets.GITHUB_TOKEN }}', so Quick Checks fails on every open PR. Update the wiring assertions and the self-test fixtures to the new literal. Both `--self-test` (52 tests) and the full `validate()` pass locally. * Pace diagnostic job result redelivery (#7869) fix(connect): pace diagnostic result redelivery * Execute authenticated top.api service jobs (#7872) feat(connect): execute top.api service jobs * feat(connect): execute top.locks service jobs (#7874) * fix(ci): gate Connect acceptance on x86 build job (#7875) * fix(ci): repair E2E inventory and network CLI coverage (#7871) * ci: refresh Linux full E2E inventory after SigV4 tests * test(connect): drive traffic for top net CLI export coverage * feat(connect): execute performance network jobs (#7877) * feat(connect): execute bounded drive performance jobs (#7878) * Preserve the top API service job nonce (#7879) fix(connect): preserve top api job nonce * feat(connect): execute thread profile service jobs (#7880) * Execute authenticated top.rpc service jobs (#7881) feat(connect): execute top.rpc service jobs * ci: add service diagnostic acceptance workflows (#7882) * ci: add threads and RPC acceptance workflows (#7883) * fix(connect): deliver diagnostic results until expiry (#7886) * feat(connect): export signed environment inventory (#7887) feat: export signed environment inventory * fix: recover interrupted GETs and heal operations (#7876) * fix(heal): retry interrupted internode shard writes * fix(deps): update rustls for RUSTSEC-2026-0285 * fix(heal): retain RPC transport failures across metadata probes * test(heal): wait for replacement disk and coordinator readiness * test(connect): validate service-only diagnostic capabilities * fix(s3): resume GET bodies after erasure read quorum loss * fix: keep top locks within its capture window (#7888) * fix: recover heal after peer boot epoch changes (#7889) * test(heal): keep outage objects in the replacement erasure set * fix(rpc): retry authenticated requests after peer epoch changes * test(rpc): assert retryable rejection of stale boot epochs * ci: bound concurrent heal clusters in the full E2E gate * ci: register the table suite workflow on the default branch (#7893) * ci: register the table suite workflow on the default branch The table suite workflow (#7873) only exists on release so far, and GitHub registers workflow_dispatch targets exclusively from the default branch — the suite cannot be triggered until this file lands on main (same mechanism that keeps the fault-tolerance suite undispatchable). This adds the identical rustfs-table-test.yml (byte-for-byte the release version) plus the contract-test registration (JOBS / DIRECT_TESTS / upload allowlist), and nothing else from release. * ci: pin actions in the table workflow to full commit SHAs The default branch enforces check_workflow_pins.sh (unpinned third-party action refs fail CI). Use the same pinned SHAs as the other suite workflows on main: actions/checkout@9c091bb... (#v7) and actions/upload-artifact@b7c566a... (#v6). Note: the release copy of this workflow still uses @v4 tags; release does not run the pin gate. Release and main will converge on the pinned form during the full sync. * test(ecstore): synchronize causal cleanup observations (#7894) * fix(ecstore): create markers for repeated versioned deletes (#7884) Co-authored-by: cxymds * Preserve top.locks signed job nonce (#7898) fix(connect): preserve top locks job nonce * fix(ecstore): roll back failed delete marker writes (#7899) * chore(deps): refresh RustFS tokio tar dependency (#7907) * chore(deps): refresh RustFS tokio tar dependency Switch the astral-tokio-tar workspace alias to the released rustfs-tokio-tar 0.7.1 crate and remove the obsolete cargo-deny git source allowance. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * chore(deps): align AWS smithy dependency family Upgrade the AWS SDK and smithy crates together while patching aws-smithy-json 0.63.0 for the aws-smithy-types 1.7.0 DocumentObject API used by aws-config 1.12.0. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * revert(deps): use official AWS smithy compatible set Remove the local aws-smithy-json patch and roll the AWS smithy family back to the latest official combination that compiles with aws-config 1.12.0. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * docs(architecture): retire tokio tar cleanup entry Remove the stale cleanup-register entry after switching back to the released rustfs-tokio-tar crate, leaving no unmatched compatibility marker for the architecture guard. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * ci: restore table workflow failure contract Align the table functional workflow with the quick-check guard by letting suite failures stay red, generating PASS reports only from complete successful evidence, and preserving the shared backlog issue fallback for harness failures. Co-Authored-By: heihutu Co-Authored-By: zhi22915 * fix(deps): expose rustfs tokio tar dependency name Use the published rustfs-tokio-tar dependency key throughout the workspace while keeping source imports on the crate's tokio_tar lib target name. Co-Authored-By: heihutu Co-Authored-By: zhi22915 --------- Co-authored-by: zhi22915 * fix(ci): retain failed functional chain reports and check runners (#7906) --------- Signed-off-by: houseme Signed-off-by: dependabot[bot] Signed-off-by: Hauser Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> Co-authored-by: Henry Guo Co-authored-by: 唐小鸭 Co-authored-by: houseme Co-authored-by: zhi22915 Co-authored-by: cxymds Co-authored-by: cui fliter Co-authored-by: RJ Regenold Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: Jason Kossis Co-authored-by: Hiroaki KAWAI Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .../references/advisory-patterns.md | 6 +- .config/functional-script-revision.txt | 1 + .config/make/tests.mak | 3 + .../actions/schedule-failure-issue/README.md | 34 +- .../actions/schedule-failure-issue/action.yml | 54 +- .github/workflows/audit.yml | 2 +- .github/workflows/build.yml | 22 +- .github/workflows/ci.yml | 4 +- .../connect-performance-drive-acceptance.yml | 192 ++ ...connect-performance-network-acceptance.yml | 192 ++ .../connect-profile-cpu-acceptance.yml | 192 ++ .../connect-profile-threads-acceptance.yml | 192 ++ .../workflows/connect-top-api-acceptance.yml | 192 ++ .../workflows/connect-top-disk-acceptance.yml | 142 ++ .../connect-top-locks-acceptance.yml | 192 ++ .../workflows/connect-top-rpc-acceptance.yml | 192 ++ .github/workflows/coverage.yml | 2 +- .github/workflows/e2e-distributed.yml | 2 +- .github/workflows/e2e-replication-nightly.yml | 2 +- .github/workflows/e2e-s3tests.yml | 2 +- .github/workflows/functional-chain-health.yml | 31 + .github/workflows/fuzz.yml | 2 +- .github/workflows/minio-interop.yml | 2 +- .github/workflows/mint.yml | 2 +- .github/workflows/nightly-gnu.yml | 2 +- .github/workflows/oidc-keycloak.yml | 2 +- .../workflows/on-demand-migration-interop.yml | 2 +- .github/workflows/performance-ab.yml | 2 +- .github/workflows/runner-hygiene.yml | 2 +- .../workflows/rustfs-fault-tolerance-test.yml | 91 +- .github/workflows/rustfs-functional-chain.yml | 217 ++- .github/workflows/rustfs-heal-test.yml | 69 +- .github/workflows/rustfs-kms-test.yml | 67 +- .github/workflows/rustfs-performance-test.yml | 83 +- .github/workflows/rustfs-pool-expand-test.yml | 85 +- .github/workflows/rustfs-replication-test.yml | 67 +- .github/workflows/rustfs-s3-compat-test.yml | 67 +- .github/workflows/rustfs-security-test.yml | 66 +- .github/workflows/rustfs-storage-test.yml | 67 +- .github/workflows/rustfs-table-test.yml | 85 +- .github/workflows/rustfs-tier-test.yml | 72 +- .github/workflows/rustfs-upgrade-test.yml | 64 +- .../schedule-failure-alert-drill.yml | 6 +- .../scheduled-validation-freshness.yml | 2 +- .../scheduled-validation-watchdog.yml | 2 +- .github/workflows/targets-integration.yml | 2 +- Cargo.lock | 335 ++-- Cargo.toml | 30 +- crates/common/src/trace_bus.rs | 124 ++ crates/config/README.md | 31 + crates/config/src/constants/app.rs | 5 + crates/e2e_test/Cargo.toml | 2 +- crates/ecstore/Cargo.toml | 1 - crates/ecstore/src/api/mod.rs | 29 +- .../bucket/lifecycle/bucket_lifecycle_ops.rs | 28 +- .../bucket/lifecycle/object_lock_boundary.rs | 2 +- crates/ecstore/src/bucket/migration.rs | 51 +- .../src/bucket/object_lock/objectlock.rs | 10 +- .../src/bucket/object_lock/objectlock_sys.rs | 49 +- .../replication_filemeta_boundary.rs | 13 + .../replication/replication_object_config.rs | 4 +- .../replication/replication_resyncer.rs | 11 +- .../replication_target_boundary.rs | 49 +- crates/ecstore/src/cluster/rpc/mod.rs | 5 + .../ecstore/src/cluster/rpc/network_probe.rs | 318 +++ crates/ecstore/src/data_movement/mod.rs | 34 +- crates/ecstore/src/io_support/rio.rs | 2 +- crates/ecstore/src/object_api/mod.rs | 8 +- .../persisted_metadata_keys_tests.rs | 192 ++ crates/ecstore/src/object_api/types.rs | 15 +- .../ecstore/src/services/tier/warm_backend.rs | 75 +- crates/ecstore/src/set_disk/metadata.rs | 3 +- crates/ecstore/src/set_disk/mod.rs | 48 +- crates/ecstore/src/set_disk/ops/multipart.rs | 57 +- crates/ecstore/src/set_disk/ops/object.rs | 249 ++- crates/ecstore/src/set_disk/replication.rs | 17 +- .../src/set_disk/transition_matrix_tests.rs | 2 +- crates/ecstore/src/store/init.rs | 66 +- crates/ecstore/src/store/object.rs | 96 +- crates/ecstore/src/store/utils.rs | 7 +- crates/filemeta/Cargo.toml | 4 +- crates/filemeta/src/fileinfo.rs | 74 +- crates/filemeta/src/filemeta.rs | 58 +- crates/filemeta/src/filemeta/inline_data.rs | 3 +- crates/filemeta/src/filemeta/version.rs | 12 +- crates/filemeta/src/lib.rs | 1 + crates/filemeta/src/metadata_keys.rs | 258 +++ crates/filemeta/src/test_data.rs | 8 + .../persisted_metadata_keys_pre_a3a.hex | 1 + .../tests/admin_pool_set_contract_test.rs | 12 +- crates/heal/tests/mrf_partial_write_test.rs | 32 +- crates/io-metrics/src/lib.rs | 1 + crates/io-metrics/src/s3_http_metrics.rs | 52 + crates/lifecycle/src/evaluator.rs | 12 +- crates/lifecycle/src/object_lock.rs | 28 +- crates/lock/src/fast_lock/manager.rs | 10 + crates/lock/src/fast_lock/shard.rs | 10 + crates/lock/src/fast_lock/state.rs | 7 + crates/lock/src/fast_lock/tests.rs | 34 + crates/lock/src/lib.rs | 5 + crates/protocols/Cargo.toml | 4 +- crates/protocols/src/swift/bulk.rs | 2 +- crates/rio/Cargo.toml | 1 - crates/rio/src/hash_reader.rs | 246 ++- crates/rio/src/lib.rs | 3 + crates/rio/src/trailer.rs | 66 + crates/signer/src/lib.rs | 1 + crates/signer/src/request_signature_v4.rs | 13 + crates/storage-api/src/lib.rs | 4 + crates/zip/Cargo.toml | 2 +- deny.toml | 9 +- docs/architecture/compat-cleanup-register.md | 2 +- docs/testing/ci-timing.md | 9 + docs/testing/functional-chain.md | 29 + flake.lock | 12 +- .../agent/v1/fixtures/bundle/MANIFEST.sha256 | 1 + .../bundle/typed-offline-import-vectors.json | 550 ++++++ .../diagnostic-scheduler/MANIFEST.sha256 | 3 + .../diagnostic-scheduler/accept-vectors.json | 169 ++ .../diagnostic-scheduler/receipt-vectors.json | 69 + .../diagnostic-scheduler/reject-vectors.json | 112 ++ protocol/agent/v1/fixtures/fixture-sets.json | 27 +- .../v1/fixtures/heartbeat/MANIFEST.sha256 | 5 +- .../heartbeat/producer-capabilities.json | 54 + .../agent/v1/fixtures/heartbeat/unknown.json | 6 +- .../agent/v1/fixtures/heartbeat/valid.json | 2 +- .../v1/fixtures/inventory/MANIFEST.sha256 | 1 + .../v1/fixtures/inventory/accept-vectors.json | 58 + protocol/agent/v1/fixtures/manifest.sha256 | 13 + .../network-performance/MANIFEST.sha256 | 2 + .../network-performance/accept-vectors.json | 294 +++ .../network-performance/reject-vectors.json | 505 +++++ .../object-performance/MANIFEST.sha256 | 2 + .../object-performance/accept-vectors.json | 222 +++ .../object-performance/reject-vectors.json | 319 ++++ .../MANIFEST.sha256 | 2 + .../accept-vectors.json | 184 ++ .../reject-vectors.json | 590 ++++++ .../v1/fixtures/telemetry/MANIFEST.sha256 | 2 + .../v1/fixtures/telemetry/accept-vectors.json | 138 ++ .../v1/fixtures/telemetry/reject-vectors.json | 28 + rustfs/Cargo.toml | 11 +- rustfs/src/admin/console.rs | 189 +- rustfs/src/admin/handlers/diagnostics.rs | 202 +- .../admin/handlers/gateway_key_inventory.rs | 421 ++++ rustfs/src/admin/handlers/mod.rs | 1 + rustfs/src/admin/handlers/oidc.rs | 32 +- rustfs/src/admin/handlers/system.rs | 28 +- rustfs/src/admin/mod.rs | 5 +- rustfs/src/admin/route_policy.rs | 18 + rustfs/src/admin/route_registration_test.rs | 2 + rustfs/src/app/mod.rs | 1 + rustfs/src/app/multipart_usecase.rs | 9 +- rustfs/src/app/object/extract.rs | 3 +- rustfs/src/app/object/mod.rs | 8 +- rustfs/src/app/object/put.rs | 5 +- .../app/object/request_body/tests/protocol.rs | 37 +- rustfs/src/app/object/restore.rs | 5 +- rustfs/src/app/storage_api.rs | 2 +- rustfs/src/app/trailer_adapter.rs | 44 + rustfs/src/config/cli.rs | 1431 +++++++++++++- rustfs/src/config/config_test.rs | 33 + rustfs/src/config/mod.rs | 22 + rustfs/src/config/opt.rs | 28 +- rustfs/src/connect/client.rs | 130 +- rustfs/src/connect/config.rs | 436 ++++- rustfs/src/connect/diagnostics/inspect.rs | 1412 ++++++++++++++ rustfs/src/connect/diagnostics/job.rs | 1697 +++++++++++++++++ .../src/connect/diagnostics/job_delivery.rs | 821 ++++++++ rustfs/src/connect/diagnostics/logs.rs | 878 +++++++++ rustfs/src/connect/diagnostics/mod.rs | 165 ++ rustfs/src/connect/diagnostics/perf_client.rs | 1255 ++++++++++++ rustfs/src/connect/diagnostics/perf_drive.rs | 1145 +++++++++++ .../src/connect/diagnostics/perf_network.rs | 1018 ++++++++++ rustfs/src/connect/diagnostics/perf_object.rs | 1345 +++++++++++++ .../diagnostics/perf_site_replication.rs | 1672 ++++++++++++++++ rustfs/src/connect/diagnostics/profile_cpu.rs | 1175 ++++++++++++ .../src/connect/diagnostics/profile_memory.rs | 163 ++ .../connect/diagnostics/profile_threads.rs | 200 ++ .../connect/diagnostics/receipt_delivery.rs | 112 ++ rustfs/src/connect/diagnostics/schedule.rs | 735 +++++++ rustfs/src/connect/diagnostics/top_api.rs | 1132 +++++++++++ rustfs/src/connect/diagnostics/top_disk.rs | 130 ++ rustfs/src/connect/diagnostics/top_locks.rs | 84 + rustfs/src/connect/diagnostics/top_net.rs | 115 ++ rustfs/src/connect/diagnostics/top_rpc.rs | 112 ++ .../src/connect/diagnostics/trace_analysis.rs | 149 ++ rustfs/src/connect/diagnostics/trace_otlp.rs | 271 +++ .../src/connect/diagnostics/trace_record.rs | 1118 +++++++++++ .../src/connect/diagnostics/trace_replay.rs | 127 ++ .../src/connect/diagnostics/trace_runtime.rs | 526 +++++ .../diagnostics/trace_runtime_unsupported.rs | 57 + rustfs/src/connect/environment.rs | 864 +++++++++ rustfs/src/connect/heartbeat.rs | 92 +- rustfs/src/connect/license.rs | 774 ++++++++ rustfs/src/connect/license_relay.rs | 377 ++++ rustfs/src/connect/license_renewal.rs | 636 ++++++ rustfs/src/connect/mod.rs | 105 +- rustfs/src/connect/offline/collectors.rs | 202 +- rustfs/src/connect/registration_bootstrap.rs | 21 +- rustfs/src/connect/relay.rs | 728 +++++++ rustfs/src/connect/report_bundle.rs | 603 ++++++ rustfs/src/connect/report_upload.rs | 713 +++++++ rustfs/src/connect/runtime.rs | 114 +- rustfs/src/connect/telemetry.rs | 53 +- rustfs/src/server/compress.rs | 8 +- rustfs/src/server/http.rs | 157 +- rustfs/src/server/layer.rs | 339 +++- rustfs/src/server/mod.rs | 3 +- rustfs/src/server/prefix.rs | 223 ++- rustfs/src/server/rate_limit.rs | 21 +- rustfs/src/server/readiness.rs | 16 +- rustfs/src/startup_entrypoint.rs | 1564 ++++++++++++++- rustfs/src/startup_lifecycle.rs | 4 + rustfs/src/startup_server.rs | 1 + rustfs/src/startup_services.rs | 36 +- rustfs/src/storage/rpc/http_service.rs | 40 +- rustfs/src/storage/storage_api.rs | 12 +- rustfs/src/storage_api.rs | 4 + rustfs/tests/agent_protocol_fixtures.rs | 37 +- rustfs/tests/connect_diagnostic_schedule.rs | 142 ++ rustfs/tests/connect_environment.rs | 184 ++ rustfs/tests/connect_heartbeat.rs | 167 +- rustfs/tests/connect_inspect.rs | 466 +++++ rustfs/tests/connect_inventory.rs | 2 + rustfs/tests/connect_license.rs | 652 +++++++ rustfs/tests/connect_logs.rs | 372 ++++ rustfs/tests/connect_perf_client.rs | 584 ++++++ rustfs/tests/connect_perf_drive.rs | 672 +++++++ rustfs/tests/connect_perf_network.rs | 468 +++++ rustfs/tests/connect_perf_object.rs | 628 ++++++ rustfs/tests/connect_profile_cpu.rs | 175 ++ rustfs/tests/connect_profile_memory.rs | 290 +++ rustfs/tests/connect_profile_threads.rs | 160 ++ rustfs/tests/connect_registration.rs | 312 ++- .../tests/connect_registration_bootstrap.rs | 2 +- rustfs/tests/connect_relay.rs | 300 +++ rustfs/tests/connect_report_upload.rs | 447 +++++ rustfs/tests/connect_top_api.rs | 125 ++ rustfs/tests/connect_top_disk.rs | 106 + rustfs/tests/connect_top_locks.rs | 78 + rustfs/tests/connect_top_net.rs | 513 +++++ rustfs/tests/connect_top_rpc.rs | 184 ++ rustfs/tests/connect_trace_analysis.rs | 46 + rustfs/tests/connect_trace_otlp.rs | 222 +++ rustfs/tests/connect_trace_record.rs | 568 ++++++ rustfs/tests/connect_trace_replay.rs | 135 ++ .../connect-license-ed25519-vector.json | 7 + rustfs/tests/fixtures/connect-logs-v1.json | 55 + scripts/check_embedded_secrets.sh | 3 + scripts/check_functional_runners.py | 27 + scripts/check_s3s_footprint.sh | 10 +- scripts/check_test_wiring.py | 30 +- .../ci/check_connect_profile_cpu_artifact.sh | 291 +++ scripts/ci/check_connect_top_disk_artifact.sh | 181 ++ scripts/ci_timing_report.py | 148 ++ scripts/functional_chain_evidence.py | 239 +++ scripts/functional_chain_health.py | 150 ++ scripts/prepare_functional_package.py | 101 + scripts/resolve_functional_candidate.py | 124 ++ .../check_preview_release_workflow.sh | 5 + scripts/test_ci_timing_report.py | 74 + scripts/test_functional_chain.py | 367 ++++ scripts/test_functional_chain_health.py | 108 ++ scripts/test_nightly_candidate.py | 2 + scripts/test_prepare_functional_package.py | 73 + scripts/test_security_workflow.py | 43 +- 267 files changed, 46210 insertions(+), 1422 deletions(-) create mode 100644 .config/functional-script-revision.txt create mode 100644 .github/workflows/connect-performance-drive-acceptance.yml create mode 100644 .github/workflows/connect-performance-network-acceptance.yml create mode 100644 .github/workflows/connect-profile-cpu-acceptance.yml create mode 100644 .github/workflows/connect-profile-threads-acceptance.yml create mode 100644 .github/workflows/connect-top-api-acceptance.yml create mode 100644 .github/workflows/connect-top-disk-acceptance.yml create mode 100644 .github/workflows/connect-top-locks-acceptance.yml create mode 100644 .github/workflows/connect-top-rpc-acceptance.yml create mode 100644 .github/workflows/functional-chain-health.yml create mode 100644 crates/ecstore/src/cluster/rpc/network_probe.rs create mode 100644 crates/ecstore/src/object_api/persisted_metadata_keys_tests.rs create mode 100644 crates/filemeta/src/metadata_keys.rs create mode 100644 crates/filemeta/tests/fixtures/persisted_metadata_keys_pre_a3a.hex create mode 100644 crates/rio/src/trailer.rs create mode 100644 docs/testing/ci-timing.md create mode 100644 docs/testing/functional-chain.md create mode 100644 protocol/agent/v1/fixtures/bundle/typed-offline-import-vectors.json create mode 100644 protocol/agent/v1/fixtures/diagnostic-scheduler/MANIFEST.sha256 create mode 100644 protocol/agent/v1/fixtures/diagnostic-scheduler/accept-vectors.json create mode 100644 protocol/agent/v1/fixtures/diagnostic-scheduler/receipt-vectors.json create mode 100644 protocol/agent/v1/fixtures/diagnostic-scheduler/reject-vectors.json create mode 100644 protocol/agent/v1/fixtures/heartbeat/producer-capabilities.json create mode 100644 protocol/agent/v1/fixtures/inventory/accept-vectors.json create mode 100644 protocol/agent/v1/fixtures/manifest.sha256 create mode 100644 protocol/agent/v1/fixtures/network-performance/MANIFEST.sha256 create mode 100644 protocol/agent/v1/fixtures/network-performance/accept-vectors.json create mode 100644 protocol/agent/v1/fixtures/network-performance/reject-vectors.json create mode 100644 protocol/agent/v1/fixtures/object-performance/MANIFEST.sha256 create mode 100644 protocol/agent/v1/fixtures/object-performance/accept-vectors.json create mode 100644 protocol/agent/v1/fixtures/object-performance/reject-vectors.json create mode 100644 protocol/agent/v1/fixtures/site-replication-performance/MANIFEST.sha256 create mode 100644 protocol/agent/v1/fixtures/site-replication-performance/accept-vectors.json create mode 100644 protocol/agent/v1/fixtures/site-replication-performance/reject-vectors.json create mode 100644 protocol/agent/v1/fixtures/telemetry/MANIFEST.sha256 create mode 100644 protocol/agent/v1/fixtures/telemetry/accept-vectors.json create mode 100644 protocol/agent/v1/fixtures/telemetry/reject-vectors.json create mode 100644 rustfs/src/admin/handlers/gateway_key_inventory.rs create mode 100644 rustfs/src/app/trailer_adapter.rs create mode 100644 rustfs/src/connect/diagnostics/inspect.rs create mode 100644 rustfs/src/connect/diagnostics/job.rs create mode 100644 rustfs/src/connect/diagnostics/job_delivery.rs create mode 100644 rustfs/src/connect/diagnostics/logs.rs create mode 100644 rustfs/src/connect/diagnostics/mod.rs create mode 100644 rustfs/src/connect/diagnostics/perf_client.rs create mode 100644 rustfs/src/connect/diagnostics/perf_drive.rs create mode 100644 rustfs/src/connect/diagnostics/perf_network.rs create mode 100644 rustfs/src/connect/diagnostics/perf_object.rs create mode 100644 rustfs/src/connect/diagnostics/perf_site_replication.rs create mode 100644 rustfs/src/connect/diagnostics/profile_cpu.rs create mode 100644 rustfs/src/connect/diagnostics/profile_memory.rs create mode 100644 rustfs/src/connect/diagnostics/profile_threads.rs create mode 100644 rustfs/src/connect/diagnostics/receipt_delivery.rs create mode 100644 rustfs/src/connect/diagnostics/schedule.rs create mode 100644 rustfs/src/connect/diagnostics/top_api.rs create mode 100644 rustfs/src/connect/diagnostics/top_disk.rs create mode 100644 rustfs/src/connect/diagnostics/top_locks.rs create mode 100644 rustfs/src/connect/diagnostics/top_net.rs create mode 100644 rustfs/src/connect/diagnostics/top_rpc.rs create mode 100644 rustfs/src/connect/diagnostics/trace_analysis.rs create mode 100644 rustfs/src/connect/diagnostics/trace_otlp.rs create mode 100644 rustfs/src/connect/diagnostics/trace_record.rs create mode 100644 rustfs/src/connect/diagnostics/trace_replay.rs create mode 100644 rustfs/src/connect/diagnostics/trace_runtime.rs create mode 100644 rustfs/src/connect/diagnostics/trace_runtime_unsupported.rs create mode 100644 rustfs/src/connect/environment.rs create mode 100644 rustfs/src/connect/license.rs create mode 100644 rustfs/src/connect/license_relay.rs create mode 100644 rustfs/src/connect/license_renewal.rs create mode 100644 rustfs/src/connect/relay.rs create mode 100644 rustfs/src/connect/report_bundle.rs create mode 100644 rustfs/src/connect/report_upload.rs create mode 100644 rustfs/tests/connect_diagnostic_schedule.rs create mode 100644 rustfs/tests/connect_environment.rs create mode 100644 rustfs/tests/connect_inspect.rs create mode 100644 rustfs/tests/connect_license.rs create mode 100644 rustfs/tests/connect_logs.rs create mode 100644 rustfs/tests/connect_perf_client.rs create mode 100644 rustfs/tests/connect_perf_drive.rs create mode 100644 rustfs/tests/connect_perf_network.rs create mode 100644 rustfs/tests/connect_perf_object.rs create mode 100644 rustfs/tests/connect_profile_cpu.rs create mode 100644 rustfs/tests/connect_profile_memory.rs create mode 100644 rustfs/tests/connect_profile_threads.rs create mode 100644 rustfs/tests/connect_relay.rs create mode 100644 rustfs/tests/connect_report_upload.rs create mode 100644 rustfs/tests/connect_top_api.rs create mode 100644 rustfs/tests/connect_top_disk.rs create mode 100644 rustfs/tests/connect_top_locks.rs create mode 100644 rustfs/tests/connect_top_net.rs create mode 100644 rustfs/tests/connect_top_rpc.rs create mode 100644 rustfs/tests/connect_trace_analysis.rs create mode 100644 rustfs/tests/connect_trace_otlp.rs create mode 100644 rustfs/tests/connect_trace_record.rs create mode 100644 rustfs/tests/connect_trace_replay.rs create mode 100644 rustfs/tests/fixtures/connect-license-ed25519-vector.json create mode 100644 rustfs/tests/fixtures/connect-logs-v1.json create mode 100644 scripts/check_functional_runners.py create mode 100755 scripts/ci/check_connect_profile_cpu_artifact.sh create mode 100755 scripts/ci/check_connect_top_disk_artifact.sh create mode 100644 scripts/ci_timing_report.py create mode 100644 scripts/functional_chain_evidence.py create mode 100644 scripts/functional_chain_health.py create mode 100644 scripts/prepare_functional_package.py create mode 100644 scripts/resolve_functional_candidate.py create mode 100644 scripts/test_ci_timing_report.py create mode 100644 scripts/test_functional_chain.py create mode 100644 scripts/test_functional_chain_health.py create mode 100644 scripts/test_prepare_functional_package.py diff --git a/.agents/skills/security-advisory-lessons/references/advisory-patterns.md b/.agents/skills/security-advisory-lessons/references/advisory-patterns.md index 70b388d72..ca7117347 100644 --- a/.agents/skills/security-advisory-lessons/references/advisory-patterns.md +++ b/.agents/skills/security-advisory-lessons/references/advisory-patterns.md @@ -48,7 +48,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect ### S3 object actions, copy, multipart, and upload policy validation -- `GHSA-g8w9-qw9q-fghr`: a valid presigned `PutObject` accepted extra `x-amz-tagging`, website redirect, and storage-class headers omitted from `SignedHeaders`. Lesson: a presigned URL is a bounded capability; reject `x-amz-*` headers that are not cryptographically bound by the signature so unsigned metadata cannot change authorization, lifecycle, redirect, cost, or durability semantics. +- `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage. - `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial. - `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies. - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`. @@ -120,7 +120,7 @@ Use these targeted searches when a diff touches security-sensitive code: ```bash rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs -rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|presign|SignedHeaders|content-length-range|starts-with" rustfs crates +rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|presign|SignedHeaders|x-amz-copy-source|collect_signed_headers|content-length-range|starts-with" rustfs crates rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates @@ -137,7 +137,7 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates - Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend. - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy. - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases. -- Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, and storage-class headers; require rejection before storage access, and verify explicitly signed equivalents still work. +- Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work. - Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases. - Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode. - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks. diff --git a/.config/functional-script-revision.txt b/.config/functional-script-revision.txt new file mode 100644 index 000000000..93019ecd1 --- /dev/null +++ b/.config/functional-script-revision.txt @@ -0,0 +1 @@ +27e9584a2d776db9416d0edfd040d524d48b5d4d diff --git a/.config/make/tests.mak b/.config/make/tests.mak index fa4fda911..56d5d79df 100644 --- a/.config/make/tests.mak +++ b/.config/make/tests.mak @@ -55,6 +55,9 @@ script-tests: ## Run shell script tests $(RUSTFS_PYTHON_BIN) ./scripts/check_scheduled_validation_freshness.py --self-test $(RUSTFS_PYTHON_BIN) ./scripts/test_security_workflow.py $(RUSTFS_PYTHON_BIN) ./scripts/test_nightly_candidate.py + $(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain.py + $(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py + $(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py $(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py bash -n ./scripts/validate_object_data_cache_cold_stampede.sh $(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test diff --git a/.github/actions/schedule-failure-issue/README.md b/.github/actions/schedule-failure-issue/README.md index d2e780f69..e704c0e77 100644 --- a/.github/actions/schedule-failure-issue/README.md +++ b/.github/actions/schedule-failure-issue/README.md @@ -5,20 +5,38 @@ This is the single alerting mechanism for all timed pipelines (rustfs/backlog#1149 ci-8, arbitration G3): scheduled workflows must consume this action instead of inventing their own notification paths. +Issues are filed in **rustfs/backlog** (override with `target-repository`), +not the repository the workflow runs in, so scheduled-failure noise stays +out of the code repository's issue tracker. + ## Behavior - Issue title: `[scheduled-failure] ` — the workflow name is the dedupe key. -- If an **open** issue with that exact title exists, the failure is appended - as a comment; otherwise a new issue is created (labeled `infrastructure` by - default). Closing the issue resets the cycle: the next failure opens a - fresh one. -- The issue body / comment includes the run URL, run attempt, event, ref and - the names of the failed (or timed-out/cancelled) jobs of the current run - attempt. +- If an **open** issue with that exact title exists in the target + repository, the failure is appended as a comment; otherwise a new issue is + created (labeled `infrastructure` by default; the label must exist in the + target repository). Closing the issue resets the cycle: the next failure + opens a fresh one. +- The issue body / comment includes the source repository, run URL, run + attempt, event, ref and the names of the failed (or timed-out/cancelled) + jobs of the current run attempt. - Requires `gh` and `jq` on the runner (both preinstalled on GitHub-hosted runners such as `ubuntu-latest`). +## Tokens + +Two tokens are involved: + +- `github-token` authenticates issue create/comment/search against the + target repository. The workflow-scoped `secrets.GITHUB_TOKEN` is scoped to + the code repository and **cannot** open issues in rustfs/backlog, so this + must be a PAT or GitHub App token with `issues: write` on the target + repository, stored as the `BACKLOG_ISSUE_TOKEN` repository secret. +- `source-token` (defaults to `${{ github.token }}`) reads the failed-job + list from the reported run, which lives in the repository running the + workflow. + ## Usage Add a final job to the workflow. `always()` is required — without it the job @@ -41,7 +59,7 @@ this job only; no other job may gain permissions. - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} ``` Notes: diff --git a/.github/actions/schedule-failure-issue/action.yml b/.github/actions/schedule-failure-issue/action.yml index d505387e4..76b3e7fa7 100644 --- a/.github/actions/schedule-failure-issue/action.yml +++ b/.github/actions/schedule-failure-issue/action.yml @@ -16,18 +16,31 @@ name: "Schedule Failure Issue" description: >- Open (or update) a tracking issue when a scheduled workflow run fails or does not complete normally. + Issues are filed in the target repository (default rustfs/backlog), not + the repository the workflow runs in. Dedupes by workflow name: if an open issue titled - "[scheduled-failure] " already exists, the result is + "[scheduled-failure] " already exists there, the result is appended as a comment; otherwise a new issue is created. This is the single alerting mechanism for all scheduled pipelines (backlog#1149 ci-8). inputs: github-token: description: >- - Token with issues:write on the repository. Pass secrets.GITHUB_TOKEN - from a job that declares `permissions: issues: write` (scope the - permission to the alert job only, never workflow-wide). + Token with issues:write on the target repository. The workflow-scoped + secrets.GITHUB_TOKEN is scoped to the code repository and cannot open + issues in rustfs/backlog, so pass a PAT / GitHub App token stored as a + repository secret (BACKLOG_ISSUE_TOKEN). required: true + target-repository: + description: "Repository the failure-tracking issue is filed in." + required: false + default: rustfs/backlog + source-token: + description: >- + Token used to read the failed-job list from the reported run. Defaults + to the workflow's own token, which can read runs of its own repository. + required: false + default: ${{ github.token }} workflow-name: description: "Workflow name used for the issue title (the dedupe key)." required: false @@ -35,8 +48,8 @@ inputs: label: description: >- Label applied when a new issue is created. Must already exist in the - repository; if applying it fails, the issue is created without a label. - Set to an empty string to skip labeling. + target repository; if applying it fails, the issue is created without a + label. Set to an empty string to skip labeling. required: false default: "infrastructure" source-run-id: @@ -71,6 +84,8 @@ runs: shell: bash env: GH_TOKEN: ${{ inputs.github-token }} + SOURCE_TOKEN: ${{ inputs.source-token }} + TARGET_REPOSITORY: ${{ inputs.target-repository }} WORKFLOW_NAME: ${{ inputs.workflow-name }} ISSUE_LABEL: ${{ inputs.label }} SOURCE_RUN_ID: ${{ inputs.source-run-id }} @@ -86,8 +101,10 @@ runs: run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}" # Inspect the reported run attempt. It can be the current in-workflow - # failure or a completed run observed by the external watchdog. - failed_jobs="$(gh api \ + # failure or a completed run observed by the external watchdog. The + # run lives in the repository running the workflow, so authenticate + # with the source token, not the target-repository token. + failed_jobs="$(GH_TOKEN="${SOURCE_TOKEN}" gh api \ "repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}/attempts/${SOURCE_RUN_ATTEMPT}/jobs" \ --paginate \ --jq '.jobs[] @@ -109,6 +126,7 @@ runs: body="$(cat </dev/null + + artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}") + [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]] + [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]] + [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]] + [[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]] + [[ $(jq -r '.expired' <<<"$artifact") == false ]] + + - name: Download exact build artifact + uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7 + with: + artifact-ids: ${{ inputs.artifact_id }} + path: artifact + run-id: ${{ inputs.build_run_id }} + github-token: ${{ github.token }} + + - name: Extract exact RustFS binary + shell: bash + env: + BINARY_SHA256: ${{ inputs.binary_sha256 }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + short_sha=${SOURCE_SHA:0:7} + package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit) + [[ -n "$package" ]] + mkdir -p binary + unzip -qq "$package" rustfs -d binary + chmod +x binary/rustfs + printf '%s %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict + + - name: Run signed service job acceptance + shell: bash + env: + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref + npm --prefix connect-harness/web ci + connect-harness/web/node_modules/.bin/playwright install --with-deps chromium + make -C connect-harness e2e-connected-dispatch-check + RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ + CONNECT_E2E_PERFORMANCE_DRIVE_EVIDENCE="$GITHUB_WORKSPACE/performance-drive-service-job-evidence.json" \ + make -C connect-harness e2e-connected E2E_SCENARIO=performance-drive + + - name: Bind workflow and artifact provenance + if: ${{ always() && hashFiles('performance-drive-service-job-evidence.json') != '' }} + shell: bash + env: + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + jq \ + --arg workflowRunId "$GITHUB_RUN_ID" \ + --arg buildRunId "$BUILD_RUN_ID" \ + --arg artifactId "$ARTIFACT_ID" \ + --arg artifactDigest "$ARTIFACT_DIGEST" \ + --arg connectSha "$CONNECT_SHA" \ + --arg sourceSha "$SOURCE_SHA" \ + '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \ + performance-drive-service-job-evidence.json >performance-drive-service-job-evidence.bound.json + mv performance-drive-service-job-evidence.bound.json performance-drive-service-job-evidence.json + + - name: Upload acceptance evidence + if: ${{ always() && hashFiles('performance-drive-service-job-evidence.json') != '' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: connect-performance-drive-evidence-${{ github.run_id }} + path: | + performance-drive-service-job-evidence.json + retention-days: 14 + if-no-files-found: error diff --git a/.github/workflows/connect-performance-network-acceptance.yml b/.github/workflows/connect-performance-network-acceptance.yml new file mode 100644 index 000000000..0e660b5f0 --- /dev/null +++ b/.github/workflows/connect-performance-network-acceptance.yml @@ -0,0 +1,192 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Connect performance.network service-job acceptance + +on: + workflow_dispatch: + inputs: + build_run_id: + description: Main-branch Build and Release workflow run ID + required: true + type: string + artifact_id: + description: Linux x86_64 GNU artifact ID from that run + required: true + type: string + source_sha: + description: Exact 40-character source commit + required: true + type: string + artifact_digest: + description: GitHub artifact digest including sha256 prefix + required: true + type: string + binary_sha256: + description: Expected rustfs binary SHA-256 + required: true + type: string + connect_sha: + description: Exact 40-character RustFS Connect harness commit + required: true + type: string + +permissions: + actions: read + contents: read + +jobs: + performance-network: + name: Verify native Linux x86_64 performance.network service job + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Checkout acceptance harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + + - name: Checkout exact RustFS source + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + path: rustfs-source + persist-credentials: false + ref: ${{ inputs.source_sha }} + + - name: Checkout exact Connect harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/connect + path: connect-harness + persist-credentials: false + ref: ${{ inputs.connect_sha }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + + - name: Set up Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 22.22.2 + cache: npm + cache-dependency-path: connect-harness/web/package-lock.json + + - name: Verify source run and artifact identity + shell: bash + env: + GH_TOKEN: ${{ github.token }} + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + SOURCE_SHA: ${{ inputs.source_sha }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + run: | + set -euo pipefail + [[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]] + [[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]] + [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] + [[ "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]] + [[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]] + [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]] + [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]] + + run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}") + [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]] + [[ $(jq -r '.head_branch' <<<"$run") == main ]] + [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] + [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] + + expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") + jq -e --arg name "$expected_job" ' + [.[].jobs[] | select(.name == $name)] as $matches + | (($matches | length) == 1 and $matches[0].conclusion == "success") + ' <<<"$jobs" >/dev/null + + artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}") + [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]] + [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]] + [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]] + [[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]] + [[ $(jq -r '.expired' <<<"$artifact") == false ]] + + - name: Download exact build artifact + uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7 + with: + artifact-ids: ${{ inputs.artifact_id }} + path: artifact + run-id: ${{ inputs.build_run_id }} + github-token: ${{ github.token }} + + - name: Extract exact RustFS binary + shell: bash + env: + BINARY_SHA256: ${{ inputs.binary_sha256 }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + short_sha=${SOURCE_SHA:0:7} + package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit) + [[ -n "$package" ]] + mkdir -p binary + unzip -qq "$package" rustfs -d binary + chmod +x binary/rustfs + printf '%s %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict + + - name: Run signed service job acceptance + shell: bash + env: + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref + npm --prefix connect-harness/web ci + connect-harness/web/node_modules/.bin/playwright install --with-deps chromium + make -C connect-harness e2e-connected-dispatch-check + RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ + CONNECT_E2E_PERFORMANCE_NETWORK_EVIDENCE="$GITHUB_WORKSPACE/performance-network-service-job-evidence.json" \ + make -C connect-harness e2e-connected E2E_SCENARIO=performance-network + + - name: Bind workflow and artifact provenance + if: ${{ always() && hashFiles('performance-network-service-job-evidence.json') != '' }} + shell: bash + env: + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + jq \ + --arg workflowRunId "$GITHUB_RUN_ID" \ + --arg buildRunId "$BUILD_RUN_ID" \ + --arg artifactId "$ARTIFACT_ID" \ + --arg artifactDigest "$ARTIFACT_DIGEST" \ + --arg connectSha "$CONNECT_SHA" \ + --arg sourceSha "$SOURCE_SHA" \ + '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \ + performance-network-service-job-evidence.json >performance-network-service-job-evidence.bound.json + mv performance-network-service-job-evidence.bound.json performance-network-service-job-evidence.json + + - name: Upload acceptance evidence + if: ${{ always() && hashFiles('performance-network-service-job-evidence.json') != '' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: connect-performance-network-evidence-${{ github.run_id }} + path: | + performance-network-service-job-evidence.json + retention-days: 14 + if-no-files-found: error diff --git a/.github/workflows/connect-profile-cpu-acceptance.yml b/.github/workflows/connect-profile-cpu-acceptance.yml new file mode 100644 index 000000000..3c3889b2a --- /dev/null +++ b/.github/workflows/connect-profile-cpu-acceptance.yml @@ -0,0 +1,192 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Connect profile.cpu service-job acceptance + +on: + workflow_dispatch: + inputs: + build_run_id: + description: Main-branch Build and Release workflow run ID + required: true + type: string + artifact_id: + description: Linux x86_64 GNU artifact ID from that run + required: true + type: string + source_sha: + description: Exact 40-character source commit + required: true + type: string + artifact_digest: + description: GitHub artifact digest including sha256 prefix + required: true + type: string + binary_sha256: + description: Expected rustfs binary SHA-256 + required: true + type: string + connect_sha: + description: Exact 40-character RustFS Connect harness commit + required: true + type: string + +permissions: + actions: read + contents: read + +jobs: + profile-cpu: + name: Verify native Linux x86_64 profile.cpu service job + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Checkout acceptance harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + + - name: Checkout exact RustFS source + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + path: rustfs-source + persist-credentials: false + ref: ${{ inputs.source_sha }} + + - name: Checkout exact Connect harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/connect + path: connect-harness + persist-credentials: false + ref: ${{ inputs.connect_sha }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + + - name: Set up Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 22.22.2 + cache: npm + cache-dependency-path: connect-harness/web/package-lock.json + + - name: Verify source run and artifact identity + shell: bash + env: + GH_TOKEN: ${{ github.token }} + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + SOURCE_SHA: ${{ inputs.source_sha }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + run: | + set -euo pipefail + [[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]] + [[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]] + [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] + [[ "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]] + [[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]] + [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]] + [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]] + + run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}") + [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]] + [[ $(jq -r '.head_branch' <<<"$run") == main ]] + [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] + [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] + + expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") + jq -e --arg name "$expected_job" ' + [.[].jobs[] | select(.name == $name)] as $matches + | (($matches | length) == 1 and $matches[0].conclusion == "success") + ' <<<"$jobs" >/dev/null + + artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}") + [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]] + [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]] + [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]] + [[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]] + [[ $(jq -r '.expired' <<<"$artifact") == false ]] + + - name: Download exact build artifact + uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7 + with: + artifact-ids: ${{ inputs.artifact_id }} + path: artifact + run-id: ${{ inputs.build_run_id }} + github-token: ${{ github.token }} + + - name: Extract exact RustFS binary + shell: bash + env: + BINARY_SHA256: ${{ inputs.binary_sha256 }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + short_sha=${SOURCE_SHA:0:7} + package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit) + [[ -n "$package" ]] + mkdir -p binary + unzip -qq "$package" rustfs -d binary + chmod +x binary/rustfs + printf '%s %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict + + - name: Run signed service job acceptance + shell: bash + env: + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref + npm --prefix connect-harness/web ci + connect-harness/web/node_modules/.bin/playwright install --with-deps chromium + make -C connect-harness e2e-connected-dispatch-check + RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ + CONNECT_E2E_PROFILE_EVIDENCE="$GITHUB_WORKSPACE/profile-service-job-evidence.json" \ + make -C connect-harness e2e-connected E2E_SCENARIO=profile + + - name: Bind workflow and artifact provenance + if: ${{ always() && hashFiles('profile-service-job-evidence.json') != '' }} + shell: bash + env: + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + jq \ + --arg workflowRunId "$GITHUB_RUN_ID" \ + --arg buildRunId "$BUILD_RUN_ID" \ + --arg artifactId "$ARTIFACT_ID" \ + --arg artifactDigest "$ARTIFACT_DIGEST" \ + --arg connectSha "$CONNECT_SHA" \ + --arg sourceSha "$SOURCE_SHA" \ + '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \ + profile-service-job-evidence.json >profile-service-job-evidence.bound.json + mv profile-service-job-evidence.bound.json profile-service-job-evidence.json + + - name: Upload acceptance evidence + if: ${{ always() && hashFiles('profile-service-job-evidence.json') != '' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: connect-profile-cpu-evidence-${{ github.run_id }} + path: | + profile-service-job-evidence.json + retention-days: 14 + if-no-files-found: error diff --git a/.github/workflows/connect-profile-threads-acceptance.yml b/.github/workflows/connect-profile-threads-acceptance.yml new file mode 100644 index 000000000..966509b33 --- /dev/null +++ b/.github/workflows/connect-profile-threads-acceptance.yml @@ -0,0 +1,192 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Connect profile.threads service-job acceptance + +on: + workflow_dispatch: + inputs: + build_run_id: + description: Main-branch Build and Release workflow run ID + required: true + type: string + artifact_id: + description: Linux x86_64 GNU artifact ID from that run + required: true + type: string + source_sha: + description: Exact 40-character source commit + required: true + type: string + artifact_digest: + description: GitHub artifact digest including sha256 prefix + required: true + type: string + binary_sha256: + description: Expected rustfs binary SHA-256 + required: true + type: string + connect_sha: + description: Exact 40-character RustFS Connect harness commit + required: true + type: string + +permissions: + actions: read + contents: read + +jobs: + profile-threads: + name: Verify native Linux x86_64 profile.threads service job + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Checkout acceptance harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + + - name: Checkout exact RustFS source + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + path: rustfs-source + persist-credentials: false + ref: ${{ inputs.source_sha }} + + - name: Checkout exact Connect harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/connect + path: connect-harness + persist-credentials: false + ref: ${{ inputs.connect_sha }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + + - name: Set up Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 22.22.2 + cache: npm + cache-dependency-path: connect-harness/web/package-lock.json + + - name: Verify source run and artifact identity + shell: bash + env: + GH_TOKEN: ${{ github.token }} + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + SOURCE_SHA: ${{ inputs.source_sha }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + run: | + set -euo pipefail + [[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]] + [[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]] + [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] + [[ "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]] + [[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]] + [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]] + [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]] + + run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}") + [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]] + [[ $(jq -r '.head_branch' <<<"$run") == main ]] + [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] + [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] + + expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") + jq -e --arg name "$expected_job" ' + [.[].jobs[] | select(.name == $name)] as $matches + | (($matches | length) == 1 and $matches[0].conclusion == "success") + ' <<<"$jobs" >/dev/null + + artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}") + [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]] + [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]] + [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]] + [[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]] + [[ $(jq -r '.expired' <<<"$artifact") == false ]] + + - name: Download exact build artifact + uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7 + with: + artifact-ids: ${{ inputs.artifact_id }} + path: artifact + run-id: ${{ inputs.build_run_id }} + github-token: ${{ github.token }} + + - name: Extract exact RustFS binary + shell: bash + env: + BINARY_SHA256: ${{ inputs.binary_sha256 }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + short_sha=${SOURCE_SHA:0:7} + package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit) + [[ -n "$package" ]] + mkdir -p binary + unzip -qq "$package" rustfs -d binary + chmod +x binary/rustfs + printf '%s %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict + + - name: Run signed service job acceptance + shell: bash + env: + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref + npm --prefix connect-harness/web ci + connect-harness/web/node_modules/.bin/playwright install --with-deps chromium + make -C connect-harness e2e-connected-dispatch-check + RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ + CONNECT_E2E_PROFILE_THREADS_EVIDENCE="$GITHUB_WORKSPACE/profile-threads-service-job-evidence.json" \ + make -C connect-harness e2e-connected E2E_SCENARIO=profile-threads + + - name: Bind workflow and artifact provenance + if: ${{ always() && hashFiles('profile-threads-service-job-evidence.json') != '' }} + shell: bash + env: + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + jq \ + --arg workflowRunId "$GITHUB_RUN_ID" \ + --arg buildRunId "$BUILD_RUN_ID" \ + --arg artifactId "$ARTIFACT_ID" \ + --arg artifactDigest "$ARTIFACT_DIGEST" \ + --arg connectSha "$CONNECT_SHA" \ + --arg sourceSha "$SOURCE_SHA" \ + '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \ + profile-threads-service-job-evidence.json >profile-threads-service-job-evidence.bound.json + mv profile-threads-service-job-evidence.bound.json profile-threads-service-job-evidence.json + + - name: Upload acceptance evidence + if: ${{ always() && hashFiles('profile-threads-service-job-evidence.json') != '' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: connect-profile-threads-evidence-${{ github.run_id }} + path: | + profile-threads-service-job-evidence.json + retention-days: 14 + if-no-files-found: error diff --git a/.github/workflows/connect-top-api-acceptance.yml b/.github/workflows/connect-top-api-acceptance.yml new file mode 100644 index 000000000..4ec9bb8fa --- /dev/null +++ b/.github/workflows/connect-top-api-acceptance.yml @@ -0,0 +1,192 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Connect top.api service-job acceptance + +on: + workflow_dispatch: + inputs: + build_run_id: + description: Main-branch Build and Release workflow run ID + required: true + type: string + artifact_id: + description: Linux x86_64 GNU artifact ID from that run + required: true + type: string + source_sha: + description: Exact 40-character source commit + required: true + type: string + artifact_digest: + description: GitHub artifact digest including sha256 prefix + required: true + type: string + binary_sha256: + description: Expected rustfs binary SHA-256 + required: true + type: string + connect_sha: + description: Exact 40-character RustFS Connect harness commit + required: true + type: string + +permissions: + actions: read + contents: read + +jobs: + top-api: + name: Verify native Linux x86_64 top.api service job + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Checkout acceptance harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + + - name: Checkout exact RustFS source + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + path: rustfs-source + persist-credentials: false + ref: ${{ inputs.source_sha }} + + - name: Checkout exact Connect harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/connect + path: connect-harness + persist-credentials: false + ref: ${{ inputs.connect_sha }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + + - name: Set up Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 22.22.2 + cache: npm + cache-dependency-path: connect-harness/web/package-lock.json + + - name: Verify source run and artifact identity + shell: bash + env: + GH_TOKEN: ${{ github.token }} + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + SOURCE_SHA: ${{ inputs.source_sha }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + run: | + set -euo pipefail + [[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]] + [[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]] + [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] + [[ "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]] + [[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]] + [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]] + [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]] + + run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}") + [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]] + [[ $(jq -r '.head_branch' <<<"$run") == main ]] + [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] + [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] + + expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") + jq -e --arg name "$expected_job" ' + [.[].jobs[] | select(.name == $name)] as $matches + | (($matches | length) == 1 and $matches[0].conclusion == "success") + ' <<<"$jobs" >/dev/null + + artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}") + [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]] + [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]] + [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]] + [[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]] + [[ $(jq -r '.expired' <<<"$artifact") == false ]] + + - name: Download exact build artifact + uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7 + with: + artifact-ids: ${{ inputs.artifact_id }} + path: artifact + run-id: ${{ inputs.build_run_id }} + github-token: ${{ github.token }} + + - name: Extract exact RustFS binary + shell: bash + env: + BINARY_SHA256: ${{ inputs.binary_sha256 }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + short_sha=${SOURCE_SHA:0:7} + package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit) + [[ -n "$package" ]] + mkdir -p binary + unzip -qq "$package" rustfs -d binary + chmod +x binary/rustfs + printf '%s %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict + + - name: Run signed service job acceptance + shell: bash + env: + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref + npm --prefix connect-harness/web ci + connect-harness/web/node_modules/.bin/playwright install --with-deps chromium + make -C connect-harness e2e-connected-dispatch-check + RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ + CONNECT_E2E_TOP_API_EVIDENCE="$GITHUB_WORKSPACE/top-api-service-job-evidence.json" \ + make -C connect-harness e2e-connected E2E_SCENARIO=top-api + + - name: Bind workflow and artifact provenance + if: ${{ always() && hashFiles('top-api-service-job-evidence.json') != '' }} + shell: bash + env: + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + jq \ + --arg workflowRunId "$GITHUB_RUN_ID" \ + --arg buildRunId "$BUILD_RUN_ID" \ + --arg artifactId "$ARTIFACT_ID" \ + --arg artifactDigest "$ARTIFACT_DIGEST" \ + --arg connectSha "$CONNECT_SHA" \ + --arg sourceSha "$SOURCE_SHA" \ + '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \ + top-api-service-job-evidence.json >top-api-service-job-evidence.bound.json + mv top-api-service-job-evidence.bound.json top-api-service-job-evidence.json + + - name: Upload acceptance evidence + if: ${{ always() && hashFiles('top-api-service-job-evidence.json') != '' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: connect-top-api-evidence-${{ github.run_id }} + path: | + top-api-service-job-evidence.json + retention-days: 14 + if-no-files-found: error diff --git a/.github/workflows/connect-top-disk-acceptance.yml b/.github/workflows/connect-top-disk-acceptance.yml new file mode 100644 index 000000000..34a7a1ad2 --- /dev/null +++ b/.github/workflows/connect-top-disk-acceptance.yml @@ -0,0 +1,142 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Connect top.disk artifact acceptance + +on: + workflow_dispatch: + inputs: + build_run_id: + description: Successful main-branch Build and Release workflow run ID + required: true + type: string + artifact_id: + description: Linux x86_64 GNU artifact ID from that run + required: true + type: string + source_sha: + description: Exact 40-character source commit + required: true + type: string + artifact_digest: + description: GitHub artifact digest including sha256 prefix + required: true + type: string + binary_sha256: + description: Expected rustfs binary SHA-256 + required: true + type: string + +permissions: + actions: read + contents: read + +jobs: + top-disk: + name: Verify native Linux x86_64 top.disk artifact + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout acceptance harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + + - name: Verify source run and artifact identity + shell: bash + env: + GH_TOKEN: ${{ github.token }} + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + SOURCE_SHA: ${{ inputs.source_sha }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + run: | + set -euo pipefail + [[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]] + [[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]] + [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] + + run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}") + [[ $(jq -r '.conclusion' <<<"$run") == success ]] + [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]] + [[ $(jq -r '.head_branch' <<<"$run") == main ]] + [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] + [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] + + artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}") + [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]] + [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]] + [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]] + [[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]] + [[ $(jq -r '.expired' <<<"$artifact") == false ]] + + - name: Download exact build artifact + uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7 + with: + artifact-ids: ${{ inputs.artifact_id }} + path: artifact + run-id: ${{ inputs.build_run_id }} + github-token: ${{ github.token }} + + - name: Extract exact RustFS binary + shell: bash + env: + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + short_sha=${SOURCE_SHA:0:7} + package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit) + [[ -n "$package" ]] + mkdir -p binary + unzip -qq "$package" rustfs -d binary + chmod +x binary/rustfs + + - name: Run top.disk acceptance + shell: bash + env: + SOURCE_SHA: ${{ inputs.source_sha }} + BINARY_SHA256: ${{ inputs.binary_sha256 }} + run: | + set -euo pipefail + scripts/ci/check_connect_top_disk_artifact.sh \ + binary/rustfs "$SOURCE_SHA" "$BINARY_SHA256" \ + top-disk-runtime-evidence.json top-disk.zip + + - name: Bind workflow and artifact provenance + shell: bash + env: + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + run: | + set -euo pipefail + jq \ + --arg workflowRunId "$GITHUB_RUN_ID" \ + --arg buildRunId "$BUILD_RUN_ID" \ + --arg artifactId "$ARTIFACT_ID" \ + --arg artifactDigest "$ARTIFACT_DIGEST" \ + '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest}' \ + top-disk-runtime-evidence.json >top-disk-runtime-evidence.bound.json + mv top-disk-runtime-evidence.bound.json top-disk-runtime-evidence.json + + - name: Upload acceptance evidence + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: connect-top-disk-evidence-${{ github.run_id }} + path: | + top-disk-runtime-evidence.json + top-disk.zip + retention-days: 14 + if-no-files-found: error diff --git a/.github/workflows/connect-top-locks-acceptance.yml b/.github/workflows/connect-top-locks-acceptance.yml new file mode 100644 index 000000000..ff3e34911 --- /dev/null +++ b/.github/workflows/connect-top-locks-acceptance.yml @@ -0,0 +1,192 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Connect top.locks service-job acceptance + +on: + workflow_dispatch: + inputs: + build_run_id: + description: Main-branch Build and Release workflow run ID + required: true + type: string + artifact_id: + description: Linux x86_64 GNU artifact ID from that run + required: true + type: string + source_sha: + description: Exact 40-character source commit + required: true + type: string + artifact_digest: + description: GitHub artifact digest including sha256 prefix + required: true + type: string + binary_sha256: + description: Expected rustfs binary SHA-256 + required: true + type: string + connect_sha: + description: Exact 40-character RustFS Connect harness commit + required: true + type: string + +permissions: + actions: read + contents: read + +jobs: + top-locks: + name: Verify native Linux x86_64 top.locks service job + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Checkout acceptance harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + + - name: Checkout exact RustFS source + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + path: rustfs-source + persist-credentials: false + ref: ${{ inputs.source_sha }} + + - name: Checkout exact Connect harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/connect + path: connect-harness + persist-credentials: false + ref: ${{ inputs.connect_sha }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + + - name: Set up Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 22.22.2 + cache: npm + cache-dependency-path: connect-harness/web/package-lock.json + + - name: Verify source run and artifact identity + shell: bash + env: + GH_TOKEN: ${{ github.token }} + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + SOURCE_SHA: ${{ inputs.source_sha }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + run: | + set -euo pipefail + [[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]] + [[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]] + [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] + [[ "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]] + [[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]] + [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]] + [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]] + + run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}") + [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]] + [[ $(jq -r '.head_branch' <<<"$run") == main ]] + [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] + [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] + + expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") + jq -e --arg name "$expected_job" ' + [.[].jobs[] | select(.name == $name)] as $matches + | (($matches | length) == 1 and $matches[0].conclusion == "success") + ' <<<"$jobs" >/dev/null + + artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}") + [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]] + [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]] + [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]] + [[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]] + [[ $(jq -r '.expired' <<<"$artifact") == false ]] + + - name: Download exact build artifact + uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7 + with: + artifact-ids: ${{ inputs.artifact_id }} + path: artifact + run-id: ${{ inputs.build_run_id }} + github-token: ${{ github.token }} + + - name: Extract exact RustFS binary + shell: bash + env: + BINARY_SHA256: ${{ inputs.binary_sha256 }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + short_sha=${SOURCE_SHA:0:7} + package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit) + [[ -n "$package" ]] + mkdir -p binary + unzip -qq "$package" rustfs -d binary + chmod +x binary/rustfs + printf '%s %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict + + - name: Run signed service job acceptance + shell: bash + env: + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref + npm --prefix connect-harness/web ci + connect-harness/web/node_modules/.bin/playwright install --with-deps chromium + make -C connect-harness e2e-connected-dispatch-check + RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ + CONNECT_E2E_TOP_LOCKS_EVIDENCE="$GITHUB_WORKSPACE/top-locks-service-job-evidence.json" \ + make -C connect-harness e2e-connected E2E_SCENARIO=top-locks + + - name: Bind workflow and artifact provenance + if: ${{ always() && hashFiles('top-locks-service-job-evidence.json') != '' }} + shell: bash + env: + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + jq \ + --arg workflowRunId "$GITHUB_RUN_ID" \ + --arg buildRunId "$BUILD_RUN_ID" \ + --arg artifactId "$ARTIFACT_ID" \ + --arg artifactDigest "$ARTIFACT_DIGEST" \ + --arg connectSha "$CONNECT_SHA" \ + --arg sourceSha "$SOURCE_SHA" \ + '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \ + top-locks-service-job-evidence.json >top-locks-service-job-evidence.bound.json + mv top-locks-service-job-evidence.bound.json top-locks-service-job-evidence.json + + - name: Upload acceptance evidence + if: ${{ always() && hashFiles('top-locks-service-job-evidence.json') != '' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: connect-top-locks-evidence-${{ github.run_id }} + path: | + top-locks-service-job-evidence.json + retention-days: 14 + if-no-files-found: error diff --git a/.github/workflows/connect-top-rpc-acceptance.yml b/.github/workflows/connect-top-rpc-acceptance.yml new file mode 100644 index 000000000..a3097ddb0 --- /dev/null +++ b/.github/workflows/connect-top-rpc-acceptance.yml @@ -0,0 +1,192 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Connect top.rpc service-job acceptance + +on: + workflow_dispatch: + inputs: + build_run_id: + description: Main-branch Build and Release workflow run ID + required: true + type: string + artifact_id: + description: Linux x86_64 GNU artifact ID from that run + required: true + type: string + source_sha: + description: Exact 40-character source commit + required: true + type: string + artifact_digest: + description: GitHub artifact digest including sha256 prefix + required: true + type: string + binary_sha256: + description: Expected rustfs binary SHA-256 + required: true + type: string + connect_sha: + description: Exact 40-character RustFS Connect harness commit + required: true + type: string + +permissions: + actions: read + contents: read + +jobs: + top-rpc: + name: Verify native Linux x86_64 top.rpc service job + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Checkout acceptance harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + + - name: Checkout exact RustFS source + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + path: rustfs-source + persist-credentials: false + ref: ${{ inputs.source_sha }} + + - name: Checkout exact Connect harness + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/connect + path: connect-harness + persist-credentials: false + ref: ${{ inputs.connect_sha }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + + - name: Set up Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 22.22.2 + cache: npm + cache-dependency-path: connect-harness/web/package-lock.json + + - name: Verify source run and artifact identity + shell: bash + env: + GH_TOKEN: ${{ github.token }} + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + SOURCE_SHA: ${{ inputs.source_sha }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + run: | + set -euo pipefail + [[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]] + [[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]] + [[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]] + [[ "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]] + [[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]] + [[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]] + [[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]] + [[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]] + + run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}") + [[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]] + [[ $(jq -r '.head_branch' <<<"$run") == main ]] + [[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]] + [[ $(jq -r '.name' <<<"$run") == "Build and Release" ]] + + expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)' + jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100") + jq -e --arg name "$expected_job" ' + [.[].jobs[] | select(.name == $name)] as $matches + | (($matches | length) == 1 and $matches[0].conclusion == "success") + ' <<<"$jobs" >/dev/null + + artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}") + [[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]] + [[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]] + [[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]] + [[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]] + [[ $(jq -r '.expired' <<<"$artifact") == false ]] + + - name: Download exact build artifact + uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7 + with: + artifact-ids: ${{ inputs.artifact_id }} + path: artifact + run-id: ${{ inputs.build_run_id }} + github-token: ${{ github.token }} + + - name: Extract exact RustFS binary + shell: bash + env: + BINARY_SHA256: ${{ inputs.binary_sha256 }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + short_sha=${SOURCE_SHA:0:7} + package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit) + [[ -n "$package" ]] + mkdir -p binary + unzip -qq "$package" rustfs -d binary + chmod +x binary/rustfs + printf '%s %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict + + - name: Run signed service job acceptance + shell: bash + env: + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref + npm --prefix connect-harness/web ci + connect-harness/web/node_modules/.bin/playwright install --with-deps chromium + make -C connect-harness e2e-connected-dispatch-check + RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \ + RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \ + CONNECT_E2E_TOP_RPC_EVIDENCE="$GITHUB_WORKSPACE/top-rpc-service-job-evidence.json" \ + make -C connect-harness e2e-connected E2E_SCENARIO=top-rpc + + - name: Bind workflow and artifact provenance + if: ${{ always() && hashFiles('top-rpc-service-job-evidence.json') != '' }} + shell: bash + env: + BUILD_RUN_ID: ${{ inputs.build_run_id }} + ARTIFACT_ID: ${{ inputs.artifact_id }} + ARTIFACT_DIGEST: ${{ inputs.artifact_digest }} + CONNECT_SHA: ${{ inputs.connect_sha }} + SOURCE_SHA: ${{ inputs.source_sha }} + run: | + set -euo pipefail + jq \ + --arg workflowRunId "$GITHUB_RUN_ID" \ + --arg buildRunId "$BUILD_RUN_ID" \ + --arg artifactId "$ARTIFACT_ID" \ + --arg artifactDigest "$ARTIFACT_DIGEST" \ + --arg connectSha "$CONNECT_SHA" \ + --arg sourceSha "$SOURCE_SHA" \ + '. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \ + top-rpc-service-job-evidence.json >top-rpc-service-job-evidence.bound.json + mv top-rpc-service-job-evidence.bound.json top-rpc-service-job-evidence.json + + - name: Upload acceptance evidence + if: ${{ always() && hashFiles('top-rpc-service-job-evidence.json') != '' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: connect-top-rpc-evidence-${{ github.run_id }} + path: | + top-rpc-service-job-evidence.json + retention-days: 14 + if-no-files-found: error diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 8a843fdd5..0b38932e4 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -140,4 +140,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/e2e-distributed.yml b/.github/workflows/e2e-distributed.yml index 64289cad7..8b58fca17 100644 --- a/.github/workflows/e2e-distributed.yml +++ b/.github/workflows/e2e-distributed.yml @@ -212,4 +212,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/e2e-replication-nightly.yml b/.github/workflows/e2e-replication-nightly.yml index d08a7c0a8..39155aac2 100644 --- a/.github/workflows/e2e-replication-nightly.yml +++ b/.github/workflows/e2e-replication-nightly.yml @@ -242,4 +242,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/e2e-s3tests.yml b/.github/workflows/e2e-s3tests.yml index 7e24d4e90..30eba6d68 100644 --- a/.github/workflows/e2e-s3tests.yml +++ b/.github/workflows/e2e-s3tests.yml @@ -460,4 +460,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/functional-chain-health.yml b/.github/workflows/functional-chain-health.yml new file mode 100644 index 000000000..973327090 --- /dev/null +++ b/.github/workflows/functional-chain-health.yml @@ -0,0 +1,31 @@ +name: Functional chain health +on: + schedule: + - cron: '23 * * * *' + workflow_dispatch: +permissions: + contents: read + actions: read +concurrency: + group: functional-chain-health + cancel-in-progress: false +jobs: + collect: + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Collect and publish verified chain health + env: + GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} + NIGHTLY_SOURCE_REF: ${{ vars.NIGHTLY_BRANCH || 'main' }} + run: python3 scripts/functional_chain_health.py --source-ref "${NIGHTLY_SOURCE_REF}" --publish --output "${RUNNER_TEMP}/chain-health.json" + - name: Retain health observation + if: always() + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-health-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-health.json + if-no-files-found: error diff --git a/.github/workflows/fuzz.yml b/.github/workflows/fuzz.yml index 395dc1848..a5956c6e8 100644 --- a/.github/workflows/fuzz.yml +++ b/.github/workflows/fuzz.yml @@ -252,4 +252,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/minio-interop.yml b/.github/workflows/minio-interop.yml index 1658fe2cb..ffd9c60a4 100644 --- a/.github/workflows/minio-interop.yml +++ b/.github/workflows/minio-interop.yml @@ -143,4 +143,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/mint.yml b/.github/workflows/mint.yml index 425e13609..b0b5a9af7 100644 --- a/.github/workflows/mint.yml +++ b/.github/workflows/mint.yml @@ -270,4 +270,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/nightly-gnu.yml b/.github/workflows/nightly-gnu.yml index 4809d3b42..d283c31f0 100644 --- a/.github/workflows/nightly-gnu.yml +++ b/.github/workflows/nightly-gnu.yml @@ -543,4 +543,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/oidc-keycloak.yml b/.github/workflows/oidc-keycloak.yml index 31a57efbc..580a22f3c 100644 --- a/.github/workflows/oidc-keycloak.yml +++ b/.github/workflows/oidc-keycloak.yml @@ -107,4 +107,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/on-demand-migration-interop.yml b/.github/workflows/on-demand-migration-interop.yml index 60dbe1cad..d2971e441 100644 --- a/.github/workflows/on-demand-migration-interop.yml +++ b/.github/workflows/on-demand-migration-interop.yml @@ -312,4 +312,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/performance-ab.yml b/.github/workflows/performance-ab.yml index 61a8eed41..af82dd382 100644 --- a/.github/workflows/performance-ab.yml +++ b/.github/workflows/performance-ab.yml @@ -378,4 +378,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/runner-hygiene.yml b/.github/workflows/runner-hygiene.yml index bdebf2c77..8b642bd74 100644 --- a/.github/workflows/runner-hygiene.yml +++ b/.github/workflows/runner-hygiene.yml @@ -78,4 +78,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/rustfs-fault-tolerance-test.yml b/.github/workflows/rustfs-fault-tolerance-test.yml index 9bdf3677f..def607650 100644 --- a/.github/workflows/rustfs-fault-tolerance-test.yml +++ b/.github/workflows/rustfs-fault-tolerance-test.yml @@ -20,6 +20,12 @@ name: RustFS Fault-Tolerance Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: package_url: @@ -49,7 +55,7 @@ permissions: # The suite stops services and hides drive dirs on the shared fleet; only one # functional suite may touch the environment at a time. concurrency: - group: rustfs-shared-functional-tests + group: rustfs-shared-functional-tests-v2 cancel-in-progress: false defaults: @@ -68,8 +74,13 @@ jobs: fault-tolerance-test: runs-on: smoke-testing timeout-minutes: 480 - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} steps: + - name: Checkout repository (for report parser) + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Initialize functional evidence id: evidence run: | @@ -84,25 +95,22 @@ jobs: printf 'EVIDENCE_DIR=%s/evidence\n' "${FUNCTIONAL_ARTIFACTS_DIR}" } >> "${GITHUB_ENV}" - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + # Pin the private suite checkout when the chain supplies a candidate. + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Show environment run: | @@ -116,19 +124,27 @@ jobs: run: | ./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}" + - name: Stage verified candidate package + id: chain_package + if: ${{ inputs.chain_manifest != '' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/prepare_functional_package.py prepare + - name: Run fault-tolerance scenarios (A, B, C, C2) id: test - # Case failures keep the run green: the report and the backlog - # issue manager carry the product signal. + # Standalone case failures are reported by the backlog manager. + # Chain evidence separately requires complete passing cases. continue-on-error: true run: | - ARGS=(--all -y --package-url "${{ inputs.package_url || env.RUSTFS_NIGHTLY_PACKAGE_URL }}" --log-file "${LOG_FILE}") + ARGS=(--all -y --package-url "${{ steps.chain_package.outputs.package_url || inputs.package_url || env.RUSTFS_NIGHTLY_PACKAGE_URL }}" --log-file "${LOG_FILE}") if [ "${{ inputs.strict }}" = "true" ]; then ARGS+=(--strict) fi ./auto-testing/rustfs-fault-tolerance-test.sh "${ARGS[@]}" - name: Generate report + id: chain_report if: ${{ always() && steps.evidence.outcome == 'success' }} run: | set -euo pipefail @@ -204,7 +220,7 @@ jobs: --outcome "${{ steps.test.outcome }}" \ --report-file "${REPORT_FILE}" \ --log "${LOG_FILE}" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -282,3 +298,30 @@ jobs: echo "RustFS fault-tolerance test failed" echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}" echo "See the uploaded log artifact and FT-CASE lines for details." + + - name: Remove verified candidate package + if: ${{ always() && inputs.chain_manifest != '' && steps.chain.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/prepare_functional_package.py cleanup + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite fault-tolerance + --report "${FUNCTIONAL_ARTIFACTS_DIR}/suite.log" + --output "${RUNNER_TEMP}/chain-fault-tolerance-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/fault-tolerance.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}/fault-tolerance.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-functional-chain.yml b/.github/workflows/rustfs-functional-chain.yml index 3c6383260..8c951ac26 100644 --- a/.github/workflows/rustfs-functional-chain.yml +++ b/.github/workflows/rustfs-functional-chain.yml @@ -12,50 +12,207 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Functional chain driver: runs the ten functional suites in a fixed order -# (upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security -> -# replication -> fault-tolerance -> performance). Each suite attempts the next handoff even -# when its tests fail. -# -# Each suite workflow can still be dispatched standalone (workflow_dispatch); -# only chain-triggered runs forward to the next suite via repository_dispatch, -# so a standalone run never drags the rest of the chain behind it. -# -# Why not workflow_run chaining: GitHub does not guarantee delivery of -# workflow_run events (they are fire-and-forget), and the head-SHA filter made -# newly added suites (storage) unable to trigger at all. Explicit -# repository_dispatch handoffs are verifiable and re-drivable. - +# Reusable workflows run from this driver's commit in one Actions run. Each +# suite still runs after an earlier suite fails; the final job requires all twelve. name: RustFS Functional Chain on: workflow_dispatch: + inputs: + build_run_id: + description: Successful nightly build run on main + type: string + required: true + build_run_attempt: + description: Exact successful build attempt + type: string + required: true workflow_run: - # Entry point: start the chain after the nightly build completes. The - # build's own conclusion does not gate the chain; each suite reports its - # own result to rustfs/backlog and the dashboard. workflows: ["Nightly GNU Build"] types: [completed] permissions: contents: read + actions: read + +concurrency: + group: rustfs-functional-chain-runs + cancel-in-progress: false jobs: - start-chain: - name: Start functional chain (upgrade first) + prepare: + if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule' }} runs-on: ubuntu-latest timeout-minutes: 10 - if: ${{ github.event_name == 'workflow_dispatch' || (github.event_name == 'workflow_run' && github.event.workflow_run.event == 'schedule') }} + outputs: + manifest: ${{ steps.candidate.outputs.manifest }} steps: - - name: Dispatch first suite (upgrade) + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Resolve published candidate + id: candidate + env: + GH_TOKEN: ${{ github.token }} + BUILD_RUN_ID: ${{ inputs.build_run_id }} + BUILD_RUN_ATTEMPT: ${{ inputs.build_run_attempt }} + CHAIN_OUTPUT: ${{ runner.temp }}/chain-candidate.json + run: python3 scripts/resolve_functional_candidate.py + - name: Retain candidate identity + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-candidate-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-candidate.json + if-no-files-found: error + + - name: Check shared functional runners before scheduling suites env: GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - if [ -z "${GH_TOKEN:-}" ]; then - echo "PF_TESTING_GH_TOKEN is not configured; cannot start the functional chain" >&2 - exit 1 - fi - gh api --method POST repos/rustfs/rustfs/dispatches \ - -f event_type='rustfs-chain-upgrade' \ - -F 'client_payload[from_suite]=nightly-build' + run: python3 scripts/check_functional_runners.py smoke-testing pf-testing + + upgrade: + needs: [prepare] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-upgrade-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + s3: + needs: [prepare, upgrade] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-s3-compat-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + kms: + needs: [prepare, s3] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-kms-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + tier: + needs: [prepare, kms] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-tier-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + storage: + needs: [prepare, tier] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-storage-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + heal: + needs: [prepare, storage] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-heal-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + pool: + needs: [prepare, heal] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-pool-expand-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + security: + needs: [prepare, pool] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-security-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + replication: + needs: [prepare, security] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-replication-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + fault-tolerance: + needs: [prepare, replication] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-fault-tolerance-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + table: + needs: [prepare, fault-tolerance] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-table-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + performance: + needs: [prepare, table] + if: ${{ always() && needs.prepare.result == 'success' }} + uses: ./.github/workflows/rustfs-performance-test.yml + with: + chain_manifest: ${{ needs.prepare.outputs.manifest }} + secrets: inherit + + complete-chain: + needs: [prepare, upgrade, s3, kms, tier, storage, heal, pool, security, replication, fault-tolerance, table, performance] + if: ${{ always() && needs.prepare.result != 'skipped' }} + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Download suite evidence + id: download + continue-on-error: true + uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7 + with: + pattern: functional-chain-*-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-evidence + merge-multiple: true + - name: Summarize every suite including failures and missing evidence + if: always() + env: + CHAIN_MANIFEST: ${{ needs.prepare.outputs.manifest }} + CHAIN_NEEDS: ${{ toJSON(needs) }} + run: >- + python3 scripts/functional_chain_evidence.py summarize + --directory "${RUNNER_TEMP}/chain-evidence" + --output "${RUNNER_TEMP}/chain-report.json" + - name: Retain chain report regardless of test verdict + if: always() + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-report-${{ github.run_id }}-${{ github.run_attempt }} + path: | + ${{ runner.temp }}/chain-report.json + ${{ runner.temp }}/chain-report.md + if-no-files-found: error + - name: Verify every required suite + if: always() + env: + CHAIN_MANIFEST: ${{ needs.prepare.outputs.manifest }} + CHAIN_NEEDS: ${{ toJSON(needs) }} + run: >- + python3 scripts/functional_chain_evidence.py aggregate + --directory "${RUNNER_TEMP}/chain-evidence" + --output "${RUNNER_TEMP}/chain-complete.json" + - name: Upload complete-chain evidence + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-complete-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-complete.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-heal-test.yml b/.github/workflows/rustfs-heal-test.yml index 49d9382f2..4736425f6 100644 --- a/.github/workflows/rustfs-heal-test.yml +++ b/.github/workflows/rustfs-heal-test.yml @@ -1,6 +1,12 @@ name: RustFS Heal Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: package_url: @@ -57,8 +63,13 @@ jobs: timeout-minutes: 480 # Standalone manual run, or one link of the nightly functional chain # (storage -> heal -> pool). Pool expansion no longer re-runs heal. - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} steps: + - name: Checkout chain tooling + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Initialize functional evidence id: evidence run: | @@ -74,25 +85,21 @@ jobs: printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}" } >> "${GITHUB_ENV}" - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Show environment run: | @@ -155,6 +162,7 @@ jobs: --log-file "${LOG_FILE}" - name: Generate report + id: chain_report if: ${{ always() && steps.evidence.outcome == 'success' }} run: | set -euo pipefail @@ -340,7 +348,7 @@ jobs: --outcome "${{ steps.test.outcome }}" \ --report-file "${REPORT_FILE}" \ --log "${LOG_FILE}" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -434,3 +442,24 @@ jobs: echo "RustFS heal test failed" echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}" echo "See the uploaded log artifact for details." + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite heal + --report "${FUNCTIONAL_ARTIFACTS_DIR}/steps.md" + --output "${RUNNER_TEMP}/chain-heal-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/heal.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-heal-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-heal-${{ github.run_id }}-${{ github.run_attempt }}/heal.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-kms-test.yml b/.github/workflows/rustfs-kms-test.yml index 51145b257..6ad4c59c6 100644 --- a/.github/workflows/rustfs-kms-test.yml +++ b/.github/workflows/rustfs-kms-test.yml @@ -1,11 +1,18 @@ name: RustFS KMS Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: rustfs_version: - description: 'RustFS release tag to test (leave empty to use the latest nightly deb)' + description: 'RustFS release tag to test. Leave empty for nightly.' required: false + default: '' package_url: description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.' required: false @@ -49,7 +56,7 @@ jobs: kms-test: runs-on: smoke-testing timeout-minutes: 420 - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} steps: - name: Checkout repository (for report parser) uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 @@ -70,25 +77,21 @@ jobs: printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}" } >> "${GITHUB_ENV}" - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Show environment run: | @@ -158,6 +161,7 @@ jobs: ./auto-testing/rustfs-kms-test.sh "${ARGS[@]}" - name: Generate report + id: chain_report if: ${{ always() && steps.evidence.outcome == 'success' }} run: | set -euo pipefail @@ -295,7 +299,7 @@ jobs: --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" \ --report-file "${REPORT_FILE}" \ --log "${LOG_FILE}" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -387,3 +391,24 @@ jobs: run: | echo "RustFS KMS suite failed" echo "See the uploaded report and log artifacts for details." + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite kms + --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" + --output "${RUNNER_TEMP}/chain-kms-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/kms.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-kms-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-kms-${{ github.run_id }}-${{ github.run_attempt }}/kms.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-performance-test.yml b/.github/workflows/rustfs-performance-test.yml index e2efad1a9..e43794b5e 100644 --- a/.github/workflows/rustfs-performance-test.yml +++ b/.github/workflows/rustfs-performance-test.yml @@ -1,6 +1,12 @@ name: RustFS Performance Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: package_url: @@ -79,7 +85,21 @@ env: PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} jobs: + runner-check: + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Check performance runner before entering its queue + env: + GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} + run: python3 scripts/check_functional_runners.py pf-testing + performance-test: + needs: runner-check runs-on: pf-testing timeout-minutes: 900 env: @@ -90,8 +110,13 @@ jobs: RUSTFS_WARP_CONCURRENCY: ${{ inputs.warp_concurrency || '64' }} # Run on manual dispatch, or when the nightly build completed successfully. # Skipped when nightly failed. - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} steps: + - name: Checkout chain tooling + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Initialize functional evidence id: evidence run: | @@ -108,25 +133,21 @@ jobs: printf 'VERSION_FILE=%s/version.txt\n' "${FUNCTIONAL_ARTIFACTS_DIR}" } >> "${GITHUB_ENV}" - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Show environment run: | @@ -192,6 +213,7 @@ jobs: } > "${VERSION_FILE}" - name: Upload report to dashboard (reports/YYYY-MM-DD.md) + id: chain_report if: ${{ steps.benchmark.conclusion == 'success' }} env: GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }} @@ -272,7 +294,7 @@ jobs: --outcome "${{ steps.benchmark.outcome }}" \ --report-file "${REPORT_FILE}" \ --log "${LOG_FILE}" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -308,3 +330,24 @@ jobs: echo "RustFS performance test failed" echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}" echo "See the uploaded log artifact for details." + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.benchmark.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite performance + --report "${RUSTFS_RESULT_DIR}/summary.tsv" + --output "${RUNNER_TEMP}/chain-performance-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/performance.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-performance-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-performance-${{ github.run_id }}-${{ github.run_attempt }}/performance.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-pool-expand-test.yml b/.github/workflows/rustfs-pool-expand-test.yml index fcd305dda..ed6b819b1 100644 --- a/.github/workflows/rustfs-pool-expand-test.yml +++ b/.github/workflows/rustfs-pool-expand-test.yml @@ -1,6 +1,12 @@ name: RustFS Pool Expansion Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: rustfs_version: @@ -63,7 +69,7 @@ env: RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }} RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }} RUSTFS_API_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }} - RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }} + RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES || 'vm000 vm001 vm002' }} RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }} PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} # Package used by the nightly run (workflow_dispatch inputs are empty for @@ -77,7 +83,7 @@ jobs: name: Pool expansion / decommission test runs-on: smoke-testing timeout-minutes: 360 - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} env: RUSTFS_POOL_ADMIN_ENDPOINT: ${{ secrets.RUSTFS_POOL_ADMIN_ENDPOINT || vars.RUSTFS_POOL_ADMIN_ENDPOINT || 'http://rustfs-node1:9000' }} RUSTFS_POOL_PROXY_ENDPOINT: http://127.0.0.1:19000 @@ -85,27 +91,29 @@ jobs: RUSTFS_SHARED_PROXY_ENDPOINT: ${{ secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT }} RUSTFS_POOL_NODE_ENDPOINTS: ${{ secrets.RUSTFS_POOL_NODE_ENDPOINTS || vars.RUSTFS_POOL_NODE_ENDPOINTS || 'http://rustfs-node1:9000 http://rustfs-node2:9000 http://rustfs-node3:9000 http://rustfs-node4:9000' }} steps: - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + - name: Checkout chain tooling + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Initialize pool test artifacts + id: evidence run: | set -euo pipefail ARTIFACT_DIR="${RUNNER_TEMP}/rustfs-pool-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" @@ -120,6 +128,17 @@ jobs: warp --version || true df -h /data | tail -1 + - name: Validate pool topology before destructive cleanup + id: topology + run: | + set -euo pipefail + read -r -a NODES <<< "${RUSTFS_NODES}" + read -r -a ENDPOINTS <<< "${RUSTFS_POOL_NODE_ENDPOINTS}" + if [ "${#NODES[@]}" -lt 3 ] || [ "${#ENDPOINTS[@]}" -lt "${#NODES[@]}" ]; then + echo "Pool tests require at least three nodes and a direct endpoint for every node" >&2 + exit 1 + fi + - name: Cleanup environment (before) if: ${{ inputs.cleanup_before != 'false' }} run: | @@ -366,6 +385,7 @@ jobs: fi - name: Generate report + id: chain_report if: always() run: | set -euo pipefail @@ -614,7 +634,7 @@ jobs: --outcome "${{ steps.pool_test.outcome }}" \ --report-file "${POOL_ARTIFACT_DIR}/pool-report.md" \ --log "${POOL_ARTIFACT_DIR}/pool-test.log" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -641,7 +661,7 @@ jobs: ./auto-testing/rustfs_pool_nginx_stage.sh cleanup - name: Cleanup environment (after) - if: ${{ always() && inputs.cleanup_after != 'false' }} + if: ${{ always() && steps.topology.outcome == 'success' && inputs.cleanup_after != 'false' }} run: | set -euo pipefail read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}" @@ -714,3 +734,24 @@ jobs: echo "RustFS pool expansion test failed" echo "Package source: ${{ inputs.package_url || inputs.rustfs_version || 'nightly (R2 latest)' }}" echo "See the uploaded log artifact for details." + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.pool_test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite pool + --report "${POOL_ARTIFACT_DIR}/pool-steps.md" + --output "${RUNNER_TEMP}/chain-pool-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/pool.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-pool-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-pool-${{ github.run_id }}-${{ github.run_attempt }}/pool.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-replication-test.yml b/.github/workflows/rustfs-replication-test.yml index 90e7464d4..73f77d3b4 100644 --- a/.github/workflows/rustfs-replication-test.yml +++ b/.github/workflows/rustfs-replication-test.yml @@ -15,11 +15,18 @@ name: RustFS Replication Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: rustfs_version: - description: 'RustFS release tag to test (leave empty to use the latest nightly deb)' + description: 'RustFS release tag to test. Leave empty for nightly.' required: false + default: '' package_url: description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.' required: false @@ -61,7 +68,7 @@ jobs: replication-test: runs-on: smoke-testing timeout-minutes: 360 - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} steps: - name: Checkout repository (for report parser) uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 @@ -82,25 +89,21 @@ jobs: printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}" } >> "${GITHUB_ENV}" - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Show environment run: | @@ -155,6 +158,7 @@ jobs: ./auto-testing/rustfs-replication-test.sh "${ARGS[@]}" - name: Generate report + id: chain_report if: ${{ always() && steps.evidence.outcome == 'success' }} run: | set -euo pipefail @@ -303,7 +307,7 @@ jobs: --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" \ --report-file "${REPORT_FILE}" \ --log "${LOG_FILE}" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -392,3 +396,24 @@ jobs: echo "RustFS replication suite failed" echo "Package source: ${{ inputs.package_url || inputs.rustfs_version || 'nightly (R2 latest)' }}" echo "See the uploaded report and log artifacts for details." + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite replication + --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" + --output "${RUNNER_TEMP}/chain-replication-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/replication.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-replication-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-replication-${{ github.run_id }}-${{ github.run_attempt }}/replication.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-s3-compat-test.yml b/.github/workflows/rustfs-s3-compat-test.yml index 100397062..1540eaa35 100644 --- a/.github/workflows/rustfs-s3-compat-test.yml +++ b/.github/workflows/rustfs-s3-compat-test.yml @@ -1,11 +1,18 @@ name: RustFS S3 Compatibility Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: rustfs_version: - description: 'RustFS release tag to test (leave empty to use the latest nightly deb)' + description: 'RustFS release tag to test. Leave empty for nightly.' required: false + default: '' package_url: description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.' required: false @@ -37,7 +44,7 @@ jobs: s3-compat-test: runs-on: smoke-testing timeout-minutes: 360 - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} steps: - name: Checkout repository (for report parser) uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 @@ -58,25 +65,21 @@ jobs: printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}" } >> "${GITHUB_ENV}" - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Show environment run: | @@ -124,6 +127,7 @@ jobs: ./auto-testing/rustfs-s3-compat-test.sh "${ARGS[@]}" - name: Generate report + id: chain_report if: ${{ always() && steps.evidence.outcome == 'success' }} run: | set -euo pipefail @@ -272,7 +276,7 @@ jobs: --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" \ --report-file "${REPORT_FILE}" \ --log "${LOG_FILE}" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -364,3 +368,24 @@ jobs: run: | echo "RustFS S3 compatibility suite failed" echo "See the uploaded report and log artifacts for details." + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite s3 + --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" + --output "${RUNNER_TEMP}/chain-s3-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/s3.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-s3-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-s3-${{ github.run_id }}-${{ github.run_attempt }}/s3.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-security-test.yml b/.github/workflows/rustfs-security-test.yml index ee69934d0..67024f314 100644 --- a/.github/workflows/rustfs-security-test.yml +++ b/.github/workflows/rustfs-security-test.yml @@ -15,11 +15,18 @@ name: RustFS Security Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: rustfs_version: - description: 'RustFS release tag to test (leave empty to use the latest nightly deb)' + description: 'RustFS release tag to test. Leave empty for nightly.' required: false + default: '' package_url: description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.' required: false @@ -74,7 +81,7 @@ jobs: security-test: runs-on: smoke-testing timeout-minutes: 360 - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} steps: # Checkout the repository into its own subdirectory. Checking out at # the workspace root would wipe the auto-testing clone above (that is @@ -94,25 +101,21 @@ jobs: mkdir -- "${SECURITY_ARTIFACTS_DIR}" "${SECURITY_ARTIFACTS_DIR}-scratch" printf 'SECURITY_ARTIFACTS_DIR=%s\n' "${SECURITY_ARTIFACTS_DIR}" >> "${GITHUB_ENV}" - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Show environment run: | @@ -306,7 +309,7 @@ jobs: --outcome "${{ steps.test.outcome }}" \ --report-file "${SECURITY_ARTIFACTS_DIR}/report.md" \ --log "${SECURITY_ARTIFACTS_DIR}/suite.log" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -369,3 +372,24 @@ jobs: echo "RustFS security test failed" echo "Package source: ${{ inputs.package_url || 'nightly (R2 latest)' }}" echo "See the uploaded report and logs for details." + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite security + --report "${SECURITY_ARTIFACTS_DIR}/suite-report.md" + --output "${RUNNER_TEMP}/chain-security-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/security.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-security-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-security-${{ github.run_id }}-${{ github.run_attempt }}/security.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-storage-test.yml b/.github/workflows/rustfs-storage-test.yml index 7342d68ef..d06835f21 100644 --- a/.github/workflows/rustfs-storage-test.yml +++ b/.github/workflows/rustfs-storage-test.yml @@ -1,11 +1,18 @@ name: RustFS Storage Engine Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: rustfs_version: - description: 'RustFS release tag to test (leave empty to use the latest nightly deb)' + description: 'RustFS release tag to test. Leave empty for nightly.' required: false + default: '' package_url: description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.' required: false @@ -46,7 +53,7 @@ jobs: storage-test: runs-on: smoke-testing timeout-minutes: 360 - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} steps: - name: Checkout repository (for report parser) uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 @@ -67,25 +74,21 @@ jobs: printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}" } >> "${GITHUB_ENV}" - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Show environment run: | @@ -139,6 +142,7 @@ jobs: ./auto-testing/rustfs-storage-test.sh "${ARGS[@]}" - name: Generate report + id: chain_report if: ${{ always() && steps.evidence.outcome == 'success' }} run: | set -euo pipefail @@ -287,7 +291,7 @@ jobs: --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" \ --report-file "${REPORT_FILE}" \ --log "${LOG_FILE}" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -379,3 +383,24 @@ jobs: run: | echo "RustFS storage engine suite failed" echo "See the uploaded report and log artifacts for details." + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite storage + --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" + --output "${RUNNER_TEMP}/chain-storage-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/storage.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-storage-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-storage-${{ github.run_id }}-${{ github.run_attempt }}/storage.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-table-test.yml b/.github/workflows/rustfs-table-test.yml index a90ce7e0c..eb51811ed 100644 --- a/.github/workflows/rustfs-table-test.yml +++ b/.github/workflows/rustfs-table-test.yml @@ -1,6 +1,12 @@ name: RustFS Table Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: package_url: @@ -40,7 +46,7 @@ jobs: timeout-minutes: 90 steps: - name: Checkout repository (for report parser) - uses: actions/checkout@v4 + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 with: persist-credentials: false @@ -58,23 +64,21 @@ jobs: printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}" } >> "${GITHUB_ENV}" - - name: Checkout auto-testing scripts + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Show environment run: | @@ -101,15 +105,22 @@ jobs: ' || echo "pre-cleanup failed on ${node} (continuing)" done + - name: Stage verified candidate package + id: chain_package + if: ${{ inputs.chain_manifest != '' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/prepare_functional_package.py prepare + - name: Run table suite id: test - # Case failures keep the run green: the report and the backlog issue - # manager carry the product signal. + # Standalone case failures are reported by the backlog manager. + # Chain evidence separately requires complete passing cases. continue-on-error: true run: | set -euo pipefail chmod +x auto-testing/rustfs-table-test.sh - PACKAGE_URL='${{ inputs.package_url }}' + PACKAGE_URL='${{ steps.chain_package.outputs.package_url || inputs.package_url }}' RUSTFS_VERSION='${{ inputs.rustfs_version }}' SMOKE_PATH="${GITHUB_WORKSPACE:-$PWD}/scripts/table-catalog/pyiceberg_smoke.py" ARGS=(-y --smoke-script "${SMOKE_PATH}") @@ -126,6 +137,7 @@ jobs: ./auto-testing/rustfs-table-test.sh "${ARGS[@]}" --log-file "${LOG_FILE}" - name: Generate report + id: chain_report if: ${{ always() && steps.evidence.outcome == 'success' }} run: | set -euo pipefail @@ -247,7 +259,9 @@ jobs: python3 auto-testing/scripts/issue_manager.py handle \ --repo rustfs/backlog \ --suite table --category table --suite-label "S3 Tables" \ - --run-url "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" \ + --outcome '${{ steps.test.outcome }}' \ + --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" \ + --run-url "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -259,7 +273,7 @@ jobs: - name: Upload report and logs if: ${{ always() && steps.evidence.outcome == 'success' }} - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 with: name: rustfs-table-test-${{ github.run_id }}-${{ github.run_attempt }} path: | @@ -308,3 +322,30 @@ jobs: - name: Notify on failure if: failure() run: echo "RustFS table test workflow failed (harness/environment breakdown); see logs and report artifact." + + - name: Remove verified candidate package + if: ${{ always() && inputs.chain_manifest != '' && steps.chain.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/prepare_functional_package.py cleanup + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite table + --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" + --output "${RUNNER_TEMP}/chain-table-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/table.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-table-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-table-${{ github.run_id }}-${{ github.run_attempt }}/table.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-tier-test.yml b/.github/workflows/rustfs-tier-test.yml index 7b75b1fbf..6956c218c 100644 --- a/.github/workflows/rustfs-tier-test.yml +++ b/.github/workflows/rustfs-tier-test.yml @@ -1,11 +1,18 @@ name: RustFS Tier Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: rustfs_version: - description: 'RustFS release tag to test (leave empty to use the latest nightly deb)' + description: 'RustFS release tag to test. Leave empty for nightly.' required: false + default: '' package_url: description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.' required: false @@ -61,8 +68,13 @@ jobs: tier-test: runs-on: smoke-testing timeout-minutes: 420 - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} steps: + - name: Checkout chain tooling + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + persist-credentials: false + - name: Initialize run evidence directory id: evidence run: | @@ -75,25 +87,21 @@ jobs: test -d "${TIER_ARTIFACTS_DIR}" test ! -L "${TIER_ARTIFACTS_DIR}" - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Prepare pinned RustFS CLI id: rc @@ -300,6 +308,7 @@ jobs: mv "${TMP_FILE}" "${RESULT_FILE}" - name: Generate report + id: chain_report if: ${{ always() && steps.evidence.outcome == 'success' }} env: PACKAGE_URL_INPUT: ${{ inputs.package_url }} @@ -546,7 +555,7 @@ jobs: --report "${TIER_ARTIFACTS_DIR}/rustfs-tier-cases.md" \ --report-file "${TIER_ARTIFACTS_DIR}/rustfs-tier-report.md" \ --log "${TIER_ARTIFACTS_DIR}/rustfs-tier.log" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -608,3 +617,24 @@ jobs: run: | echo "RustFS tier suite failed" echo "See the uploaded report and log artifacts for details." + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite tier + --report "${TIER_ARTIFACTS_DIR}/rustfs-tier-cases.md" + --output "${RUNNER_TEMP}/chain-tier-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/tier.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-tier-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-tier-${{ github.run_id }}-${{ github.run_attempt }}/tier.json + if-no-files-found: error diff --git a/.github/workflows/rustfs-upgrade-test.yml b/.github/workflows/rustfs-upgrade-test.yml index 83d1eae47..c14ee3bf3 100644 --- a/.github/workflows/rustfs-upgrade-test.yml +++ b/.github/workflows/rustfs-upgrade-test.yml @@ -15,6 +15,12 @@ name: RustFS Upgrade Test on: + workflow_call: + inputs: + chain_manifest: + description: Verified candidate and chain attempt from the chain driver + type: string + required: true workflow_dispatch: inputs: from_version: @@ -80,7 +86,7 @@ jobs: upgrade-test: runs-on: smoke-testing timeout-minutes: 420 - if: ${{ github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} + if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }} steps: - name: Checkout repository (for report parser) uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 @@ -101,25 +107,21 @@ jobs: printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}" } >> "${GITHUB_ENV}" - # auto-testing is private: clone it with the dedicated PF token (not - # GITHUB_TOKEN) and retry transient GitHub/network failures. - - name: Checkout auto-testing scripts (with retry) + - name: Bind functional candidate + id: chain + if: ${{ inputs.chain_manifest != '' }} env: - GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }} - run: | - set -euo pipefail - rm -rf auto-testing - for attempt in 1 2 3 4 5; do - if gh repo clone rustfs/auto-testing auto-testing -- --depth 1 --quiet; then - echo "auto-testing cloned (attempt ${attempt})" - exit 0 - fi - rm -rf auto-testing - echo "clone attempt ${attempt} failed; retrying in $((attempt * 15))s" >&2 - sleep $((attempt * 15)) - done - echo "ERROR: unable to clone rustfs/auto-testing after 5 attempts" >&2 - exit 1 + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + run: python3 scripts/functional_chain_evidence.py consume + + - name: Checkout auto-testing scripts + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + with: + repository: rustfs/auto-testing + ref: ${{ steps.chain.outputs.testing_sha || 'main' }} + token: ${{ secrets.PF_TESTING_GH_TOKEN }} + path: auto-testing + persist-credentials: false - name: Show environment run: | @@ -221,6 +223,7 @@ jobs: ./auto-testing/rustfs-upgrade-test.sh "${ARGS[@]}" - name: Generate report + id: chain_report if: ${{ always() && steps.evidence.outcome == 'success' }} run: | set -euo pipefail @@ -380,7 +383,7 @@ jobs: --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" \ --report-file "${REPORT_FILE}" \ --log "${LOG_FILE}" \ - --run-url "${RUN_URL}" \ + --run-url "${RUN_URL}/attempts/${GITHUB_RUN_ATTEMPT}#summary" \ --run-id "${GITHUB_RUN_ID}" \ --attempt "${GITHUB_RUN_ATTEMPT}" \ --commit "${GITHUB_SHA}" \ @@ -476,3 +479,24 @@ jobs: echo "From: ${{ inputs.from_url || inputs.from_version || 'release (default)' }}" echo "To: ${{ inputs.to_url || inputs.to_version || 'nightly (R2 latest)' }}" echo "See the uploaded report and logs for details." + + - name: Record chain evidence + id: chain_record + if: ${{ always() && inputs.chain_manifest != '' && steps.evidence.outcome == 'success' }} + env: + CHAIN_MANIFEST: ${{ inputs.chain_manifest }} + CHAIN_JOB_STATUS: ${{ job.status }} + CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }} + CHAIN_REPORT_OUTCOME: ${{ steps.chain_report.outcome }} + run: >- + python3 scripts/functional_chain_evidence.py record --suite upgrade + --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md" + --output "${RUNNER_TEMP}/chain-upgrade-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/upgrade.json" + + - name: Upload chain evidence + if: ${{ always() && steps.chain_record.outputs.written == 'true' }} + uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6 + with: + name: functional-chain-upgrade-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/chain-upgrade-${{ github.run_id }}-${{ github.run_attempt }}/upgrade.json + if-no-files-found: error diff --git a/.github/workflows/schedule-failure-alert-drill.yml b/.github/workflows/schedule-failure-alert-drill.yml index d9380aca0..8224a92dd 100644 --- a/.github/workflows/schedule-failure-alert-drill.yml +++ b/.github/workflows/schedule-failure-alert-drill.yml @@ -19,7 +19,9 @@ # 'failure')` wiring used by the real consumers (e2e-s3tests, mint, fuzz, # performance-ab). Dispatch it twice to verify both the issue-creation and # the dedupe-comment paths, then close the resulting -# "[scheduled-failure] Schedule Failure Alert Drill" issue. +# "[scheduled-failure] Schedule Failure Alert Drill" issue in rustfs/backlog +# (the default target repository). Requires the BACKLOG_ISSUE_TOKEN secret +# (a token with issues:write on rustfs/backlog). # # The run itself is expected to end red (the forced failure); only the # alert-on-failure job result matters. @@ -68,4 +70,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/.github/workflows/scheduled-validation-freshness.yml b/.github/workflows/scheduled-validation-freshness.yml index 69ddc844c..964ed8430 100644 --- a/.github/workflows/scheduled-validation-freshness.yml +++ b/.github/workflows/scheduled-validation-freshness.yml @@ -54,5 +54,5 @@ jobs: if: failure() uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} details-file: ${{ runner.temp }}/scheduled-validation-freshness.md diff --git a/.github/workflows/scheduled-validation-watchdog.yml b/.github/workflows/scheduled-validation-watchdog.yml index 154d541d3..3f9facbc6 100644 --- a/.github/workflows/scheduled-validation-watchdog.yml +++ b/.github/workflows/scheduled-validation-watchdog.yml @@ -55,7 +55,7 @@ jobs: - name: Open or update incomplete-run issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} workflow-name: ${{ github.event.workflow_run.name }} source-run-id: ${{ github.event.workflow_run.id }} source-run-attempt: ${{ github.event.workflow_run.run_attempt }} diff --git a/.github/workflows/targets-integration.yml b/.github/workflows/targets-integration.yml index 0b668896b..5b73dda53 100644 --- a/.github/workflows/targets-integration.yml +++ b/.github/workflows/targets-integration.yml @@ -189,4 +189,4 @@ jobs: - name: Open or update failure-tracking issue uses: ./.github/actions/schedule-failure-issue with: - github-token: ${{ secrets.GITHUB_TOKEN }} + github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }} diff --git a/Cargo.lock b/Cargo.lock index 7a49b39db..268406b13 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -271,7 +271,7 @@ version = "1.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -282,7 +282,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" dependencies = [ "anstyle", "once_cell_polyfill", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -658,22 +658,6 @@ dependencies = [ "serde_json", ] -[[package]] -name = "astral-tokio-tar" -version = "0.7.0" -source = "git+https://github.com/cxymds/tokio-tar.git?rev=603756478b7668436e464519c77ccac22a99ba96#603756478b7668436e464519c77ccac22a99ba96" -dependencies = [ - "futures-core", - "libc", - "portable-atomic", - "rustc-hash", - "rustix", - "tokio", - "tokio-stream", - "xattr", - "zerocopy", -] - [[package]] name = "async-channel" version = "2.5.0" @@ -701,9 +685,9 @@ dependencies = [ [[package]] name = "async-compression" -version = "0.4.46" +version = "0.4.47" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4f10dafd0c8d2e51ae9a748805777613ed0bbe17bf586b76c8311f45c020a32f" +checksum = "ef217a77a86a6e3dab9a5b3c81dc445b603fe743a90c1cb10a2f2144628d8cfa" dependencies = [ "compression-codecs", "compression-core", @@ -1031,9 +1015,9 @@ dependencies = [ [[package]] name = "aws-sdk-s3" -version = "1.146.0" +version = "1.146.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37b5ffaae346b9bd486ebdc3eb7053ec8ab8a1ad0f19a332eefeff036ed559e6" +checksum = "2cd651b4400d4011b8927b83a9552bf90ff11e6e5da0b9f0a7583247aceec971" dependencies = [ "arc-swap", "aws-credential-types", @@ -1885,9 +1869,9 @@ checksum = "cd17eb909a8c6a894926bfcc3400a4bb0e732f5a57d37b1f14e8b29e329bace8" [[package]] name = "camino" -version = "1.2.5" +version = "1.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d" +checksum = "bbbad30e4b4c14a39e3cc8aed085a12a327257c316619c93581e017bc52be591" dependencies = [ "serde_core", ] @@ -1906,7 +1890,7 @@ dependencies = [ "maybe-owned", "rustix", "rustix-linux-procfs", - "windows-sys 0.61.2", + "windows-sys 0.60.2", "winx", ] @@ -2000,9 +1984,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.4.5" +version = "1.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "005ec2760ca554fae18df7a11195552ec576cd665632a881bc011d5bb2fd4d80" +checksum = "a3eb0f42d6c360dc3f8a821f6bf2fdea7f72bfd36b3076eb0e6d1e9e0752fff4" dependencies = [ "find-msvc-tools", "jobserver", @@ -2128,9 +2112,9 @@ dependencies = [ [[package]] name = "clap" -version = "4.6.6" +version = "4.6.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca" +checksum = "aa8876b300ab35ba921adea3dfd70157a46249b33f95c9084ae5709785478946" dependencies = [ "clap_builder", "clap_derive", @@ -2138,9 +2122,9 @@ dependencies = [ [[package]] name = "clap_builder" -version = "4.6.6" +version = "4.6.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889" +checksum = "ec0797fb7aeb1406c84efac526901f7ec3ead2124f946b494e72879d4b54704d" dependencies = [ "anstream", "anstyle", @@ -2150,9 +2134,9 @@ dependencies = [ [[package]] name = "clap_derive" -version = "4.6.4" +version = "4.6.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" +checksum = "f9c751b79415d4e559e3d1fcf128e09e720eb673a06d26cf6f392d37d75b66e0" dependencies = [ "heck 0.5.0", "proc-macro2", @@ -2162,9 +2146,9 @@ dependencies = [ [[package]] name = "clap_lex" -version = "1.1.0" +version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" +checksum = "1c133bc6a41be0d194c306b5506d15e6feeea7b1d6604bd3f8310dfb2ca96486" [[package]] name = "cmake" @@ -2238,9 +2222,9 @@ dependencies = [ [[package]] name = "compression-codecs" -version = "0.4.41" +version = "0.4.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "58a6d0db8759036a783bc7c3f7a07f8cef3bf9470eb1db3bc86e8bcd1c5d0fe8" +checksum = "257c7085cbb71be72d8fb97edff08b03d86d5d9f2222b9cc34a6bec87093bc10" dependencies = [ "brotli 8.0.4", "bzip2", @@ -2483,9 +2467,9 @@ dependencies = [ [[package]] name = "crc32fast" -version = "1.5.1" +version = "1.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8498c871161e1742aaa9d52551b2d6ebdd4c3d45a3be423e3728f33b955be550" +checksum = "01a7799fd6b852db0e61728dde9a204c423b44d689dbd432522543614b490e78" dependencies = [ "cfg-if", ] @@ -2889,9 +2873,9 @@ checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06" [[package]] name = "datafusion" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "96f76f0167ed0842b29a3d1e41be3c034c0a46409a3a703cc4cc84ee8c24abf4" +checksum = "14313430439ef858ed5f52ca9c840e9eea9844da34d3716d05a476449634da5d" dependencies = [ "arrow", "arrow-schema", @@ -2938,9 +2922,9 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d79ec3460f6ed5c58f9b3f2d873fbc77748b82653bff1b4cdaf06de33bb4e05f" +checksum = "2c0756ebc770bcebae366dfd273dad50935f6103b4d212253cb17e2e53c5427f" dependencies = [ "arrow", "async-trait", @@ -2963,9 +2947,9 @@ dependencies = [ [[package]] name = "datafusion-catalog-listing" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b48cef241e2efcfd496fe05ae4d0d5de20793451862faefe406c397a467e12d4" +checksum = "bbde2961698705d8402f0f382f08876fc6f1126ffde790ef17833cf0ab87cd59" dependencies = [ "arrow", "async-trait", @@ -2987,9 +2971,9 @@ dependencies = [ [[package]] name = "datafusion-common" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f72810485975c258f1b4d00baab31728470676c60c5546f366ebd0d99f05ab6" +checksum = "a6e06fe711ff815c8cf3143158547aea599702fd43f76f5866c70d5a5b3693a0" dependencies = [ "arrow", "arrow-ipc", @@ -3014,9 +2998,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "533c28e75dba52f41bde187d23a1cb24ab91c7c097966824fa471e67b60320ea" +checksum = "f7a5107acff4e3ed50a7732acec06bba40e74ab49df021397ed250d60f04e42a" dependencies = [ "futures", "log", @@ -3025,9 +3009,9 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b00a1fa0da26f6087136a82fea7f13c76a672cbab452d4086952a7cf770a19b" +checksum = "f69f45d2d700785be8df9d4b44410e1c8e4a856f10cc728b418a1066c0b594fb" dependencies = [ "arrow", "async-trait", @@ -3055,9 +3039,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-arrow" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ad17ec881bff2ed7768b4bfe971d3efbf3473f2fd1f9d365447bccbdf908678" +checksum = "81650af64894ef34b46d2ef818ef244b40a81d90818f3bc642bf8fbf6758f8e0" dependencies = [ "arrow", "arrow-ipc", @@ -3079,9 +3063,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5345285b0c3eaab412e7539b706973c083bd7e5bce575de5e0a3da488d08d1d" +checksum = "b5e2519bed59a6907c79298b9c21fe807bbc34e4ff8414577d699b0741083593" dependencies = [ "arrow", "async-trait", @@ -3102,9 +3086,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da02fb9324f56bd8c53f1ee2e949547425cb66f76adc6832b10d44f80a1221d2" +checksum = "635e61290e171597f6e50427e0cb486eda468ab47e880fa24d15bbb0fa24ac7e" dependencies = [ "arrow", "async-trait", @@ -3125,9 +3109,9 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c0b0dc1453952952fd5c69ad1c7f6042176e69ed233011d47e07cf74ed0949e" +checksum = "07cd518f85b3a027d913047b4ab0811111a2b734fdf84a51b78234455f10d8fa" dependencies = [ "arrow", "arrow-schema", @@ -3157,15 +3141,15 @@ dependencies = [ [[package]] name = "datafusion-doc" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a88fd985bc0550c36f557db69543cc9d6393b1509783520b30e902f23c555da6" +checksum = "d44c5919b560862dd1cb20a5c75b3e48c3d9e49719c91d78da5c38eaad5495c4" [[package]] name = "datafusion-execution" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a98f1052f91b4991f0bf2ce1e4e36dfbdcda454a956b8c8d562c7c845e8fce1d" +checksum = "7620ad535af8730186bdec05d720e40d9a4584b40c7eb75f72c43aa7978e5484" dependencies = [ "arrow", "arrow-buffer", @@ -3189,9 +3173,9 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "464625a1f0e4b9df552d894fafcc8aac953ebbc8b0fa0acdaf20975fd615040e" +checksum = "ea229af8b73e566fc6ebb1c8dd42ab250f8855c5304bd173f4b50ead78b64304" dependencies = [ "arrow", "arrow-schema", @@ -3212,9 +3196,9 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2604994999d5aeca1d1df645ffc98bc787447aaff05dde27aad0342b48fc1fe0" +checksum = "6a18baa840f77760dcc9eaad281329bc57536c5c39e65e8649596f2a51216be7" dependencies = [ "arrow", "datafusion-common", @@ -3224,9 +3208,9 @@ dependencies = [ [[package]] name = "datafusion-functions" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "051e97533e6af53e4aa0a0667cadc886abcaf36c4a5925019c55c0aa4c218fde" +checksum = "763bbfeec40b55a8a961d7349c0fcfcec4f3f66cc234d6295250b95371b85e34" dependencies = [ "arrow", "arrow-buffer", @@ -3252,9 +3236,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2d0f1bb166d3572b6ed40e1afb2faaacade962abc08c2fcf04babee74681c56b" +checksum = "46c0c2b2e856883ecf6a4fa8b23e7c6041183af8a68d0583dc95d1d21e4eaee0" dependencies = [ "arrow", "datafusion-common", @@ -3273,9 +3257,9 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ed756770f5f98369e181d692fd5ee6b1127ffd7322caba92f3730f9f5c92333" +checksum = "e35e42c87dfbff06934ea0e99cfed3440ac614c3f3bfb8c33be35ca0587d4140" dependencies = [ "arrow", "datafusion-common", @@ -3285,9 +3269,9 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91173fdb5c0ff2a41169a8ffa1b385b8844f18728747bb0a37e35ad7d5772a4f" +checksum = "08079b0604cdcc43c7d6cf006cfd5d3ed9c235ade0c1ebe91085a62f875603df" dependencies = [ "arrow", "arrow-ord", @@ -3310,9 +3294,9 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b1bcdfb286a745461b126719c32700777e83df4f17cc44db5d71ebce5731e840" +checksum = "fe2070ecf5638a93c81c8babc7858339ef52165a9f54da1411afd69adf7d85ce" dependencies = [ "arrow", "async-trait", @@ -3326,9 +3310,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ec4b508f1f93f00038ba3e737e894ec6c775528b4369413386655ae6125f0fc" +checksum = "0e9a164ebe1998c5c3f9e15c9a0c4e6b599a0c8739b5471bfb089752441963f1" dependencies = [ "arrow", "datafusion-common", @@ -3343,9 +3327,9 @@ dependencies = [ [[package]] name = "datafusion-functions-window-common" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b352020834140073fbf5b46ee0ceb926e5074a9d0bcae1dbd91d0586d999cde" +checksum = "98965b183f92201c16798994820ae032649c4821e089c4c80e7d0f4485d41df5" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", @@ -3353,9 +3337,9 @@ dependencies = [ [[package]] name = "datafusion-macros" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "15192effab05d38cce10e92a6fb48c967b5f166b27b7195a165a72b232569c58" +checksum = "3a711bd5a06ec85d9683e142d28a98ad0679a7220669f5c1b8589e57e9b29e46" dependencies = [ "datafusion-doc", "quote", @@ -3364,9 +3348,9 @@ dependencies = [ [[package]] name = "datafusion-optimizer" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "854445d9f7847e1e46089cf61b8d341a64382f14484e912c83a0f23b31216896" +checksum = "ff3256b62ff44616ec83c7f04741d21124970833bb377ed56d9abc5ec6f707ec" dependencies = [ "arrow", "chrono", @@ -3384,11 +3368,12 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "671558dad1d2aa253c39c0a4c52515958b99eb91abf649f4b88d5e69cc55282f" +checksum = "c1e37a25b5aa1f3a1db114f0ee3605bf01639b83a831dcde35119ee61f0b11d6" dependencies = [ "arrow", + "arrow-schema", "datafusion-common", "datafusion-expr", "datafusion-expr-common", @@ -3406,9 +3391,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-adapter" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ffae3d78c2da80ecc829cb58536cc5aca2e99cf1365eda694fc75bfe288861e0" +checksum = "3a4d12787d42e88b26b0d3ec006192fa23ceb832ef2a9e87439ce820a8e038cc" dependencies = [ "arrow", "datafusion-common", @@ -3421,9 +3406,9 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-common" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d9092ed15e7203fbd0903215172f7c9d18f10d94cba35137f3b3836f7c46f16" +checksum = "d13bf7f50fd902f5afc46d50759f5185b5620c40b24054f80f7c648897d3e5a6" dependencies = [ "arrow", "chrono", @@ -3438,9 +3423,9 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9005b6cf50b57b72d476c6ed4662b04be7ca6be5320ba9127c6d0b7e4218095b" +checksum = "11fbe7e6665072957d44ed9587676e17a1f9ab3acfadfc287dac23750585f18a" dependencies = [ "arrow", "datafusion-common", @@ -3458,9 +3443,9 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5787e4fcff4adc4fce8948441103a99705018b49c8dff0720b650bd7a15da112" +checksum = "1265d58e5bce07d154e642a51ff43033b576a6ae40d50a29ac2b9f311004eb52" dependencies = [ "arrow", "arrow-data", @@ -3493,9 +3478,9 @@ dependencies = [ [[package]] name = "datafusion-pruning" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e651c8df0b90daed6a7be5921ec0ee379e6909705f063eeff70fd4e35010e4c" +checksum = "74b06b333405c05015836ed36cea02cc29d1e2ee8fb414a77532d8346fb56492" dependencies = [ "arrow", "datafusion-common", @@ -3509,9 +3494,9 @@ dependencies = [ [[package]] name = "datafusion-session" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fb56667ee38217efab19b895d9a936052cfb47ed438a19663351bdc42a6214a1" +checksum = "e7616b17b1d2fee3e9b13c651eac91b768d153265e6ba88e06e0b4af7883f69c" dependencies = [ "arrow-schema", "async-trait", @@ -3524,9 +3509,9 @@ dependencies = [ [[package]] name = "datafusion-sql" -version = "55.0.0" +version = "55.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c29067cb9d32f8e603c45e15d61ea18f1069f96ceafeceb4e18466b8e5b31d9" +checksum = "5b4848be66dd008d55adbe1dbc8cbd6a2502588ad0c0b8483bab0f43fd4495d0" dependencies = [ "arrow", "bigdecimal", @@ -3942,7 +3927,7 @@ dependencies = [ "libc", "option-ext", "redox_users", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -3989,7 +3974,6 @@ name = "e2e_test" version = "1.0.0" dependencies = [ "anyhow", - "astral-tokio-tar", "async-compression", "async-trait", "aws-config", @@ -4033,6 +4017,7 @@ dependencies = [ "rustfs-protos", "rustfs-rio", "rustfs-signer", + "rustfs-tokio-tar", "rustfs-utils", "rustls", "s3s", @@ -4287,7 +4272,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -4491,7 +4476,7 @@ checksum = "94e7099f6313ecacbe1256e8ff9d617b75d1bcb16a6fddef94866d225a01a14a" dependencies = [ "io-lifetimes 2.0.4", "rustix", - "windows-sys 0.59.0", + "windows-sys 0.52.0", ] [[package]] @@ -5798,9 +5783,9 @@ checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" [[package]] name = "jiff" -version = "0.2.35" +version = "0.2.37" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" +checksum = "0ab1baf72f08796de0260609515130699b890ac25f30e610ad894bc5856cafdb" dependencies = [ "defmt", "jiff-core", @@ -5815,18 +5800,19 @@ dependencies = [ [[package]] name = "jiff-core" -version = "0.1.0" +version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" +checksum = "5e52fe76043ccecc9005d2305ebaadf7d7fc0cc89ca6baa10a94d6bc68c7128c" dependencies = [ "defmt", + "log", ] [[package]] name = "jiff-static" -version = "0.2.35" +version = "0.2.37" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" +checksum = "378268a1116ad67ae6228701118ac9f491d78fda38a40a1f1a9e1348de6f7212" dependencies = [ "jiff-core", "proc-macro2", @@ -6163,9 +6149,9 @@ dependencies = [ [[package]] name = "liblzma-sys" -version = "0.4.8" +version = "0.4.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a0dad045e4b1b7b170be4b60b54b780cafb4490165461bac7d1cf7b703f61d5f" +checksum = "7c7e3581f367a78d7b7e7ae948d023310556f0cfc13156c2e4e00e25616492b9" dependencies = [ "cc", "libc", @@ -6180,9 +6166,9 @@ checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" [[package]] name = "libredox" -version = "0.1.23" +version = "0.1.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8d8f1ea3f21fd3405dcaf6c9b5c1630af9afc422d9073ea39c5f6d6c772e08ed" +checksum = "6480ccc157a1389bb2e4891b24751b0f798ba640d22386f23143fbcc89da195a" dependencies = [ "libc", ] @@ -6295,9 +6281,9 @@ dependencies = [ [[package]] name = "lru-slab" -version = "0.1.2" +version = "0.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" +checksum = "4050469837a6ff301cd14c1f8f24f88549e6d548f24f64e2148eb0f72cebc51f" [[package]] name = "lz4" @@ -6963,7 +6949,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -7123,7 +7109,7 @@ version = "5.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d" dependencies = [ - "base64 0.22.1", + "base64 0.21.7", "chrono", "getrandom 0.2.17", "http 1.5.0", @@ -8140,9 +8126,9 @@ checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" [[package]] name = "ppmd-rust" -version = "1.4.1" +version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e9219bcb9d7aca6b2f63c83cf100cf78bcd619ac46e6ecbd0dd90869a39345d" +checksum = "196a7c80b9a7652aba7cc070827516c2abe4ccdf53d128e1944003cf5726cff1" [[package]] name = "ppv-lite86" @@ -8660,9 +8646,9 @@ dependencies = [ [[package]] name = "quinn" -version = "0.11.11" +version = "0.11.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8" +checksum = "4051e23e9185c255a7e33ef59cdbca87a22d359052eecd22fc6b901fb37d9d11" dependencies = [ "bytes", "cfg_aliases", @@ -8680,9 +8666,9 @@ dependencies = [ [[package]] name = "quinn-proto" -version = "0.11.17" +version = "0.11.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "04759210543be93709136e28212294a659ef5001836ff4eab4d663e4529bba83" +checksum = "a9746dbde176634f4f2f1faf2404e30a31b2bc1e9cafb5329c95d8177a18c9fc" dependencies = [ "aws-lc-rs", "bytes", @@ -8712,7 +8698,7 @@ dependencies = [ "once_cell", "socket2", "tracing", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -9481,7 +9467,6 @@ dependencies = [ "aes-gcm", "anyhow", "apache-avro", - "astral-tokio-tar", "async-trait", "async_zip", "atoi 3.1.0", @@ -9498,6 +9483,7 @@ dependencies = [ "clap", "const-str", "datafusion", + "ed25519-dalek 3.0.0", "faster-hex", "flatbuffers", "flate2", @@ -9525,12 +9511,15 @@ dependencies = [ "mime_guess", "moka", "opentelemetry", + "opentelemetry-proto", "opentelemetry_sdk", "p256 0.14.0", "parking_lot", "percent-encoding", "pin-project-lite", "proptest", + "prost 0.14.4", + "pyroscope", "quick-xml", "rand 0.10.2", "rcgen", @@ -9582,9 +9571,11 @@ dependencies = [ "rustfs-targets", "rustfs-test-utils", "rustfs-tls-runtime", + "rustfs-tokio-tar", "rustfs-trusted-proxies", "rustfs-utils", "rustfs-zip", + "rustix", "rustls", "rustls-pki-types", "s3s", @@ -9835,7 +9826,6 @@ dependencies = [ "s3s", "serde", "serde_json", - "serde_urlencoded", "serial_test", "sha2 0.11.0", "shadow-rs", @@ -9906,7 +9896,6 @@ dependencies = [ "rmp-serde", "rustfs-config", "rustfs-utils", - "s3s", "serde", "serde_json", "sha2 0.11.0", @@ -10444,7 +10433,6 @@ dependencies = [ name = "rustfs-protocols" version = "1.0.0" dependencies = [ - "astral-tokio-tar", "async-compression", "async-trait", "axum", @@ -10478,6 +10466,7 @@ dependencies = [ "rustfs-storage-api", "rustfs-test-utils", "rustfs-tls-runtime", + "rustfs-tokio-tar", "rustfs-trusted-proxies", "rustfs-utils", "rustls", @@ -10575,7 +10564,6 @@ dependencies = [ "rustfs-tls-runtime", "rustfs-utils", "rustls-pki-types", - "s3s", "serde", "serde_json", "sha1 0.11.0", @@ -10923,6 +10911,23 @@ dependencies = [ "tracing", ] +[[package]] +name = "rustfs-tokio-tar" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7451523c123cf217e6f492b7cf0f9f07749c0c2aa5d569b3915b62af3b36cc3b" +dependencies = [ + "futures-core", + "libc", + "portable-atomic", + "rustc-hash", + "rustix", + "tokio", + "tokio-stream", + "xattr", + "zerocopy", +] + [[package]] name = "rustfs-trusted-proxies" version = "1.0.0" @@ -11006,11 +11011,11 @@ dependencies = [ name = "rustfs-zip" version = "1.0.0" dependencies = [ - "astral-tokio-tar", "async-compression", "futures", "hotpath", "rustfs-rio", + "rustfs-tokio-tar", "serde", "serde_json", "sha2 0.11.0", @@ -11073,7 +11078,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -11156,7 +11161,7 @@ dependencies = [ "security-framework", "security-framework-sys", "webpki-root-certs", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -11212,8 +11217,9 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" [[package]] name = "s3s" -version = "0.15.0" -source = "git+https://github.com/s3s-project/s3s.git?rev=f3e17541f366696bf0cbaf380fcbd8b44c17eba4#f3e17541f366696bf0cbaf380fcbd8b44c17eba4" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d29a4db6a51f1cd3269a36354ce4aa7db2be8816cee42a642b9c85cd3769577" dependencies = [ "arc-swap", "arrayvec", @@ -11270,8 +11276,9 @@ dependencies = [ [[package]] name = "s3s-rfc2047" -version = "0.16.0-alpha.1" -source = "git+https://github.com/s3s-project/s3s.git?rev=f3e17541f366696bf0cbaf380fcbd8b44c17eba4#f3e17541f366696bf0cbaf380fcbd8b44c17eba4" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9370a2f353a7929b79ae7b5f692b4758c59aa19526b18feff6a977e8ff17da5a" dependencies = [ "base64-simd", "thiserror 2.0.20", @@ -11279,8 +11286,9 @@ dependencies = [ [[package]] name = "s3s-sigv2" -version = "0.16.0-alpha.1" -source = "git+https://github.com/s3s-project/s3s.git?rev=f3e17541f366696bf0cbaf380fcbd8b44c17eba4#f3e17541f366696bf0cbaf380fcbd8b44c17eba4" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f277b63ffc9a2135c49d52726199b7436c2f86746d625496984f5bb5cabef050" dependencies = [ "base64-simd", "hmac 0.13.0", @@ -11292,8 +11300,9 @@ dependencies = [ [[package]] name = "s3s-sigv4" -version = "0.16.0-alpha.1" -source = "git+https://github.com/s3s-project/s3s.git?rev=f3e17541f366696bf0cbaf380fcbd8b44c17eba4#f3e17541f366696bf0cbaf380fcbd8b44c17eba4" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a979a401f6ae4035bc22e4ac0f7a9a5923ba93841c931e8e28652fe211d991bc" dependencies = [ "arrayvec", "base64-simd", @@ -11916,9 +11925,9 @@ dependencies = [ [[package]] name = "smallvec" -version = "1.16.0" +version = "1.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9be42f50aa861c555654aa3a37f52f4b1074bacf4e48fe0ef7fa584e80f1f0f" +checksum = "ba467056f1b547ed52077911161fc86985becbc60e8e1857c8a144dab0def891" dependencies = [ "serde", ] @@ -11957,7 +11966,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -12108,7 +12117,7 @@ dependencies = [ "cfg-if", "libc", "psm", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -12413,10 +12422,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" dependencies = [ "fastrand", - "getrandom 0.4.3", + "getrandom 0.3.4", "once_cell", "rustix", - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -12586,18 +12595,9 @@ dependencies = [ [[package]] name = "tinyvec" -version = "1.13.2" +version = "1.13.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4cf0ded5c4e56918d8f8a339e1bb67d038d3bc6d144ac407904015ba2e4cde9b" -dependencies = [ - "tinyvec_macros", -] - -[[package]] -name = "tinyvec_macros" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" +checksum = "fd3ca314f692efd6c868f8408f53fe444634a845f96c028b97d35f6a1f79f0ee" [[package]] name = "tls_codec" @@ -12769,9 +12769,9 @@ dependencies = [ [[package]] name = "toml_edit" -version = "0.25.14+spec-1.1.0" +version = "0.25.15+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2195eec204e2764644a4ea619704f9fbe5e0673038eded55ad9956f24fca0cc" +checksum = "1340ea94a5856333492c9064b02c778b191dd2c853778d9609debdcdfea3a614" dependencies = [ "indexmap 2.14.2", "toml_datetime", @@ -13518,7 +13518,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.60.2", ] [[package]] @@ -13648,15 +13648,6 @@ dependencies = [ "windows-targets 0.52.6", ] -[[package]] -name = "windows-sys" -version = "0.59.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" -dependencies = [ - "windows-targets 0.52.6", -] - [[package]] name = "windows-sys" version = "0.60.2" @@ -13829,7 +13820,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f3fd376f71958b862e7afb20cfe5a22830e1963462f3a17f49d82a6c1d1f42d" dependencies = [ "bitflags 2.13.2", - "windows-sys 0.59.0", + "windows-sys 0.52.0", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 4c3d0d52e..ab2eb19e7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -72,7 +72,7 @@ resolver = "3" edition = "2024" license = "Apache-2.0" repository = "https://github.com/rustfs/rustfs" -rust-version = "1.98.0" +rust-version = "1.98.1" version = "1.0.0" homepage = "https://rustfs.com" description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. " @@ -145,7 +145,7 @@ rustfs-zip = { path = "./crates/zip", version = "1.0.0" } async-channel = "2.5.0" async_zip = { default-features = false, version = "0.0.19" } mysql_async = { default-features = false, version = "0.37.1" } -async-compression = { version = "0.4.46" } +async-compression = { version = "0.4.47" } async-recursion = "1.1.1" async-trait = "0.1.92" async-nats = { version = "0.50.0", default-features = false } @@ -224,7 +224,7 @@ proptest = "1" # Time and Date chrono = { version = "0.4.45" } humantime = "2.4.0" -jiff = { version = "0.2.35" } +jiff = { version = "0.2.37" } time = { version = "0.3.55" } # Database @@ -235,8 +235,7 @@ tokio-postgres-rustls = "0.14.0" # Utilities and Tools anyhow = "1.0.104" arc-swap = "1.9.2" -# RUSTFS_COMPAT_TODO(tokio-tar-extension-limits): keep the fork pin while Snowball and Swift still depend on it. Remove after Snowball uses a released tar-codec/tar-framing API that exposes precedence-resolved MinIO vendor records, RustFS preserves cancellation-safe ownership of large streamed members, footerless minio-go input is accepted only at an authenticated complete request boundary, the existing resource-limit, cancellation, and error-fuse regressions pass, and Swift no longer needs this fork. -astral-tokio-tar = { git = "https://github.com/cxymds/tokio-tar.git", rev = "603756478b7668436e464519c77ccac22a99ba96" } +rustfs-tokio-tar = { version = "0.7.1" } # Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures. tar-codec = "0.0.14" tar-framing = "0.0.14" @@ -245,7 +244,7 @@ atomic_enum = "0.3.0" aws-config = { version = "1.12.0" } aws-credential-types = { version = "1.3.0" } aws-sdk-kms = { default-features = false, version = "1.118.0" } -aws-sdk-s3 = { default-features = false, version = "1.146.0" } +aws-sdk-s3 = { default-features = false, version = "1.146.1" } aws-sdk-sts = { default-features = false, version = "1.114.0" } aws-smithy-async = { version = "1.3.0" } aws-smithy-http-client = { default-features = false, version = "1.4.0" } @@ -253,7 +252,7 @@ aws-smithy-runtime-api = { version = "1.16.0" } aws-smithy-types = { version = "1.6.3" } base64-simd = "0.8.0" brotli = "9.0.0" -clap = { version = "4.6.6" } +clap = { version = "4.6.7" } const-str = { version = "1.1.0" } convert_case = "0.12.0" criterion = { version = "0.8" } @@ -261,8 +260,9 @@ crossbeam-queue = "0.3.14" crossbeam-channel = "0.5.17" crossbeam-deque = "0.8.8" crossbeam-utils = "0.8.23" -datafusion = { default-features = false, version = "55.0.0" } +datafusion = { default-features = false, version = "55.1.0" } derive_builder = "0.20.2" +ed25519-dalek = "3.0.0" enumset = "1.1.14" faster-hex = "0.10.0" flate2 = "1.1.10" @@ -312,12 +312,12 @@ rustify = { version = "0.7", default-features = false } rustix = { version = "1.1.4" } rust-embed = { version = "8.12.0" } rustc-hash = { version = "2.1.3" } -s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "f3e17541f366696bf0cbaf380fcbd8b44c17eba4", version = "0.15.0", features = ["minio"] } -s3s-sigv4 = { git = "https://github.com/s3s-project/s3s.git", rev = "f3e17541f366696bf0cbaf380fcbd8b44c17eba4", version = "0.16.0-alpha.1" } +s3s = { version = "0.16.0", features = ["minio"] } +s3s-sigv4 = { version = "0.16.0" } serial_test = "4.0.1" shadow-rs = { default-features = false, version = "2.0.0" } siphasher = "1.0.3" -smallvec = { version = "1.16.0" } +smallvec = { version = "1.16.1" } compact_str = "0.10.0" snap = "1.1.2" starshard = { version = "2.3.0" } @@ -388,6 +388,14 @@ ignored = ["hotpath", "rustfs"] # CARGO_PROFILE_DEV_DEBUG=full cargo build debug = "line-tables-only" +# To further speed up the process, completely disable debug information for dependencies. +[profile.dev.package."*"] +debug = false +# When you really need to debug, you can enable full debugging information using the `--profile debugging` option. +[profile.debugging] +inherits = "dev" +debug = true + [profile.release] opt-level = 3 lto = "thin" diff --git a/crates/common/src/trace_bus.rs b/crates/common/src/trace_bus.rs index 5e6a90d60..9239fb5cd 100644 --- a/crates/common/src/trace_bus.rs +++ b/crates/common/src/trace_bus.rs @@ -127,6 +127,44 @@ pub struct TraceEvent { pub attrs: SmallVec<[TraceAttr; TRACE_ATTR_INLINE_CAPACITY]>, } +/// A telemetry operation whose published shape cannot retain request data. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TelemetryTraceOperation { + GetObject, + PutObject, + HeadObject, + ListObjects, + InternalRpc, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TelemetryTraceStatus { + Ok, + Error, +} + +/// A pre-classified S3 or internode RPC observation. +/// +/// This event deliberately has no request path, headers, object identity, +/// payload, or free-form attributes. Producers must classify the operation +/// and outcome before publishing it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct TelemetryTraceEvent { + pub operation: TelemetryTraceOperation, + pub duration: Duration, + pub status: TelemetryTraceStatus, +} + +impl TelemetryTraceEvent { + pub const fn new(operation: TelemetryTraceOperation, duration: Duration, status: TelemetryTraceStatus) -> Self { + Self { + operation, + duration, + status, + } + } +} + impl TraceEvent { pub fn new(kind: TraceKind, func: TraceFunc) -> Self { Self { @@ -174,15 +212,20 @@ impl TraceEvent { pub struct TraceBus { sender: broadcast::Sender>, subscriber_count: Arc, + telemetry_sender: broadcast::Sender>, + telemetry_subscriber_count: Arc, } impl TraceBus { pub fn new(capacity: usize) -> Self { let capacity = capacity.max(1); let (sender, _receiver) = broadcast::channel(capacity); + let (telemetry_sender, _telemetry_receiver) = broadcast::channel(capacity); Self { sender, subscriber_count: Arc::new(AtomicUsize::new(0)), + telemetry_sender, + telemetry_subscriber_count: Arc::new(AtomicUsize::new(0)), } } @@ -206,6 +249,27 @@ impl TraceBus { self.sender.send(Arc::new(build())).is_ok() } + + pub fn telemetry_subscriber_count(&self) -> usize { + self.telemetry_subscriber_count.load(Ordering::Acquire) + } + + pub fn subscribe_telemetry(&self) -> TelemetryTraceSubscription { + let receiver = self.telemetry_sender.subscribe(); + self.telemetry_subscriber_count.fetch_add(1, Ordering::AcqRel); + TelemetryTraceSubscription { + receiver, + subscriber_count: Arc::clone(&self.telemetry_subscriber_count), + } + } + + pub fn emit_telemetry(&self, build: impl FnOnce() -> TelemetryTraceEvent) -> bool { + if self.telemetry_subscriber_count() == 0 { + return false; + } + + self.telemetry_sender.send(Arc::new(build())).is_ok() + } } impl Default for TraceBus { @@ -236,6 +300,28 @@ impl Drop for TraceSubscription { } } +#[derive(Debug)] +pub struct TelemetryTraceSubscription { + receiver: broadcast::Receiver>, + subscriber_count: Arc, +} + +impl TelemetryTraceSubscription { + pub async fn recv(&mut self) -> Result, broadcast::error::RecvError> { + self.receiver.recv().await + } + + pub fn try_recv(&mut self) -> Result, broadcast::error::TryRecvError> { + self.receiver.try_recv() + } +} + +impl Drop for TelemetryTraceSubscription { + fn drop(&mut self) { + self.subscriber_count.fetch_sub(1, Ordering::AcqRel); + } +} + pub fn global_trace_bus() -> &'static TraceBus { GLOBAL_TRACE_BUS.get_or_init(TraceBus::default) } @@ -252,6 +338,18 @@ pub fn trace_subscriber_count() -> usize { global_trace_bus().subscriber_count() } +pub fn subscribe_telemetry_trace_events() -> TelemetryTraceSubscription { + global_trace_bus().subscribe_telemetry() +} + +pub fn telemetry_trace_emit(build: impl FnOnce() -> TelemetryTraceEvent) -> bool { + global_trace_bus().emit_telemetry(build) +} + +pub fn telemetry_trace_subscriber_count() -> usize { + global_trace_bus().telemetry_subscriber_count() +} + #[cfg(test)] mod tests { use super::*; @@ -330,4 +428,30 @@ mod tests { .expect_err("receiver should observe lag instead of blocking publishers"); assert!(matches!(err, broadcast::error::RecvError::Lagged(_))); } + + #[tokio::test] + async fn telemetry_subscription_exposes_only_classified_fields() { + let bus = TraceBus::new(4); + let _unrelated_subscription = bus.subscribe(); + let built = AtomicUsize::new(0); + assert!(!bus.emit_telemetry(|| { + built.fetch_add(1, Ordering::Relaxed); + TelemetryTraceEvent::new(TelemetryTraceOperation::GetObject, Duration::ZERO, TelemetryTraceStatus::Ok) + })); + assert_eq!(built.load(Ordering::Relaxed), 0); + + let mut subscription = bus.subscribe_telemetry(); + assert_eq!(bus.telemetry_subscriber_count(), 1); + + assert!(bus.emit_telemetry(|| { + TelemetryTraceEvent::new(TelemetryTraceOperation::GetObject, Duration::from_micros(7), TelemetryTraceStatus::Ok) + })); + + let event = subscription.recv().await.expect("classified telemetry event"); + assert_eq!(event.operation, TelemetryTraceOperation::GetObject); + assert_eq!(event.duration, Duration::from_micros(7)); + assert_eq!(event.status, TelemetryTraceStatus::Ok); + drop(subscription); + assert_eq!(bus.telemetry_subscriber_count(), 0); + } } diff --git a/crates/config/README.md b/crates/config/README.md index 525129606..2e73f4d30 100644 --- a/crates/config/README.md +++ b/crates/config/README.md @@ -62,6 +62,37 @@ Current guidance: - `RUSTFS_CORS_ALLOWED_ORIGINS` defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set `*` for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins. - `RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS` defaults to `*` for the console service. +## Console URL prefix + +`RUSTFS_CONSOLE_PREFIX` changes the embedded console URL prefix. The default is +`/rustfs/console`. For example, `RUSTFS_CONSOLE_PREFIX=/console` serves the UI at +`http://localhost:9001/console/`. Nested prefixes such as `/management/console` +are supported; one trailing slash is removed. Restart the server after changing it. + +The prefix must be a non-root absolute path of at most 256 bytes, with nonempty +segments containing only ASCII letters, digits, `-`, `_`, `.`, or `~`. Dot +segments, encoded characters, and overlaps with reserved admin, RPC, health, +profiling, browser entry, and icon routes are rejected at startup. `/` is not supported. +Choose a prefix that does not collide with S3 bucket paths. + +The console routes, embedded frontend asset URLs, browser redirects, and OIDC +console redirects use this prefix. Admin API paths and the identity provider's +`/rustfs/admin/v3/oidc/callback/...` URL remain unchanged. `RUSTFS_CONSOLE_ADDRESS` +continues to control only the listening address and port. The server adapts bundled +console asset references from their build-time base path to the runtime prefix. + +OEM builds can set `RUSTFS_CONSOLE_BASE_PATH` when compiling RustFS to embed a +different default, such as `/nuofans/console`. Build the bundled console with the +same `NEXT_PUBLIC_BASE_PATH`. An unset or empty build variable retains +`/rustfs/console`. The build path must satisfy the validation rules above and must +not have a trailing slash. + +At startup, `RUSTFS_CONSOLE_PREFIX` takes precedence over the compiled default. +Changing `RUSTFS_CONSOLE_BASE_PATH` when starting an existing binary has no effect; +rebuild both components to change the embedded default. If a runtime prefix is +configured, asset adaptation uses the compiled base path as its source, including +when restoring `/rustfs/console` for a custom OEM build. + ## Browser redirect environment variables - `RUSTFS_BROWSER_REDIRECT_URL` sets the externally reachable browser origin used for OIDC callback, console success redirect, and logout fallback URLs. Configure it to the public scheme and authority without a path, for example `https://console.example.com`. In load-balancer deployments, keep OIDC authorize and callback requests on the same backend node because the in-flight OIDC `state` is local to the RustFS node. diff --git a/crates/config/src/constants/app.rs b/crates/config/src/constants/app.rs index 36ea95d5a..2d4044aa4 100644 --- a/crates/config/src/constants/app.rs +++ b/crates/config/src/constants/app.rs @@ -213,6 +213,11 @@ pub const ENV_RUSTFS_CONSOLE_ENABLE: &str = "RUSTFS_CONSOLE_ENABLE"; /// Environment variable for console server address. pub const ENV_RUSTFS_CONSOLE_ADDRESS: &str = "RUSTFS_CONSOLE_ADDRESS"; +/// URL path prefix for the embedded console, read once at server startup. +pub const ENV_RUSTFS_CONSOLE_PREFIX: &str = "RUSTFS_CONSOLE_PREFIX"; +/// Default embedded console URL path prefix. +pub const DEFAULT_CONSOLE_PREFIX: &str = "/rustfs/console"; + /// Public browser entrypoint used to build OIDC callback and console redirects. /// /// This should be the externally reachable scheme and authority, without a path. diff --git a/crates/e2e_test/Cargo.toml b/crates/e2e_test/Cargo.toml index 70d1bbeb9..6b1dd5ccf 100644 --- a/crates/e2e_test/Cargo.toml +++ b/crates/e2e_test/Cargo.toml @@ -131,7 +131,7 @@ md-5 = { workspace = true } opentelemetry-proto = { workspace = true } prost.workspace = true sha2 = { workspace = true } -astral-tokio-tar = { workspace = true } +rustfs-tokio-tar = { workspace = true } s3s = { workspace = true, features = ["minio"] } zstd.workspace = true time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] } diff --git a/crates/ecstore/Cargo.toml b/crates/ecstore/Cargo.toml index c4c30330a..4a8ec662a 100644 --- a/crates/ecstore/Cargo.toml +++ b/crates/ecstore/Cargo.toml @@ -214,7 +214,6 @@ aws-smithy-types = { workspace = true } aws-smithy-runtime-api = { workspace = true, features = ["http-1x"] } parking_lot = { workspace = true } base64-simd.workspace = true -serde_urlencoded.workspace = true google-cloud-storage = { workspace = true, optional = true } google-cloud-auth = { workspace = true, optional = true } faster-hex = { workspace = true } diff --git a/crates/ecstore/src/api/mod.rs b/crates/ecstore/src/api/mod.rs index 68b8143fe..c3f788b76 100644 --- a/crates/ecstore/src/api/mod.rs +++ b/crates/ecstore/src/api/mod.rs @@ -548,20 +548,21 @@ pub mod rio { pub mod rpc { pub use crate::cluster::rpc::{ - AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, - PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, - ScannerBucketListing, ScannerDirtyUsageAcknowledgement, ScannerPeerActivity, ScannerPeerDirtyUsageBucket, - ScannerPeerDirtyUsageSnapshot, ScannerPublicationLease, ScannerScopedDirtyUsageAckEntry, TONIC_RPC_PREFIX, - TonicInterceptor, build_put_file_auth_trailer, check_and_record_signed_rpc_nonce, decode_heal_bucket_rpc_options, - encode_heal_bucket_rpc_options, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers, - gen_tonic_signature_interceptor, node_service_time_out_client, node_service_time_out_client_no_auth, - normalize_tonic_rpc_audience, set_tonic_canonical_body_digest, sign_ns_scanner_capability, - sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, sign_tonic_rpc_response_proof, - tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, - verify_ns_scanner_capability, verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_auth_trailer, - verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest, - verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof, - verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap, + AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, MAX_NETWORK_PROBE_BYTES, MAX_NETWORK_PROBE_DURATION, + NetworkPeerProbeClient, NetworkPeerProbeError, NetworkPeerProbeMeasurement, NetworkPeerTarget, PEER_RESTDRY_RUN, + PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG, + SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerDirtyUsageAcknowledgement, ScannerPeerActivity, + ScannerPeerDirtyUsageBucket, ScannerPeerDirtyUsageSnapshot, ScannerPublicationLease, ScannerScopedDirtyUsageAckEntry, + TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer, check_and_record_signed_rpc_nonce, + decode_heal_bucket_rpc_options, encode_heal_bucket_rpc_options, gen_signature_headers, gen_tonic_replay_scope_headers, + gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client, + node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, set_tonic_canonical_body_digest, + sign_ns_scanner_capability, sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, + sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, + tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_ns_scanner_capability_with_tier_registry_generation, + verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response, + verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, + verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap, }; } diff --git a/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs b/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs index 8b5a6ee39..048ba1bfa 100644 --- a/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs +++ b/crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs @@ -82,18 +82,16 @@ use rustfs_config::{ ENV_TRANSITION_WORKERS, ENV_TRANSITION_WORKERS_ABSOLUTE_MAX, }; use rustfs_data_usage::TierStats; +use rustfs_filemeta::metadata_keys; use rustfs_filemeta::{ - FileInfo, FileInfoOpts, NULL_VERSION_ID, RestoreStatusOps, TRANSITION_COMPLETE, get_file_info, is_restored_object_on_disk, + FileInfo, FileInfoOpts, NULL_VERSION_ID, RestoreStatus, RestoreStatusOps, TRANSITION_COMPLETE, get_file_info, + is_restored_object_on_disk, }; use rustfs_scanner_metrics::metrics::{ IlmAction, Metrics, ScannerLifecycleExpiryStateUpdate, ScannerLifecycleTransitionStateUpdate, global_metrics, }; use rustfs_utils::{get_env_i64, get_env_usize, path::encode_dir_object, string::parse_bool}; -use s3s::dto::{ - BucketLifecycleConfiguration, ExpirationStatus, ObjectLockConfiguration, RestoreRequest, RestoreRequestType, RestoreStatus, - Timestamp, -}; -use s3s::header::X_AMZ_RESTORE; +use s3s::dto::{BucketLifecycleConfiguration, ExpirationStatus, ObjectLockConfiguration, RestoreRequest, RestoreRequestType}; use sha2::{Digest, Sha256}; use std::any::Any; use std::collections::{BTreeMap, HashMap, HashSet}; @@ -5174,10 +5172,10 @@ pub async fn put_restore_opts( } let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), rreq.days.unwrap_or(1)); meta.insert( - X_AMZ_RESTORE.as_str().to_string(), + metadata_keys::RESTORE.to_string(), RestoreStatus { is_restore_in_progress: Some(false), - restore_expiry_date: Some(Timestamp::from(restore_expiry)), + restore_expiry_date: Some(restore_expiry), } .to_string(), ); @@ -5913,6 +5911,7 @@ mod tests { use rustfs_config::ENV_MAX_EXPIRY_WORKERS; use rustfs_config::ENV_TRANSITION_WORKERS_ABSOLUTE_MAX; use rustfs_data_usage::TierStats; + use rustfs_filemeta::metadata_keys; use rustfs_filemeta::{FileInfo, FileMeta}; #[cfg(feature = "test-util")] use rustfs_s3_client::transition_api::ReaderImpl; @@ -5922,7 +5921,6 @@ mod tests { NoncurrentVersionExpiration, ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule, OutputLocation, RestoreRequest, RestoreRequestType, S3Location, Timestamp, Transition, TransitionStorageClass, }; - use s3s::header::{X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE}; use serial_test::serial; use sha2::{Digest, Sha256}; use std::collections::HashMap; @@ -12243,7 +12241,7 @@ mod tests { "locked historical null", ObjectInfo { user_defined: Arc::new(HashMap::from([( - X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), + metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string(), )])), ..historical_null.clone() @@ -13021,7 +13019,7 @@ mod tests { let lc = latest_expiration_lifecycle(); let object = current_object_with_metadata( ReplicationStatusType::Completed, - HashMap::from([(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), "ON".to_string())]), + HashMap::from([(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string())]), ); let event = eval_action_from_lifecycle(&lc, None, &object).await; @@ -13039,10 +13037,10 @@ mod tests { ReplicationStatusType::Completed, HashMap::from([ ( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(), ), - (X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), retain_until), + (metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until), ]), ); @@ -13064,10 +13062,10 @@ mod tests { ReplicationStatusType::Completed, HashMap::from([ ( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), ObjectLockRetentionMode::COMPLIANCE.to_string(), ), - (X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), retain_until), + (metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until), ]), ); object.transitioned_object.status = TRANSITION_COMPLETE.to_string(); diff --git a/crates/ecstore/src/bucket/lifecycle/object_lock_boundary.rs b/crates/ecstore/src/bucket/lifecycle/object_lock_boundary.rs index 21d1c0656..493926d02 100644 --- a/crates/ecstore/src/bucket/lifecycle/object_lock_boundary.rs +++ b/crates/ecstore/src/bucket/lifecycle/object_lock_boundary.rs @@ -37,7 +37,7 @@ mod tests { #[test] fn is_object_locked_by_metadata_preserves_object_lock_parser_behavior() { let mut user_defined = HashMap::new(); - user_defined.insert("x-amz-object-lock-legal-hold".to_string(), "ON".to_string()); + user_defined.insert(rustfs_filemeta::metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string()); assert!(is_object_locked_by_metadata(&user_defined, false)); assert!(!is_object_locked_by_metadata(&user_defined, true)); diff --git a/crates/ecstore/src/bucket/migration.rs b/crates/ecstore/src/bucket/migration.rs index 59d2278d3..6724a408a 100644 --- a/crates/ecstore/src/bucket/migration.rs +++ b/crates/ecstore/src/bucket/migration.rs @@ -20,7 +20,7 @@ use crate::disk::{BUCKET_META_PREFIX, MIGRATING_META_BUCKET, RUSTFS_META_BUCKET} use crate::error::{Error, Result, is_err_strict_not_found, is_err_strict_volume_not_found}; use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader}; use crate::storage_api_contracts::{ - bucket::{BucketOperations, BucketOptions}, + bucket::{BucketInfo, BucketOperations, BucketOptions}, list::{ListOperations, StorageListObjectVersionsInfo, StorageListObjectsV2Info, StorageObjectInfoOrErr, StorageWalkOptions}, object::{DeletedObject, EcstoreObjectIO, EcstoreObjectOperations, ObjectIO, ObjectOperations, ObjectToDelete}, range::HTTPRangeSpec, @@ -382,22 +382,19 @@ where DeletedObject = DeletedObject, >, { - // Older peers abort walk_dir streams when the legacy volume is absent, - // which loses the typed not-found error before listing quorum resolution. - // Stat the namespace first; only a confirmed missing volume skips migration. - match store - .get_bucket_info( - MIGRATING_META_BUCKET, - &BucketOptions { - no_metadata: true, - ..Default::default() - }, - ) - .await - { - Ok(_) => {} - Err(err) if is_err_strict_volume_not_found(&err) => return Ok(()), - Err(err) => return Err(err), + if !legacy_iam_source_exists( + store + .get_bucket_info( + MIGRATING_META_BUCKET, + &BucketOptions { + no_metadata: true, + ..Default::default() + }, + ) + .await, + )? { + debug!("No legacy IAM volume found"); + return Ok(()); } let opts = ObjectOptions { @@ -477,6 +474,14 @@ where Ok(()) } +fn legacy_iam_source_exists(result: Result) -> Result { + match result { + Ok(_) => Ok(true), + Err(error) if is_err_strict_volume_not_found(&error) => Ok(false), + Err(error) => Err(error), + } +} + fn next_iam_migration_page(truncated: bool, previous: Option, next: Option) -> Result> { if !truncated { return Ok(None); @@ -490,6 +495,18 @@ fn next_iam_migration_page(truncated: bool, previous: Option, next: Opti #[cfg(test)] mod tests { + #[test] + fn legacy_iam_source_probe_skips_only_an_absent_volume() { + use super::{BucketInfo, Error, legacy_iam_source_exists}; + + assert!(!legacy_iam_source_exists(Err(Error::VolumeNotFound)).expect("absent legacy volume is a no-op")); + assert!(legacy_iam_source_exists(Ok(BucketInfo::default())).expect("existing legacy volume must be migrated")); + assert!(matches!( + legacy_iam_source_exists(Err(Error::ErasureReadQuorum)), + Err(Error::ErasureReadQuorum) + )); + } + #[test] fn migration_errors_group_by_cause_and_retain_typed_record_context() { use super::{Error, MigrationMetadataError}; diff --git a/crates/ecstore/src/bucket/object_lock/objectlock.rs b/crates/ecstore/src/bucket/object_lock/objectlock.rs index a45e1ac50..8b5a9abdc 100644 --- a/crates/ecstore/src/bucket/object_lock/objectlock.rs +++ b/crates/ecstore/src/bucket/object_lock/objectlock.rs @@ -13,9 +13,7 @@ // limitations under the License. use super::types::{LegalHoldStatus, ObjectLegalHold, ObjectRetention, RetentionMode}; -use rustfs_utils::http::headers::{ - AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, -}; +use rustfs_filemeta::metadata_keys; use std::collections::HashMap; use time::{OffsetDateTime, format_description}; @@ -35,7 +33,7 @@ pub fn utc_now_ntp() -> OffsetDateTime { pub fn get_object_retention_meta(meta: &HashMap) -> ObjectRetention { // The persisted metadata keys are the lowercase wire header names. - let mode_str = meta.get(AMZ_OBJECT_LOCK_MODE_LOWER); + let mode_str = meta.get(metadata_keys::OBJECT_LOCK_MODE); let Some(mode_str) = mode_str else { return ObjectRetention::default(); @@ -46,7 +44,7 @@ pub fn get_object_retention_meta(meta: &HashMap) -> ObjectRetent return ObjectRetention::default(); }; - let till_str = meta.get(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER); + let till_str = meta.get(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE); let retain_until_date = till_str.and_then(|s| OffsetDateTime::parse(s, &format_description::well_known::Iso8601::DEFAULT).ok()); @@ -58,7 +56,7 @@ pub fn get_object_retention_meta(meta: &HashMap) -> ObjectRetent } pub fn get_object_legalhold_meta(meta: &HashMap) -> ObjectLegalHold { - let hold_str = meta.get(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER); + let hold_str = meta.get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD); ObjectLegalHold { status: hold_str.and_then(|s| parse_legalhold_status(s)), diff --git a/crates/ecstore/src/bucket/object_lock/objectlock_sys.rs b/crates/ecstore/src/bucket/object_lock/objectlock_sys.rs index 5c01ae85d..8fe7eb431 100644 --- a/crates/ecstore/src/bucket/object_lock/objectlock_sys.rs +++ b/crates/ecstore/src/bucket/object_lock/objectlock_sys.rs @@ -19,9 +19,7 @@ use crate::bucket::object_lock::objectlock; use crate::bucket::object_lock::types::{DefaultRetention, LegalHoldStatus, RetentionMode}; use crate::error::{Error, Result, StorageError}; use crate::object_api::{ObjectInfo, ObjectOptions}; -use rustfs_utils::http::headers::{ - AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, -}; +use rustfs_filemeta::metadata_keys; use std::sync::Arc; use time::OffsetDateTime; @@ -317,7 +315,7 @@ fn persisted_lock_value<'a>(obj_info: &'a ObjectInfo, key: &str) -> Option<&'a S /// Whether the version's persisted legal hold is ON. Any other non-empty /// value than ON/OFF is malformed metadata and fails closed. fn legal_hold_locks(obj_info: &ObjectInfo) -> Result { - let Some(status) = persisted_lock_value(obj_info, AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER) else { + let Some(status) = persisted_lock_value(obj_info, metadata_keys::OBJECT_LOCK_LEGAL_HOLD) else { return Ok(false); }; match LegalHoldStatus::parse(status) { @@ -335,8 +333,8 @@ fn active_retention( default_retention: Option<&DefaultRetention>, obj_info: &ObjectInfo, ) -> Result> { - let mode = persisted_lock_value(obj_info, AMZ_OBJECT_LOCK_MODE_LOWER); - let retain_until = persisted_lock_value(obj_info, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER); + let mode = persisted_lock_value(obj_info, metadata_keys::OBJECT_LOCK_MODE); + let retain_until = persisted_lock_value(obj_info, metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE); match (mode, retain_until) { (None, None) => {} (Some(mode), Some(retain_until)) => { @@ -413,9 +411,6 @@ pub async fn check_object_lock_for_deletion( mod tests { use super::*; use crate::bucket::metadata_sys::configured_object_lock_state_for_tests; - use rustfs_utils::http::headers::{ - AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, - }; use time::{Date, Month, PrimitiveDateTime, Time}; fn make_datetime(year: i32, month: u8, day: u8) -> OffsetDateTime { @@ -509,7 +504,7 @@ mod tests { #[test] fn deletion_rejects_incomplete_persisted_retention_metadata() { let mut user_defined = std::collections::HashMap::new(); - user_defined.insert(AMZ_OBJECT_LOCK_MODE_LOWER.to_string(), RetentionMode::COMPLIANCE.to_string()); + user_defined.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), RetentionMode::COMPLIANCE.to_string()); let obj_info = ObjectInfo { user_defined: Arc::new(user_defined), ..Default::default() @@ -535,10 +530,10 @@ mod tests { for (case, mode, retain_until, expected) in cases { let mut user_defined = std::collections::HashMap::new(); if let Some(mode) = mode { - user_defined.insert(AMZ_OBJECT_LOCK_MODE_LOWER.to_string(), mode.to_string()); + user_defined.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), mode.to_string()); } if let Some(retain_until) = retain_until { - user_defined.insert(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER.to_string(), retain_until.to_string()); + user_defined.insert(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until.to_string()); } let obj_info = ObjectInfo { user_defined: Arc::new(user_defined), @@ -579,14 +574,14 @@ mod tests { /// source timestamp of every category that currently locks the version. #[test] fn replication_write_passes_worm_gate_only_with_every_locking_category_timestamp() { - let hold = [(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, "ON")]; + let hold = [(metadata_keys::OBJECT_LOCK_LEGAL_HOLD, "ON")]; let retention = [ - (AMZ_OBJECT_LOCK_MODE_LOWER, "GOVERNANCE"), - (AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, "2099-01-01T00:00:00Z"), + (metadata_keys::OBJECT_LOCK_MODE, "GOVERNANCE"), + (metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE, "2099-01-01T00:00:00Z"), ]; let expired = [ - (AMZ_OBJECT_LOCK_MODE_LOWER, "COMPLIANCE"), - (AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, "2000-01-01T00:00:00Z"), + (metadata_keys::OBJECT_LOCK_MODE, "COMPLIANCE"), + (metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE, "2000-01-01T00:00:00Z"), ]; let absent = ObjectLockConfigState::ConfirmedAbsent; let passes = |state: &ObjectLockConfigState, entries: &[&[(&str, &str)]], opts: &ObjectOptions| { @@ -606,7 +601,7 @@ mod tests { // Expired retention and a released hold no longer lock anything. assert!(passes( &absent, - &[&expired, &[(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, "OFF")]], + &[&expired, &[(metadata_keys::OBJECT_LOCK_LEGAL_HOLD, "OFF")]], &replication_opts(false, false) )); @@ -651,7 +646,7 @@ mod tests { ); // Default retention plus a legal hold: both categories need a timestamp. - let held = lock_object_info(lock_metadata(&[&[(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, "ON")]])); + let held = lock_object_info(lock_metadata(&[&[(metadata_keys::OBJECT_LOCK_LEGAL_HOLD, "ON")]])); assert!(!replication_write_may_pass_worm_gate(&state, &held, &replication_opts(false, true)).expect("judged")); assert!(!replication_write_may_pass_worm_gate(&state, &held, &replication_opts(true, false)).expect("judged")); assert!(replication_write_may_pass_worm_gate(&state, &held, &replication_opts(true, true)).expect("judged")); @@ -672,7 +667,7 @@ mod tests { assert!(replication_write_may_pass_worm_gate(&state, &delete_marker, &tagging_only).expect("judged")); // Cleared (empty) explicit keys fall back to the bucket default. - let cleared = lock_object_info(lock_metadata(&[&[(AMZ_OBJECT_LOCK_MODE_LOWER, "")]])); + let cleared = lock_object_info(lock_metadata(&[&[(metadata_keys::OBJECT_LOCK_MODE, "")]])); assert!(!replication_write_may_pass_worm_gate(&state, &cleared, &tagging_only).expect("judged")); } } @@ -690,7 +685,7 @@ mod tests { .expect_err("fabricated bucket lock metadata must not be judged"); assert!(err.to_string().contains("not authoritative")); - let malformed = lock_object_info(lock_metadata(&[&[(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, "MAYBE")]])); + let malformed = lock_object_info(lock_metadata(&[&[(metadata_keys::OBJECT_LOCK_LEGAL_HOLD, "MAYBE")]])); let err = replication_write_may_pass_worm_gate(&ObjectLockConfigState::ConfirmedAbsent, &malformed, &opts) .expect_err("malformed legal hold must not be judged"); assert!(err.to_string().contains("legal-hold")); @@ -738,15 +733,15 @@ mod tests { let cases: [(&str, &[&str]); 3] = [ ( "cleared retention", - &[AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER], + &[metadata_keys::OBJECT_LOCK_MODE, metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE], ), - ("cleared legal hold", &[AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER]), + ("cleared legal hold", &[metadata_keys::OBJECT_LOCK_LEGAL_HOLD]), ( "all cleared", &[ - AMZ_OBJECT_LOCK_MODE_LOWER, - AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, - AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, + metadata_keys::OBJECT_LOCK_MODE, + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE, + metadata_keys::OBJECT_LOCK_LEGAL_HOLD, ], ), ]; @@ -766,7 +761,7 @@ mod tests { #[test] fn deletion_rejects_invalid_persisted_legal_hold_metadata() { let mut user_defined = std::collections::HashMap::new(); - user_defined.insert(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER.to_string(), "INVALID".to_string()); + user_defined.insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "INVALID".to_string()); let obj_info = ObjectInfo { user_defined: Arc::new(user_defined), ..Default::default() diff --git a/crates/ecstore/src/bucket/replication/replication_filemeta_boundary.rs b/crates/ecstore/src/bucket/replication/replication_filemeta_boundary.rs index 6a8c9deaa..13ee89443 100644 --- a/crates/ecstore/src/bucket/replication/replication_filemeta_boundary.rs +++ b/crates/ecstore/src/bucket/replication/replication_filemeta_boundary.rs @@ -14,6 +14,19 @@ #[cfg(test)] pub(crate) use rustfs_filemeta::ObjectPartInfo; +/// Persisted object-metadata keys (`meta_user`); filemeta owns the on-disk spelling. +pub(crate) use rustfs_filemeta::metadata_keys; + +/// `FileInfo.metadata` of the xl.meta fixture written before `metadata_keys` existed. +#[cfg(test)] +pub(crate) fn pre_metadata_keys_fixture_metadata() -> std::collections::HashMap { + rustfs_filemeta::FileMeta::load(&rustfs_filemeta::test_data::create_pre_metadata_keys_xlmeta().expect("decode fixture hex")) + .expect("load fixture xl.meta") + .into_fileinfo("bucket", "object", "0b1e5a3a-1735-4a3a-8000-00000000a3a0", false, false, false) + .expect("fixture version to FileInfo") + .metadata +} + pub use rustfs_replication::{MrfOpKind, MrfReplicateEntry}; pub(crate) use rustfs_replication::{ REPLICATE_EXISTING, REPLICATE_HEAL_DELETE, ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, diff --git a/crates/ecstore/src/bucket/replication/replication_object_config.rs b/crates/ecstore/src/bucket/replication/replication_object_config.rs index 3f609533b..12391dc05 100644 --- a/crates/ecstore/src/bucket/replication/replication_object_config.rs +++ b/crates/ecstore/src/bucket/replication/replication_object_config.rs @@ -14,8 +14,8 @@ use std::{collections::HashMap, fmt, sync::Arc}; +use super::replication_filemeta_boundary::metadata_keys; use crate::bucket::metadata::BucketMetadata; -use rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS; use s3s::dto::{BucketVersioningStatus, ReplicationConfiguration, ReplicationRuleStatus, VersioningConfiguration}; use serde::{Deserialize, Serialize}; use time::OffsetDateTime; @@ -334,7 +334,7 @@ impl ReplicationConfig { } let mut user_defined = (*oi.user_defined).clone(); - user_defined.remove(AMZ_BUCKET_REPLICATION_STATUS); + user_defined.remove(metadata_keys::REPLICATION_STATUS); let dsc = must_replicate( oi.bucket.as_str(), diff --git a/crates/ecstore/src/bucket/replication/replication_resyncer.rs b/crates/ecstore/src/bucket/replication/replication_resyncer.rs index 05b0d3be1..2b680cbf0 100644 --- a/crates/ecstore/src/bucket/replication/replication_resyncer.rs +++ b/crates/ecstore/src/bucket/replication/replication_resyncer.rs @@ -19,6 +19,7 @@ use super::replication_error_boundary::{Error, Result, is_err_object_not_found, use super::replication_event_sink::{EventArgs, send_event, send_local_event}; #[cfg(test)] use super::replication_filemeta_boundary::ReplicationGenerationSnapshot; +use super::replication_filemeta_boundary::metadata_keys; use super::replication_filemeta_boundary::{ REPLICATE_EXISTING, ReplicateDecision, ReplicateObjectInfo, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType, ReplicationType, VersionPurgeStatusType, get_replication_state, @@ -85,9 +86,9 @@ use metrics::counter; use rmp_serde; use rustfs_s3_types::EventName; use rustfs_utils::http::{ - AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, - AMZ_TAGGING_DIRECTIVE, SUFFIX_REPLICATION_RESET, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TARGET_VERSION_ARN_PREFIX, - has_internal_suffix, insert_str, replication_target_versions, + AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_TAGGING_DIRECTIVE, + SUFFIX_REPLICATION_RESET, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TARGET_VERSION_ARN_PREFIX, has_internal_suffix, + insert_str, replication_target_versions, }; use rustfs_utils::{DEFAULT_SIP_HASH_KEY, get_env_usize, sip_hash}; #[cfg(test)] @@ -202,7 +203,7 @@ fn metadata_requires_existing_target(op_type: ReplicationType, object_info: &Obj op_type == ReplicationType::Metadata && object_info .user_defined - .get(AMZ_BUCKET_REPLICATION_STATUS) + .get(metadata_keys::REPLICATION_STATUS) .is_some_and(|status| status.eq_ignore_ascii_case(ReplicationStatusType::Replica.as_str())) } @@ -6283,7 +6284,7 @@ mod tests { version_id: roi.version_id, etag: Some("source-etag".to_string()), user_defined: Arc::new(HashMap::from([( - AMZ_BUCKET_REPLICATION_STATUS.to_string(), + metadata_keys::REPLICATION_STATUS.to_string(), ReplicationStatusType::Replica.as_str().to_string(), )])), ..Default::default() diff --git a/crates/ecstore/src/bucket/replication/replication_target_boundary.rs b/crates/ecstore/src/bucket/replication/replication_target_boundary.rs index eb24d22ce..e71ae9f19 100644 --- a/crates/ecstore/src/bucket/replication/replication_target_boundary.rs +++ b/crates/ecstore/src/bucket/replication/replication_target_boundary.rs @@ -52,6 +52,7 @@ pub use rustfs_replication::{VersionIdentityCapability, version_identity_capabil use super::replication_config_store::ReplicationConfigStore; use super::replication_error_boundary::{Error, Result}; +use super::replication_filemeta_boundary::metadata_keys; use super::replication_filemeta_boundary::{ReplicationAction, ReplicationStatusType, ReplicationType}; use super::replication_storage_boundary::ObjectInfo; use super::replication_tagging_boundary::ReplicationTagFilter; @@ -94,7 +95,7 @@ fn metadata_value<'a>(metadata: &'a HashMap, name: &str) -> Opti fn classify_replication_source_encryption(metadata: &HashMap) -> ReplicationSourceEncryption { let is_ssec = replication_object_is_ssec_encrypted(metadata); - let sse = metadata_value(metadata, AMZ_SERVER_SIDE_ENCRYPTION); + let sse = metadata_value(metadata, metadata_keys::SERVER_SIDE_ENCRYPTION); let kms_key_id = metadata_value(metadata, AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID); let kms_context = metadata_value(metadata, AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT); @@ -577,6 +578,52 @@ mod tests { use time::Duration; use uuid::Uuid; + /// backlog#1735 A3b SSE key-case finding. Every RustFS writer persists the + /// SSE intent as lowercase `x-amz-server-side-encryption` + /// (`encryption_material_to_metadata`; header-derived keys come from a + /// lowercase `http::HeaderMap`). The mixed-case `X-Amz-Server-Side-Encryption` + /// spelling only appears in outbound replication user metadata, which the + /// S3 client re-lowercases on the wire. This reader has always matched + /// the key ASCII-case-insensitively; keep that, so the old lowercase bytes + /// and any title-case spelling both classify, while any non-case drift of + /// the key reads as "no SSE intent". + #[test] + fn replication_sse_classification_reads_pre_module_xlmeta_key() { + let metadata = super::super::replication_filemeta_boundary::pre_metadata_keys_fixture_metadata(); + assert_eq!(metadata.get("x-amz-server-side-encryption").map(String::as_str), Some("AES256")); + assert_eq!(classify_replication_source_encryption(&metadata), ReplicationSourceEncryption::SseS3); + + let sse_only = |key: String| HashMap::from([(key, "AES256".to_string())]); + let key = metadata_keys::SERVER_SIDE_ENCRYPTION; + for spelling in [ + key.to_string(), + "X-Amz-Server-Side-Encryption".to_string(), + key.to_ascii_uppercase(), + ] { + assert_eq!( + classify_replication_source_encryption(&sse_only(spelling.clone())), + ReplicationSourceEncryption::SseS3, + "{spelling:?}" + ); + } + for idx in 0..key.len() { + let mut bytes = key.as_bytes().to_vec(); + bytes[idx] = if bytes[idx] == b'z' { + b'y' + } else if bytes[idx].is_ascii_alphabetic() { + b'z' + } else { + b'_' + }; + let mutated = String::from_utf8(bytes).expect("ascii"); + assert_eq!( + classify_replication_source_encryption(&sse_only(mutated.clone())), + ReplicationSourceEncryption::Plaintext, + "{mutated:?} must not read as the SSE key" + ); + } + } + /// Serialize an object-level checksum record the way /// `complete_multipart_upload` persists it for a **full-object** checksum: /// the record carries the plain algorithm type, without the MULTIPART diff --git a/crates/ecstore/src/cluster/rpc/mod.rs b/crates/ecstore/src/cluster/rpc/mod.rs index dc7405719..d33335d0d 100644 --- a/crates/ecstore/src/cluster/rpc/mod.rs +++ b/crates/ecstore/src/cluster/rpc/mod.rs @@ -17,6 +17,7 @@ pub(crate) mod client; pub(crate) mod context_propagation; pub(crate) mod http_auth; pub(crate) mod internode_data_transport; +mod network_probe; pub(crate) mod peer_rest_client; pub(crate) mod peer_s3_client; pub(crate) mod remote_disk; @@ -45,6 +46,10 @@ pub use http_auth::{ #[cfg(test)] pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport; pub use internode_data_transport::build_internode_data_transport_from_env; +pub use network_probe::{ + MAX_NETWORK_PROBE_BYTES, MAX_NETWORK_PROBE_DURATION, NetworkPeerProbeClient, NetworkPeerProbeError, + NetworkPeerProbeMeasurement, NetworkPeerTarget, +}; pub(crate) use peer_rest_client::TierConfigReloadOutcome; pub use peer_rest_client::{ KMS_SIGNAL_SUBSYSTEM, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, SERVICE_SIGNAL_REFRESH_CONFIG, diff --git a/crates/ecstore/src/cluster/rpc/network_probe.rs b/crates/ecstore/src/cluster/rpc/network_probe.rs new file mode 100644 index 000000000..a3c8e9882 --- /dev/null +++ b/crates/ecstore/src/cluster/rpc/network_probe.rs @@ -0,0 +1,318 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded, authenticated inter-node network probes. + +use std::time::{Duration, Instant}; + +use bytes::Bytes; +use rustfs_protos::ChannelClass; +use rustfs_protos::models::{PingBody, PingBodyBuilder}; +use rustfs_protos::proto_gen::node_service::{PingRequest, PingResponse}; +use thiserror::Error; +use tokio_util::sync::CancellationToken; +use tonic::{Code, Request}; + +use crate::cluster::rpc::client::{TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client_for_class}; +use crate::layout::endpoints::EndpointServerPools; + +pub const MAX_NETWORK_PROBE_BYTES: u64 = 1_048_576; +pub const MAX_NETWORK_PROBE_DURATION: Duration = Duration::from_secs(30); + +const PING_PROTOCOL_VERSION: u64 = 1; +const LATENCY_PAYLOAD: &[u8] = b"network-probe-latency-v1"; +const EXPECTED_RESPONSE_PAYLOAD: &[u8] = b"hello, caller"; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct NetworkPeerTarget { + pub alias: String, + pub address: String, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct NetworkPeerProbeMeasurement { + pub transferred_bytes: u64, + pub duration: Duration, + pub latency: Duration, +} + +#[derive(Clone, Copy, Debug, Error, PartialEq, Eq)] +pub enum NetworkPeerProbeError { + #[error("network peer is not part of the current topology")] + UnknownPeer, + #[error("network peer probe exceeds its resource limits")] + LimitExceeded, + #[error("network peer probe was cancelled")] + Cancelled, + #[error("network peer is unreachable")] + Unreachable, + #[error("network peer probe timed out")] + TimedOut, + #[error("network peer returned an invalid probe response")] + ProtocolFailure, +} + +#[derive(Clone, Debug)] +pub struct NetworkPeerProbeClient { + targets: Vec, +} + +impl NetworkPeerProbeClient { + pub fn from_endpoint_pools(endpoint_pools: &EndpointServerPools) -> Self { + let targets = endpoint_pools + .peer_grid_host_slots_sorted() + .into_iter() + .filter_map(|(_, address, is_local)| (!is_local).then_some(address).flatten()) + .enumerate() + .map(|(index, address)| NetworkPeerTarget { + alias: format!("peer-{}", index + 1), + address, + }) + .collect(); + Self { targets } + } + + pub fn targets(&self) -> Vec { + self.targets.clone() + } + + pub async fn probe( + &self, + peer_alias: &str, + traffic_bytes: u64, + max_duration: Duration, + cancel: &CancellationToken, + ) -> Result { + if traffic_bytes == 0 + || traffic_bytes > MAX_NETWORK_PROBE_BYTES + || max_duration.is_zero() + || max_duration > MAX_NETWORK_PROBE_DURATION + { + return Err(NetworkPeerProbeError::LimitExceeded); + } + if cancel.is_cancelled() { + return Err(NetworkPeerProbeError::Cancelled); + } + let address = self + .targets + .iter() + .find(|target| target.alias == peer_alias) + .map(|target| target.address.as_str()) + .ok_or(NetworkPeerProbeError::UnknownPeer)?; + let probe = probe_peer(address, traffic_bytes); + tokio::select! { + () = cancel.cancelled() => Err(NetworkPeerProbeError::Cancelled), + result = tokio::time::timeout(max_duration, probe) => { + result.map_err(|_| NetworkPeerProbeError::TimedOut)? + } + } + } +} + +async fn probe_peer(address: &str, traffic_bytes: u64) -> Result { + let started = Instant::now(); + let mut control = client(address, ChannelClass::Control).await?; + let latency_started = Instant::now(); + let latency_response = control + .ping(Request::new(ping_request(LATENCY_PAYLOAD))) + .await + .map_err(map_status)? + .into_inner(); + validate_ping_response(&latency_response)?; + let latency = latency_started.elapsed(); + + let payload_len = usize::try_from(traffic_bytes).map_err(|_| NetworkPeerProbeError::LimitExceeded)?; + let payload = vec![0x5a; payload_len]; + let mut bulk = client(address, ChannelClass::Bulk).await?; + let response = bulk + .ping(Request::new(ping_request(&payload))) + .await + .map_err(map_status)? + .into_inner(); + validate_ping_response(&response)?; + + Ok(NetworkPeerProbeMeasurement { + transferred_bytes: traffic_bytes, + duration: started.elapsed(), + latency, + }) +} + +async fn client( + address: &str, + class: ChannelClass, +) -> Result< + rustfs_protos::proto_gen::node_service::node_service_client::NodeServiceClient< + tonic::service::interceptor::InterceptedService, + >, + NetworkPeerProbeError, +> { + node_service_time_out_client_for_class( + &address.to_owned(), + TonicInterceptor::Signature(gen_tonic_signature_interceptor()), + class, + ) + .await + .map_err(|_| NetworkPeerProbeError::Unreachable) +} + +fn ping_request(payload: &[u8]) -> PingRequest { + let mut builder = flatbuffers::FlatBufferBuilder::with_capacity(payload.len().saturating_add(64)); + let payload = builder.create_vector(payload); + let mut body = PingBodyBuilder::new(&mut builder); + body.add_payload(payload); + let root = body.finish(); + builder.finish(root, None); + PingRequest { + version: PING_PROTOCOL_VERSION, + body: Bytes::copy_from_slice(builder.finished_data()), + } +} + +fn validate_ping_response(response: &PingResponse) -> Result<(), NetworkPeerProbeError> { + if response.version != PING_PROTOCOL_VERSION { + return Err(NetworkPeerProbeError::ProtocolFailure); + } + let body = flatbuffers::root::(&response.body).map_err(|_| NetworkPeerProbeError::ProtocolFailure)?; + if body + .payload() + .is_none_or(|payload| payload.bytes() != EXPECTED_RESPONSE_PAYLOAD) + { + return Err(NetworkPeerProbeError::ProtocolFailure); + } + Ok(()) +} + +fn map_status(status: tonic::Status) -> NetworkPeerProbeError { + match status.code() { + Code::Unavailable | Code::Unknown => NetworkPeerProbeError::Unreachable, + Code::DeadlineExceeded => NetworkPeerProbeError::TimedOut, + _ => NetworkPeerProbeError::ProtocolFailure, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::layout::endpoint::Endpoint; + use crate::layout::endpoints::{Endpoints, PoolEndpoints}; + + fn endpoint(value: &str, is_local: bool) -> Endpoint { + let mut endpoint = Endpoint::try_from(value).expect("test endpoint should parse"); + endpoint.is_local = is_local; + endpoint + } + + fn topology() -> EndpointServerPools { + EndpointServerPools::from(vec![PoolEndpoints { + legacy: false, + set_count: 1, + drives_per_set: 4, + endpoints: Endpoints::from(vec![ + endpoint("http://node-b:9000/data", false), + endpoint("http://node-a:9000/data", true), + endpoint("http://node-c:9000/data", false), + endpoint("http://node-b:9000/other", false), + ]), + cmd_line: String::new(), + platform: String::new(), + }]) + } + + #[test] + fn targets_are_unique_sorted_remote_nodes_with_stable_aliases() { + let client = NetworkPeerProbeClient::from_endpoint_pools(&topology()); + assert_eq!( + client.targets(), + vec![ + NetworkPeerTarget { + alias: "peer-1".to_owned(), + address: "http://node-b:9000".to_owned(), + }, + NetworkPeerTarget { + alias: "peer-2".to_owned(), + address: "http://node-c:9000".to_owned(), + }, + ] + ); + } + + #[test] + fn peer_aliases_remain_in_numeric_order_past_nine_peers() { + let endpoints: Vec = (1..=12) + .map(|index| endpoint(&format!("http://node-{index:02}:9000/data"), false)) + .collect(); + let topology = EndpointServerPools::from(vec![PoolEndpoints { + legacy: false, + set_count: 1, + drives_per_set: 12, + endpoints: Endpoints::from(endpoints), + cmd_line: String::new(), + platform: String::new(), + }]); + + let aliases = NetworkPeerProbeClient::from_endpoint_pools(&topology) + .targets() + .into_iter() + .map(|target| target.alias) + .collect::>(); + + assert_eq!(aliases, (1..=12).map(|index| format!("peer-{index}")).collect::>()); + } + + #[tokio::test] + async fn invalid_limits_and_unknown_peer_fail_before_dial() { + let client = NetworkPeerProbeClient::from_endpoint_pools(&topology()); + let cancel = CancellationToken::new(); + assert_eq!( + client + .probe("peer-1", MAX_NETWORK_PROBE_BYTES + 1, Duration::from_secs(1), &cancel) + .await, + Err(NetworkPeerProbeError::LimitExceeded) + ); + assert_eq!( + client.probe("peer-3", 1, Duration::from_secs(1), &cancel).await, + Err(NetworkPeerProbeError::UnknownPeer) + ); + cancel.cancel(); + assert_eq!( + client.probe("peer-1", 1, Duration::from_secs(1), &cancel).await, + Err(NetworkPeerProbeError::Cancelled) + ); + } + + #[test] + fn ping_body_carries_exact_requested_payload_and_response_is_validated() { + let request = ping_request(&vec![0x5a; 4096]); + let body = flatbuffers::root::(&request.body).expect("probe ping should be valid"); + assert_eq!(body.payload().expect("probe payload").len(), 4096); + + let valid = PingResponse { + version: PING_PROTOCOL_VERSION, + body: ping_request(EXPECTED_RESPONSE_PAYLOAD).body, + }; + assert_eq!(validate_ping_response(&valid), Ok(())); + + let wrong_version = PingResponse { + version: 2, + body: valid.body, + }; + assert_eq!(validate_ping_response(&wrong_version), Err(NetworkPeerProbeError::ProtocolFailure)); + let malformed = PingResponse { + version: PING_PROTOCOL_VERSION, + body: Bytes::from_static(b"not-flatbuffers"), + }; + assert_eq!(validate_ping_response(&malformed), Err(NetworkPeerProbeError::ProtocolFailure)); + } +} diff --git a/crates/ecstore/src/data_movement/mod.rs b/crates/ecstore/src/data_movement/mod.rs index 3c8cba3ac..4706435fc 100644 --- a/crates/ecstore/src/data_movement/mod.rs +++ b/crates/ecstore/src/data_movement/mod.rs @@ -2133,8 +2133,8 @@ mod tests { } } use crate::bucket::replication::{ReplicationStatusType, VersionPurgeStatusType}; + use rustfs_filemeta::metadata_keys; use rustfs_rio::{Checksum, ChecksumType}; - use s3s::header::{X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE}; use std::collections::HashMap; use std::io::Cursor; use std::sync::atomic::AtomicUsize; @@ -2192,16 +2192,16 @@ mod tests { assert_eq!(source.version_purge_status_internal, target.version_purge_status_internal); assert_eq!(source.version_purge_status, target.version_purge_status); assert_eq!( - source.user_defined.get(X_AMZ_OBJECT_LOCK_MODE.as_str()), - target.user_defined.get(X_AMZ_OBJECT_LOCK_MODE.as_str()) + source.user_defined.get(metadata_keys::OBJECT_LOCK_MODE), + target.user_defined.get(metadata_keys::OBJECT_LOCK_MODE) ); assert_eq!( - source.user_defined.get(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str()), - target.user_defined.get(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str()) + source.user_defined.get(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE), + target.user_defined.get(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE) ); assert_eq!( - source.user_defined.get(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()), - target.user_defined.get(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()) + source.user_defined.get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD), + target.user_defined.get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) ); assert_eq!(source.parts.len(), target.parts.len()); for (source_part, target_part) in source.parts.iter().zip(target.parts.iter()) { @@ -2841,13 +2841,13 @@ mod tests { let mod_time = OffsetDateTime::UNIX_EPOCH; let metadata = Arc::new(HashMap::from([ ("x-amz-meta-key".to_string(), "value".to_string()), - (rustfs_utils::http::AMZ_STORAGE_CLASS.to_string(), "STANDARD_IA".to_string()), - (X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), "GOVERNANCE".to_string()), + (rustfs_filemeta::metadata_keys::STORAGE_CLASS.to_string(), "STANDARD_IA".to_string()), + (metadata_keys::OBJECT_LOCK_MODE.to_string(), "GOVERNANCE".to_string()), ( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), "2030-01-01T00:00:00Z".to_string(), ), - (X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), "ON".to_string()), + (metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string()), ])); let part = ObjectPartInfo { number: 1, @@ -2887,12 +2887,12 @@ mod tests { rustfs_utils::http::SUFFIX_REPLICATION_STATUS.to_string(), "arn:minio:target=PENDING;".to_string(), ), - (X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), "COMPLIANCE".to_string()), + (metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()), ( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), "2031-01-01T00:00:00Z".to_string(), ), - (X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), "ON".to_string()), + (metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string()), ])), ..Default::default() }; @@ -2905,15 +2905,15 @@ mod tests { Some(&"arn:minio:target=PENDING;".to_string()) ); assert_eq!( - new_multipart_opts.user_defined.get(X_AMZ_OBJECT_LOCK_MODE.as_str()), + new_multipart_opts.user_defined.get(metadata_keys::OBJECT_LOCK_MODE), Some(&"COMPLIANCE".to_string()) ); assert_eq!( - put_opts.user_defined.get(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str()), + put_opts.user_defined.get(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE), Some(&"2031-01-01T00:00:00Z".to_string()) ); assert_eq!( - new_multipart_opts.user_defined.get(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()), + new_multipart_opts.user_defined.get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD), Some(&"ON".to_string()) ); } diff --git a/crates/ecstore/src/io_support/rio.rs b/crates/ecstore/src/io_support/rio.rs index da3054ca9..de6016410 100644 --- a/crates/ecstore/src/io_support/rio.rs +++ b/crates/ecstore/src/io_support/rio.rs @@ -560,7 +560,7 @@ mod tests { let mut headers = HeaderMap::new(); headers.insert("x-amz-trailer", HeaderValue::from_static("x-amz-checksum-crc32")); reader - .add_checksum_from_s3s(&headers, None, false) + .add_checksum(&headers, None, false) .expect("attach trailing checksum metadata"); let transformed = WritePlan::new() diff --git a/crates/ecstore/src/object_api/mod.rs b/crates/ecstore/src/object_api/mod.rs index f783bd5e6..60f7e6776 100644 --- a/crates/ecstore/src/object_api/mod.rs +++ b/crates/ecstore/src/object_api/mod.rs @@ -34,13 +34,12 @@ use crate::store::utils::clean_metadata; use crate::{bucket::lifecycle::bucket_lifecycle_audit::LcAuditEvent, bucket::lifecycle::lifecycle::TransitionOptions}; use bytes::Bytes; use http::{HeaderMap, HeaderValue}; +use rustfs_filemeta::metadata_keys; use rustfs_filemeta::{FileInfo, MetaCacheEntriesSorted, ObjectPartInfo, RestoreStatusOps as _, parse_restore_obj_status}; use rustfs_rio::Checksum; use rustfs_utils::CompressionAlgorithm; use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING; -use rustfs_utils::http::{ - AMZ_BUCKET_REPLICATION_STATUS, AMZ_RESTORE, AMZ_STORAGE_CLASS, SUFFIX_PLAINTEXT_CHECKSUM, get_consistent_str, -}; +use rustfs_utils::http::{SUFFIX_PLAINTEXT_CHECKSUM, get_consistent_str}; use rustfs_utils::path::decode_dir_object; use std::collections::HashMap; use std::fmt::Debug; @@ -106,6 +105,9 @@ mod object_mutation_hook; mod readers; mod types; +#[cfg(test)] +mod persisted_metadata_keys_tests; + #[cfg(test)] pub(crate) use body_cache_hook::clear_get_object_body_cache_hook; pub use body_cache_hook::{ diff --git a/crates/ecstore/src/object_api/persisted_metadata_keys_tests.rs b/crates/ecstore/src/object_api/persisted_metadata_keys_tests.rs new file mode 100644 index 000000000..806137361 --- /dev/null +++ b/crates/ecstore/src/object_api/persisted_metadata_keys_tests.rs @@ -0,0 +1,192 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Old-bytes coverage for the ecstore/lifecycle readers of the persisted +//! `meta_user` keys (backlog#1735 A3b). +//! +//! The fixture is xl.meta written by the code that predates +//! `rustfs_filemeta::metadata_keys`, with each key taken from its historical +//! source. Every ecstore reader migrated to the authority must still find its +//! key in those bytes, and a single-character drift of any key (ASCII case +//! included, because these lookups are exact) must make the reader miss. + +use super::*; +use crate::bucket::object_lock::objectlock::{get_object_legalhold_meta, get_object_retention_meta}; +use crate::bucket::object_lock::objectlock_sys::{ObjectLockBlockReason, check_object_lock_for_deletion_with_default_retention}; +use crate::bucket::object_lock::types::{LegalHoldStatus, RetentionMode}; +use crate::bucket::replication::ReplicationStatusType; +use crate::store::utils::remove_standard_storage_class; +use rustfs_filemeta::FileMeta; +use rustfs_filemeta::metadata_keys; +use rustfs_filemeta::test_data::create_pre_metadata_keys_xlmeta; + +const FIXTURE_VERSION_ID: &str = "0b1e5a3a-1735-4a3a-8000-00000000a3a0"; + +fn fixture_file_info() -> FileInfo { + FileMeta::load(&create_pre_metadata_keys_xlmeta().expect("decode fixture hex")) + .expect("load fixture xl.meta") + .into_fileinfo("bucket", "object", FIXTURE_VERSION_ID, false, false, false) + .expect("fixture version to FileInfo") +} + +fn object_info(fi: &FileInfo) -> ObjectInfo { + ObjectInfo::from_file_info(fi, "bucket", "object", true) +} + +/// Replaces the byte at `idx` with a different one of the same class: +/// ASCII letters flip case, anything else becomes `_`. +fn mutate_at(key: &str, idx: usize) -> String { + let mut bytes = key.as_bytes().to_vec(); + let b = bytes[idx]; + bytes[idx] = if b.is_ascii_lowercase() { + b.to_ascii_uppercase() + } else if b.is_ascii_uppercase() { + b.to_ascii_lowercase() + } else { + b'_' + }; + String::from_utf8(bytes).expect("ascii mutation stays utf-8") +} + +/// The fixture version with `key` stored under `mutated` instead. +fn file_info_with_key_renamed(key: &str, mutated: &str) -> FileInfo { + let mut fi = fixture_file_info(); + let value = fi.metadata.remove(key).expect("fixture carries the key"); + fi.metadata.insert(mutated.to_string(), value); + fi +} + +fn locked_by_object_lock(oi: &ObjectInfo) -> bool { + let retention = get_object_retention_meta(&oi.user_defined); + let hold = get_object_legalhold_meta(&oi.user_defined); + retention.mode.is_some() + || hold.status.is_some() + || rustfs_lifecycle::object_lock::is_object_locked_by_metadata(&oi.user_defined, false) + || check_object_lock_for_deletion_with_default_retention(None, oi, false) + .expect("object-lock check") + .is_some() +} + +#[test] +fn pre_module_xlmeta_object_lock_is_enforced_by_ecstore_and_lifecycle() { + let oi = object_info(&fixture_file_info()); + + let retention = get_object_retention_meta(&oi.user_defined); + assert!(matches!(retention.mode, Some(RetentionMode::Compliance))); + assert_eq!(retention.retain_until_date.map(|date| date.year()), Some(2099)); + assert!(matches!(get_object_legalhold_meta(&oi.user_defined).status, Some(LegalHoldStatus::On))); + assert!(rustfs_lifecycle::object_lock::is_object_locked_by_metadata(&oi.user_defined, false)); + assert!(matches!( + check_object_lock_for_deletion_with_default_retention(None, &oi, true), + Ok(Some(ObjectLockBlockReason::LegalHold)) + )); + + let mut without_hold = fixture_file_info(); + without_hold.metadata.remove(metadata_keys::OBJECT_LOCK_LEGAL_HOLD); + assert!(matches!( + check_object_lock_for_deletion_with_default_retention(None, &object_info(&without_hold), true), + Ok(Some(ObjectLockBlockReason::Retention { + mode: RetentionMode::Compliance, + .. + })) + )); +} + +#[test] +fn pre_module_xlmeta_restore_replication_and_storage_class_read_back() { + let fi = fixture_file_info(); + let oi = object_info(&fi); + assert!(!oi.restore_ongoing); + assert_eq!(oi.restore_expires.map(|date| date.year()), Some(9999)); + + let mut replica = fixture_file_info(); + replica.metadata.insert( + metadata_keys::REPLICATION_STATUS.to_string(), + ReplicationStatusType::Replica.as_str().to_string(), + ); + assert_eq!(object_info(&replica).replication_status, ReplicationStatusType::Replica); + + let mut metadata = fi.metadata; + assert_eq!(metadata.get(metadata_keys::STORAGE_CLASS).map(String::as_str), Some("GLACIER")); + remove_standard_storage_class(&mut metadata); + assert!(metadata.contains_key(metadata_keys::STORAGE_CLASS), "non-STANDARD class must stay"); + metadata.insert( + metadata_keys::STORAGE_CLASS.to_string(), + crate::config::storageclass::STANDARD.to_string(), + ); + remove_standard_storage_class(&mut metadata); + assert!(!metadata.contains_key(metadata_keys::STORAGE_CLASS), "STANDARD must be dropped"); +} + +/// Per-character mutation over the persisted keys the migrated ecstore and +/// lifecycle readers look up exactly: each drifted spelling must be missed. +#[test] +fn single_character_key_mutation_is_missed_by_ecstore_readers() { + // Keep a single object-lock category per object so each mutation is + // judged on its own: the legal hold alone, then the retention pair alone. + for key in [ + metadata_keys::OBJECT_LOCK_LEGAL_HOLD, + metadata_keys::OBJECT_LOCK_MODE, + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE, + ] { + for idx in 0..key.len() { + let mutated = mutate_at(key, idx); + let mut fi = file_info_with_key_renamed(key, &mutated); + if key == metadata_keys::OBJECT_LOCK_LEGAL_HOLD { + fi.metadata.remove(metadata_keys::OBJECT_LOCK_MODE); + fi.metadata.remove(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE); + assert!(!locked_by_object_lock(&object_info(&fi)), "legal hold under {mutated:?} must not lock"); + } else { + fi.metadata.remove(metadata_keys::OBJECT_LOCK_LEGAL_HOLD); + let oi = object_info(&fi); + let retention = get_object_retention_meta(&oi.user_defined); + if key == metadata_keys::OBJECT_LOCK_MODE { + assert!(retention.mode.is_none(), "mode under {mutated:?} must not be read"); + } else { + assert!(retention.retain_until_date.is_none(), "retain-until under {mutated:?} must not be read"); + } + assert!( + !rustfs_lifecycle::object_lock::is_object_locked_by_metadata(&oi.user_defined, false), + "retention with {mutated:?} must not read as locked" + ); + } + } + } + + for idx in 0..metadata_keys::RESTORE.len() { + let mutated = mutate_at(metadata_keys::RESTORE, idx); + let oi = object_info(&file_info_with_key_renamed(metadata_keys::RESTORE, &mutated)); + assert!(oi.restore_expires.is_none(), "restore status under {mutated:?} must not be read"); + } + + for idx in 0..metadata_keys::REPLICATION_STATUS.len() { + let mutated = mutate_at(metadata_keys::REPLICATION_STATUS, idx); + let mut fi = fixture_file_info(); + fi.metadata.remove(metadata_keys::REPLICATION_STATUS); + fi.metadata + .insert(mutated.clone(), ReplicationStatusType::Replica.as_str().to_string()); + assert_ne!( + object_info(&fi).replication_status, + ReplicationStatusType::Replica, + "replica status under {mutated:?} must not be read" + ); + } + + for idx in 0..metadata_keys::STORAGE_CLASS.len() { + let mutated = mutate_at(metadata_keys::STORAGE_CLASS, idx); + let mut metadata = HashMap::from([(mutated.clone(), crate::config::storageclass::STANDARD.to_string())]); + remove_standard_storage_class(&mut metadata); + assert!(metadata.contains_key(&mutated), "storage class under {mutated:?} must not be matched"); + } +} diff --git a/crates/ecstore/src/object_api/types.rs b/crates/ecstore/src/object_api/types.rs index dde5d30cc..eb0851585 100644 --- a/crates/ecstore/src/object_api/types.rs +++ b/crates/ecstore/src/object_api/types.rs @@ -1611,8 +1611,7 @@ impl ObjectInfo { .user_defined .iter() .filter(|(key, _)| { - !rustfs_utils::http::is_internal_key(key) - && !key.eq_ignore_ascii_case(rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS) + !rustfs_utils::http::is_internal_key(key) && !key.eq_ignore_ascii_case(metadata_keys::REPLICATION_STATUS) }) .collect::>(); user_metadata.sort_unstable_by(|left, right| left.0.cmp(right.0).then_with(|| left.1.cmp(right.1))); @@ -1832,7 +1831,7 @@ impl ObjectInfo { let mut replication_status = replication_status_from_filemeta(fi.replication_status()); if replication_status.is_empty() - && let Some(status) = fi.metadata.get(AMZ_BUCKET_REPLICATION_STATUS).cloned() + && let Some(status) = fi.metadata.get(metadata_keys::REPLICATION_STATUS).cloned() && status == ReplicationStatusType::Replica.as_str() { replication_status = ReplicationStatusType::Replica; @@ -1860,7 +1859,7 @@ impl ObjectInfo { let storage_class = Some( storageclass::effective_class( - fi.metadata.get(AMZ_STORAGE_CLASS).map(String::as_str), + fi.metadata.get(metadata_keys::STORAGE_CLASS).map(String::as_str), (fi.transition_status == rustfs_filemeta::TRANSITION_COMPLETE && !fi.transition_tier.is_empty()) .then_some(fi.transition_tier.as_str()), ) @@ -1869,7 +1868,7 @@ impl ObjectInfo { let mut restore_ongoing = false; let mut restore_expires = None; - if let Some(restore_status) = fi.metadata.get(AMZ_RESTORE).cloned() + if let Some(restore_status) = fi.metadata.get(metadata_keys::RESTORE).cloned() && let Ok(restore_status) = parse_restore_obj_status(&restore_status) { restore_ongoing = restore_status.on_going(); @@ -2842,7 +2841,7 @@ mod tests { storageclass::GLACIER, ] { let fi = FileInfo { - metadata: HashMap::from([(AMZ_STORAGE_CLASS.to_string(), legacy_label.to_string())]), + metadata: HashMap::from([(metadata_keys::STORAGE_CLASS.to_string(), legacy_label.to_string())]), ..Default::default() }; @@ -2859,7 +2858,7 @@ mod tests { #[test] fn from_file_info_preserves_transitioned_tier_storage_class() { let fi = FileInfo { - metadata: HashMap::from([(AMZ_STORAGE_CLASS.to_string(), storageclass::STANDARD_IA.to_string())]), + metadata: HashMap::from([(metadata_keys::STORAGE_CLASS.to_string(), storageclass::STANDARD_IA.to_string())]), transition_tier: "WARM-TIER".to_string(), transition_status: TRANSITION_COMPLETE.to_string(), ..Default::default() @@ -2874,7 +2873,7 @@ mod tests { #[test] fn from_file_info_ignores_a_tier_name_without_a_completed_transition() { let fi = FileInfo { - metadata: HashMap::from([(AMZ_STORAGE_CLASS.to_string(), storageclass::STANDARD_IA.to_string())]), + metadata: HashMap::from([(metadata_keys::STORAGE_CLASS.to_string(), storageclass::STANDARD_IA.to_string())]), transition_tier: "WARM-TIER".to_string(), ..Default::default() }; diff --git a/crates/ecstore/src/services/tier/warm_backend.rs b/crates/ecstore/src/services/tier/warm_backend.rs index 6ae923b44..b82853d9c 100644 --- a/crates/ecstore/src/services/tier/warm_backend.rs +++ b/crates/ecstore/src/services/tier/warm_backend.rs @@ -35,6 +35,7 @@ use crate::services::tier::{ }; use bytes::Bytes; use http::StatusCode; +use rustfs_filemeta::metadata_keys; use rustfs_s3_client::credentials::{Credentials, SignatureType, Static, Value}; use rustfs_s3_client::transition_api::{BucketLookupType, Options, TransitionClient, TransitionClientTimeouts, TransitionCore}; use rustfs_s3_client::{ @@ -48,10 +49,6 @@ use rustfs_utils::egress::validate_outbound_url; use rustfs_utils::http::headers::{ CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _, }; -use s3s::header::{ - X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, X_AMZ_REPLICATION_STATUS, - X_AMZ_STORAGE_CLASS, -}; use s3s::{ S3ErrorCode, dto::{ObjectLockLegalHoldStatus, ObjectLockRetentionMode, ReplicationStatus}, @@ -253,36 +250,39 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap opts.expires = expires; } - if let Some(mode) = metadata.lookup(X_AMZ_OBJECT_LOCK_MODE.as_str()) { + if let Some(mode) = metadata.lookup(metadata_keys::OBJECT_LOCK_MODE) { opts.mode = ObjectLockRetentionMode::from(mode.to_ascii_uppercase()); } if let Some(retain_until_date) = metadata - .lookup(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str()) + .lookup(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE) .and_then(parse_http_timestamp) { opts.retain_until_date = retain_until_date; } - if let Some(legalhold) = metadata.lookup(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()) { + if let Some(legalhold) = metadata.lookup(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) { opts.legalhold = ObjectLockLegalHoldStatus::from(legalhold.to_ascii_uppercase()); } - for key in [ + // Promoted keys are read above through `lookup`, which accepts more than + // one spelling, so strip every ASCII-case spelling here. The replication + // status is persisted as `X-Amz-Replication-Status`; the former exact + // lowercase removal left that spelling in the forwarded user metadata. + const PROMOTED_KEYS: [&str; 11] = [ CONTENT_TYPE, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_DISPOSITION, CACHE_CONTROL, EXPIRES, - X_AMZ_OBJECT_LOCK_MODE.as_str(), - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str(), - X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str(), - X_AMZ_REPLICATION_STATUS.as_str(), - X_AMZ_STORAGE_CLASS.as_str(), - ] { - metadata.remove(key); - } + metadata_keys::OBJECT_LOCK_MODE, + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE, + metadata_keys::OBJECT_LOCK_LEGAL_HOLD, + metadata_keys::REPLICATION_STATUS, + metadata_keys::STORAGE_CLASS, + ]; + metadata.retain(|key, _| !PROMOTED_KEYS.iter().any(|promoted| key.eq_ignore_ascii_case(promoted))); for suffix in [ rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_TRANSACTION_ID, @@ -2157,9 +2157,18 @@ mod tests { #[test] fn build_transition_put_options_preserves_object_lock_headers_when_present() { let mut metadata = HashMap::new(); - metadata.insert(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), "2026-03-23T00:00:00Z".to_string()); - metadata.insert(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.to_string(), ObjectLockLegalHoldStatus::ON.to_string()); - metadata.insert(X_AMZ_OBJECT_LOCK_MODE.to_string(), ObjectLockRetentionMode::GOVERNANCE.to_string()); + metadata.insert( + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), + "2026-03-23T00:00:00Z".to_string(), + ); + metadata.insert( + metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), + ObjectLockLegalHoldStatus::ON.to_string(), + ); + metadata.insert( + metadata_keys::OBJECT_LOCK_MODE.to_string(), + ObjectLockRetentionMode::GOVERNANCE.to_string(), + ); let opts = build_transition_put_options("COLD".to_string(), metadata); @@ -2173,15 +2182,35 @@ mod tests { let mut metadata = HashMap::new(); metadata.insert("name".to_string(), "object".to_string()); metadata.insert(CONTENT_TYPE.to_string(), "text/plain".to_string()); - metadata.insert(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.to_string(), ObjectLockLegalHoldStatus::ON.to_string()); - metadata.insert(X_AMZ_REPLICATION_STATUS.to_string(), "PENDING".to_string()); + metadata.insert( + metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), + ObjectLockLegalHoldStatus::ON.to_string(), + ); + metadata.insert("x-amz-replication-status".to_string(), "PENDING".to_string()); let opts = build_transition_put_options("COLD".to_string(), metadata); assert_eq!(opts.user_metadata.get("name"), Some(&"object".to_string())); assert!(!opts.user_metadata.contains_key(CONTENT_TYPE)); - assert!(!opts.user_metadata.contains_key(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str())); - assert!(!opts.user_metadata.contains_key(X_AMZ_REPLICATION_STATUS.as_str())); + assert!(!opts.user_metadata.contains_key(metadata_keys::OBJECT_LOCK_LEGAL_HOLD)); + assert!(!opts.user_metadata.contains_key("x-amz-replication-status")); + } + + /// Object metadata read back from xl.meta carries the replication status + /// under its persisted mixed-case key; it must not be forwarded to the + /// tier as user metadata either. + #[test] + fn build_transition_put_options_filters_persisted_replication_status_key() { + let metadata = HashMap::from([ + ("name".to_string(), "object".to_string()), + (metadata_keys::REPLICATION_STATUS.to_string(), "COMPLETED".to_string()), + (metadata_keys::STORAGE_CLASS.to_string(), "STANDARD".to_string()), + ]); + + let opts = build_transition_put_options("COLD".to_string(), metadata); + + assert_eq!(opts.user_metadata.len(), 1, "{:?}", opts.user_metadata); + assert_eq!(opts.user_metadata.get("name"), Some(&"object".to_string())); } #[test] diff --git a/crates/ecstore/src/set_disk/metadata.rs b/crates/ecstore/src/set_disk/metadata.rs index ec432f891..36e3bf6d6 100644 --- a/crates/ecstore/src/set_disk/metadata.rs +++ b/crates/ecstore/src/set_disk/metadata.rs @@ -23,6 +23,7 @@ use crate::disk::DiskOption; use crate::disk::endpoint::Endpoint; #[cfg(test)] use crate::disk::new_disk; +use rustfs_filemeta::metadata_keys; use rustfs_utils::http; use sha2::Digest; @@ -701,7 +702,7 @@ impl SetDisks { } fn is_replication_quorum_metadata_key(name: &str) -> bool { - if name.eq_ignore_ascii_case(http::AMZ_BUCKET_REPLICATION_STATUS) { + if name.eq_ignore_ascii_case(metadata_keys::REPLICATION_STATUS) { return true; } diff --git a/crates/ecstore/src/set_disk/mod.rs b/crates/ecstore/src/set_disk/mod.rs index 34af24dbf..a88c7d31a 100644 --- a/crates/ecstore/src/set_disk/mod.rs +++ b/crates/ecstore/src/set_disk/mod.rs @@ -122,6 +122,7 @@ use http::HeaderMap; use md5::{Digest as Md5Digest, Md5}; use regex::Regex; use rustfs_config::MI_B; +use rustfs_filemeta::metadata_keys; use rustfs_filemeta::{ FileInfo, FileMeta, FileMetaShallowVersion, MetaCacheEntries, MetaCacheEntry, ObjectPartInfo, RawFileInfo, merge_file_meta_versions, @@ -148,7 +149,6 @@ use rustfs_s3_types::EventName; #[cfg(test)] use rustfs_utils::http::SSEC_ALGORITHM_HEADER; use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING; -use rustfs_utils::http::headers::AMZ_STORAGE_CLASS; use rustfs_utils::http::headers::{ CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, }; @@ -162,7 +162,6 @@ use rustfs_utils::{ crypto::hex, path::{SLASH_SEPARATOR, encode_dir_object, has_suffix, path_join_buf}, }; -use s3s::header::{X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, X_AMZ_RESTORE}; use sha2::Sha256; use std::hash::{BuildHasher, Hash, Hasher}; use std::mem::{self}; @@ -214,7 +213,7 @@ pub(super) fn require_restore_operation_id(metadata: &HashMap, e } pub(super) fn restore_commit_operation_id_from_metadata(metadata: &HashMap) -> Result> { - if !metadata.contains_key(X_AMZ_RESTORE.as_str()) { + if !metadata.contains_key(metadata_keys::RESTORE) { return Ok(None); } restore_operation_id_from_metadata(metadata) @@ -5997,7 +5996,7 @@ impl SetDisks { } let disks = self.disks.read().await.clone(); - let storage_class = opts.user_defined.get(AMZ_STORAGE_CLASS).map(String::as_str); + let storage_class = opts.user_defined.get(metadata_keys::STORAGE_CLASS).map(String::as_str); let layout = resolve_write_layout( &storage_class_config, self.pool_index, @@ -10500,10 +10499,7 @@ mod tests { ); assert!(meta_a.replication_state_internal.is_some()); assert_eq!( - meta_a - .metadata - .get(rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS) - .map(String::as_str), + meta_a.metadata.get(metadata_keys::REPLICATION_STATUS).map(String::as_str), Some("COMPLETED") ); @@ -11888,11 +11884,11 @@ mod tests { let mut user_defined = HashMap::new(); user_defined.insert( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(), ); user_defined.insert( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), existing_until.format(&time::format_description::well_known::Rfc3339).unwrap(), ); @@ -11923,11 +11919,11 @@ mod tests { let mut user_defined = HashMap::new(); user_defined.insert( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::GOVERNANCE.to_string(), ); user_defined.insert( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), existing_until.format(&time::format_description::well_known::Rfc3339).unwrap(), ); @@ -11953,11 +11949,11 @@ mod tests { let retain_until = OffsetDateTime::now_utc() + Duration::from_secs(60 * 60 * 24 * 60); let mut user_defined = HashMap::new(); user_defined.insert( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(), ); user_defined.insert( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until.format(&time::format_description::well_known::Rfc3339).unwrap(), ); @@ -11998,11 +11994,11 @@ mod tests { restore_expires: Some(restore_expiry), user_defined: Arc::new(HashMap::from([ ( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(), ), ( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until.format(&time::format_description::well_known::Rfc3339).unwrap(), ), ])), @@ -12072,11 +12068,11 @@ mod tests { let retain_until = OffsetDateTime::now_utc() + Duration::from_secs(60 * 60 * 24 * 60); let mut user_defined = HashMap::new(); user_defined.insert( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(), ); user_defined.insert( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until.format(&time::format_description::well_known::Rfc3339).unwrap(), ); @@ -12100,11 +12096,11 @@ mod tests { let retain_until = OffsetDateTime::now_utc() + Duration::from_secs(60 * 60 * 24 * 60); let mut user_defined = HashMap::new(); user_defined.insert( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::GOVERNANCE.to_string(), ); user_defined.insert( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until.format(&time::format_description::well_known::Rfc3339).unwrap(), ); ObjectInfo { @@ -12153,11 +12149,11 @@ mod tests { let retain_until = OffsetDateTime::now_utc() + Duration::from_secs(60 * 60 * 24 * 60); let mut user_defined = HashMap::new(); user_defined.insert( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(), ); user_defined.insert( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until.format(&time::format_description::well_known::Rfc3339).unwrap(), ); let obj_info = ObjectInfo { @@ -12176,7 +12172,7 @@ mod tests { #[tokio::test] async fn test_check_object_lock_delete_blocks_replicated_legal_hold_version_purge() { let mut user_defined = HashMap::new(); - user_defined.insert(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), "ON".to_string()); + user_defined.insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string()); let obj_info = ObjectInfo { user_defined: Arc::new(user_defined), ..Default::default() @@ -14888,7 +14884,7 @@ mod tests { let object = "object.txt"; let mod_time = OffsetDateTime::from_unix_timestamp(1_717_171_717).expect("fixed timestamp should parse"); let mut user_defined = HashMap::new(); - user_defined.insert(AMZ_STORAGE_CLASS.to_string(), storageclass::STANDARD.to_string()); + user_defined.insert(metadata_keys::STORAGE_CLASS.to_string(), storageclass::STANDARD.to_string()); user_defined.insert(SUFFIX_COMPRESSION.to_string(), "zstd".to_string()); let mut eval_metadata = HashMap::new(); eval_metadata.insert("x-amz-meta-evaluated".to_string(), "yes".to_string()); @@ -14912,7 +14908,7 @@ mod tests { assert_eq!(written.etag.as_deref(), Some("preserved-etag")); assert_eq!(written.mod_time, Some(mod_time)); assert_eq!(written.user_defined.get("x-amz-meta-evaluated").map(String::as_str), Some("yes")); - assert!(!written.user_defined.contains_key(AMZ_STORAGE_CLASS)); + assert!(!written.user_defined.contains_key(metadata_keys::STORAGE_CLASS)); let info = set_disks .get_object_info(bucket, object, &opts) @@ -14921,7 +14917,7 @@ mod tests { assert_eq!(info.etag.as_deref(), Some("preserved-etag")); assert_eq!(info.mod_time, Some(mod_time)); assert_eq!(info.user_defined.get("x-amz-meta-evaluated").map(String::as_str), Some("yes")); - assert!(!info.user_defined.contains_key(AMZ_STORAGE_CLASS)); + assert!(!info.user_defined.contains_key(metadata_keys::STORAGE_CLASS)); } #[tokio::test] diff --git a/crates/ecstore/src/set_disk/ops/multipart.rs b/crates/ecstore/src/set_disk/ops/multipart.rs index 4539750b2..54234239e 100644 --- a/crates/ecstore/src/set_disk/ops/multipart.rs +++ b/crates/ecstore/src/set_disk/ops/multipart.rs @@ -29,23 +29,22 @@ use super::super::MetadataCacheInvalidationProbe; #[cfg(test)] use super::super::capacity_scope_from_disks; use super::super::{ - AMZ_STORAGE_CLASS, Arc, Bytes, CompletePart, Cursor, DATA_MOVEMENT_MULTIPART_PREFIX, DiskError, DiskStore, - EVENT_SET_DISK_MULTIPART, Error, FileInfo, GLOBAL_MIN_PART_SIZE, HashAlgorithm, HashMap, HashReader, HashSet, - HealChannelPriority, Instant, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, ListMultipartsInfo, ListPartsInfo, - MAX_PARTS_COUNT, MULTIPART_WRITE_QUORUM_RENAME_PART, MULTIPART_WRITE_QUORUM_UPLOAD_METADATA, - MULTIPART_WRITE_QUORUM_WRITER_SETUP, MultipartInfo, MultipartUploadResult, MultipartWriteQuorumContext, NamespaceLockFence, - OBJECT_OP_IGNORED_ERRS, ObjectInfo, ObjectLockDiagGuard, ObjectOptions, ObjectPartInfo, OffsetDateTime, PartInfo, - PutObjReader, RUSTFS_META_MULTIPART_BUCKET, RUSTFS_META_TMP_BUCKET, RUSTFS_MULTIPART_BUCKET_KEY, RUSTFS_MULTIPART_OBJECT_KEY, - Result, SLASH_SEPARATOR, SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_BUCKET_INCARNATION_ID, - SUFFIX_COMPRESSION_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK, SetDisks, - SmallWritePath, StorageError, Uuid, WriteLayout, check_object_lock_for_deletion_with_state, - classify_multipart_part_write_path, coding, complete_multipart_part_error, complete_multipart_part_error_result, - complete_part_checksum, completed_multipart_object_part, contains_key_str, create_bitrot_writer, debug, disk, error, - get_complete_multipart_md5, get_header_map, get_str, insert_str, is_err_object_not_found, is_err_version_not_found, - is_min_allowed_part_size, log_multipart_write_quorum_failure, parts_after_marker, path_join_buf, - record_compression_total_memory, reduce_read_quorum_errs, reduce_write_quorum_errs, remove_header_map, resolve_write_layout, - restore_commit_operation_id_from_metadata, should_persist_encryption_original_size, strip_internal_multipart_metadata, - to_object_err, warn, + Arc, Bytes, CompletePart, Cursor, DATA_MOVEMENT_MULTIPART_PREFIX, DiskError, DiskStore, EVENT_SET_DISK_MULTIPART, Error, + FileInfo, GLOBAL_MIN_PART_SIZE, HashAlgorithm, HashMap, HashReader, HashSet, HealChannelPriority, Instant, + LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, ListMultipartsInfo, ListPartsInfo, MAX_PARTS_COUNT, + MULTIPART_WRITE_QUORUM_RENAME_PART, MULTIPART_WRITE_QUORUM_UPLOAD_METADATA, MULTIPART_WRITE_QUORUM_WRITER_SETUP, + MultipartInfo, MultipartUploadResult, MultipartWriteQuorumContext, NamespaceLockFence, OBJECT_OP_IGNORED_ERRS, ObjectInfo, + ObjectLockDiagGuard, ObjectOptions, ObjectPartInfo, OffsetDateTime, PartInfo, PutObjReader, RUSTFS_META_MULTIPART_BUCKET, + RUSTFS_META_TMP_BUCKET, RUSTFS_MULTIPART_BUCKET_KEY, RUSTFS_MULTIPART_OBJECT_KEY, Result, SLASH_SEPARATOR, + SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_BUCKET_INCARNATION_ID, SUFFIX_COMPRESSION_SIZE, + SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK, SetDisks, SmallWritePath, StorageError, + Uuid, WriteLayout, check_object_lock_for_deletion_with_state, classify_multipart_part_write_path, coding, + complete_multipart_part_error, complete_multipart_part_error_result, complete_part_checksum, completed_multipart_object_part, + contains_key_str, create_bitrot_writer, debug, disk, error, get_complete_multipart_md5, get_header_map, get_str, insert_str, + is_err_object_not_found, is_err_version_not_found, is_min_allowed_part_size, log_multipart_write_quorum_failure, + parts_after_marker, path_join_buf, record_compression_total_memory, reduce_read_quorum_errs, reduce_write_quorum_errs, + remove_header_map, resolve_write_layout, restore_commit_operation_id_from_metadata, should_persist_encryption_original_size, + strip_internal_multipart_metadata, to_object_err, warn, }; use super::bitrot_self_verify::{BitrotSelfVerifyTarget, drop_failed_writer_disks, verify_written_bitrot_shards}; #[cfg(test)] @@ -81,6 +80,7 @@ use crate::storage_api_contracts::object::ObjectOperations; use futures::{StreamExt, stream}; #[cfg(test)] use http::HeaderMap; +use rustfs_filemeta::metadata_keys; use rustfs_rio::EtagResolvable; use rustfs_rio::TryGetIndex; #[cfg(test)] @@ -1924,7 +1924,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { // Extract storage class from metadata, default to STANDARD if not found let storage_class = fi .metadata - .get(AMZ_STORAGE_CLASS) + .get(metadata_keys::STORAGE_CLASS) .cloned() .unwrap_or_else(|| storageclass::STANDARD.to_string()); @@ -2123,10 +2123,10 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { user_defined.insert("etag".to_owned(), etag.clone()); } - if let Some(sc) = user_defined.get(AMZ_STORAGE_CLASS) + if let Some(sc) = user_defined.get(metadata_keys::STORAGE_CLASS) && sc == storageclass::STANDARD { - let _ = user_defined.remove(AMZ_STORAGE_CLASS); + let _ = user_defined.remove(metadata_keys::STORAGE_CLASS); } let WriteLayout { @@ -2138,7 +2138,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { self.pool_index, disks.len(), self.default_parity_count, - user_defined.get(AMZ_STORAGE_CLASS).map(String::as_str), + user_defined.get(metadata_keys::STORAGE_CLASS).map(String::as_str), opts.max_parity, )?; @@ -2180,10 +2180,10 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { // TODO(backlog): detect content-type from part data when header is missing } - if let Some(sc) = user_defined.get(AMZ_STORAGE_CLASS) + if let Some(sc) = user_defined.get(metadata_keys::STORAGE_CLASS) && sc == storageclass::STANDARD { - let _ = user_defined.remove(AMZ_STORAGE_CLASS); + let _ = user_defined.remove(metadata_keys::STORAGE_CLASS); } if let Some(checksum) = &opts.want_checksum { @@ -2418,7 +2418,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks { opts.replication_request || opts.delete_marker_replication_status() == ReplicationStatusType::Replica; if !authorized_inbound_replica { fi.metadata - .retain(|key, _| !key.eq_ignore_ascii_case(rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS)); + .retain(|key, _| !key.eq_ignore_ascii_case(metadata_keys::REPLICATION_STATUS)); for suffix in [ rustfs_utils::http::SUFFIX_REPLICA_STATUS, rustfs_utils::http::SUFFIX_REPLICA_TIMESTAMP, @@ -8837,8 +8837,7 @@ mod tests { rustfs_utils::http::SUFFIX_REPLICA_TIMESTAMP, "foreign-replica-time".to_string(), ); - create_replication_metadata - .insert(rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS.to_string(), "REPLICA".to_string()); + create_replication_metadata.insert(metadata_keys::REPLICATION_STATUS.to_string(), "REPLICA".to_string()); let (upload_id, parts) = stage_upload_with_create_opts( &set_disks, bucket, @@ -8909,7 +8908,7 @@ mod tests { completed .user_defined .iter() - .filter(|(key, _)| key.eq_ignore_ascii_case(rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS)) + .filter(|(key, _)| key.eq_ignore_ascii_case(metadata_keys::REPLICATION_STATUS)) .all(|(_, value)| value != "REPLICA") ); @@ -8965,7 +8964,7 @@ mod tests { rustfs_utils::http::SUFFIX_REPLICA_TIMESTAMP, "authorized-inbound-time".to_string(), ); - inbound_replica_metadata.insert(rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS.to_string(), "REPLICA".to_string()); + inbound_replica_metadata.insert(metadata_keys::REPLICATION_STATUS.to_string(), "REPLICA".to_string()); let (upload_id, parts) = stage_upload_with_create_opts( &set_disks, bucket, @@ -9006,7 +9005,7 @@ mod tests { inbound .user_defined .iter() - .find(|(key, _)| key.eq_ignore_ascii_case(rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS)) + .find(|(key, _)| key.eq_ignore_ascii_case(metadata_keys::REPLICATION_STATUS)) .map(|(_, value)| value.as_str()), Some("REPLICA") ); diff --git a/crates/ecstore/src/set_disk/ops/object.rs b/crates/ecstore/src/set_disk/ops/object.rs index f5a8dd6a6..d4dacd1c8 100644 --- a/crates/ecstore/src/set_disk/ops/object.rs +++ b/crates/ecstore/src/set_disk/ops/object.rs @@ -20,27 +20,27 @@ //! SetDisks core (io_primitives) via inherent calls. use crate::core::pools::DecommissionCapacityAdmission; +use rustfs_filemeta::metadata_keys; #[cfg(test)] use super::super::MetadataCacheInvalidationProbe; use super::super::{ - AMZ_OBJECT_TAGGING, AMZ_STORAGE_CLASS, Arc, AsyncWrite, AtomicU64, BufReader, Bytes, CACHE_CONTROL, CONTENT_DISPOSITION, - CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, CompletePart, Cursor, DeleteAccounting, DeleteOptions, DeletedObject, - DiskError, DiskStore, EVENT_SET_DISK_COMMIT_TAIL_SLOW, EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY, EVENT_SET_DISK_WRITE, - EXPIRES, Error, EventArgs, EventName, FastLockGuard, FileInfo, FileInfoVersions, - GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART, GET_OBJECT_PATH_BODY_CACHE, GET_OBJECT_PATH_CODEC_STREAMING, - GET_OBJECT_PATH_DIRECT_MEMORY, GET_OBJECT_PATH_EMPTY, GET_OBJECT_PATH_INLINE_DIRECT, GET_OBJECT_PATH_INTERNAL_META, - GET_OBJECT_PATH_LEGACY_DUPLEX, GET_OBJECT_PATH_REMOTE_TRANSITION, GET_OBJECT_PATH_SET_DISK, GET_STAGE_DECODE, GET_STAGE_EMIT, - GET_STAGE_INLINE_PREPARE, GET_STAGE_LOCK_ACQUIRE, GET_STAGE_METADATA, GET_STAGE_OBJECT_INFO, GET_STAGE_PATH_DECISION, - GET_STAGE_READER_SETUP, GenericError, GetCodecStreamingDecision, GetCodecStreamingFallbackReason, GetDirectMemoryDecision, - GetObjectReader, HTTPRangeSpec, HashAlgorithm, HashMap, HashReader, HashSet, HeaderMap, HealChannelPriority, InstanceContext, - Instant, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectApiError, ObjectInfo, ObjectKey, + AMZ_OBJECT_TAGGING, Arc, AsyncWrite, AtomicU64, BufReader, Bytes, CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, + CONTENT_LANGUAGE, CONTENT_TYPE, CompletePart, Cursor, DeleteAccounting, DeleteOptions, DeletedObject, DiskError, DiskStore, + EVENT_SET_DISK_COMMIT_TAIL_SLOW, EVENT_SET_DISK_PUT_OBJECT_STAGE_SUMMARY, EVENT_SET_DISK_WRITE, EXPIRES, Error, EventArgs, + EventName, FastLockGuard, FileInfo, FileInfoVersions, GET_CODEC_STREAMING_OBJECT_CLASS_PLAIN_SINGLE_PART, + GET_OBJECT_PATH_BODY_CACHE, GET_OBJECT_PATH_CODEC_STREAMING, GET_OBJECT_PATH_DIRECT_MEMORY, GET_OBJECT_PATH_EMPTY, + GET_OBJECT_PATH_INLINE_DIRECT, GET_OBJECT_PATH_INTERNAL_META, GET_OBJECT_PATH_LEGACY_DUPLEX, + GET_OBJECT_PATH_REMOTE_TRANSITION, GET_OBJECT_PATH_SET_DISK, GET_STAGE_DECODE, GET_STAGE_EMIT, GET_STAGE_INLINE_PREPARE, + GET_STAGE_LOCK_ACQUIRE, GET_STAGE_METADATA, GET_STAGE_OBJECT_INFO, GET_STAGE_PATH_DECISION, GET_STAGE_READER_SETUP, + GenericError, GetCodecStreamingDecision, GetCodecStreamingFallbackReason, GetDirectMemoryDecision, GetObjectReader, + HTTPRangeSpec, HashAlgorithm, HashMap, HashReader, HashSet, HeaderMap, HealChannelPriority, InstanceContext, Instant, + LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_SET_DISK, OBJECT_OP_IGNORED_ERRS, ObjectApiError, ObjectInfo, ObjectKey, ObjectLockConfigSnapshot, ObjectLockConfigState, ObjectOptions, ObjectReader, ObjectToDelete, OffsetDateTime, Ordering, Pin, PutObjReader, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, ReadPathPlan, ReaderImpl, ReplicateDecision, ReplicationObjectBridge, Result, SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS, SLASH_SEPARATOR, SUFFIX_ACTUAL_SIZE, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK, SetDisks, - SmallWritePath, StorageError, TRANSITION_COMPLETE, UpdateMetadataOpts, Uuid, WriteLayout, X_AMZ_OBJECT_LOCK_LEGAL_HOLD, - X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, X_AMZ_RESTORE, adaptive_duplex_buffer_size, + SmallWritePath, StorageError, TRANSITION_COMPLETE, UpdateMetadataOpts, Uuid, WriteLayout, adaptive_duplex_buffer_size, build_get_object_info, build_inline_bitrot_readers, build_inline_bitrot_readers_from_refs, can_try_inline_data_shards_direct, check_object_lock_delete, check_object_lock_for_deletion_with_state, check_object_lock_retention_update, classify_get_codec_streaming_object_class, classify_put_write_path, classify_storage_error, @@ -1849,9 +1849,9 @@ mod duration_metrics_tests { } fn is_restore_control_metadata(key: &str) -> bool { - key.eq_ignore_ascii_case(X_AMZ_RESTORE.as_str()) - || key.eq_ignore_ascii_case(rustfs_utils::http::headers::AMZ_RESTORE_EXPIRY_DAYS) - || key.eq_ignore_ascii_case(rustfs_utils::http::headers::AMZ_RESTORE_REQUEST_DATE) + key.eq_ignore_ascii_case(metadata_keys::RESTORE) + || key.eq_ignore_ascii_case(metadata_keys::RESTORE_EXPIRY_DAYS) + || key.eq_ignore_ascii_case(metadata_keys::RESTORE_REQUEST_DATE) || rustfs_utils::http::internal_key_strip_suffix_prefix(key, SUFFIX_RESTORE_OPERATION_ID) .is_some_and(|remainder| remainder.is_empty()) || rustfs_utils::http::internal_key_strip_suffix_prefix(key, SUFFIX_RESTORE_WORKER_LOCK) @@ -1879,6 +1879,43 @@ fn restore_metadata_update_preserves_protected_metadata( mod restore_metadata_update_tests { use super::*; + /// The restore keys in pre-`metadata_keys` xl.meta are exactly the ones + /// a restore metadata update may change; every other persisted key is + /// protected (backlog#1735 A3b). + #[test] + fn pre_module_xlmeta_restore_keys_are_the_restore_control_keys() { + let fi = rustfs_filemeta::FileMeta::load( + &rustfs_filemeta::test_data::create_pre_metadata_keys_xlmeta().expect("decode fixture hex"), + ) + .expect("load fixture xl.meta") + .into_fileinfo("bucket", "object", "0b1e5a3a-1735-4a3a-8000-00000000a3a0", false, false, false) + .expect("fixture version to FileInfo"); + + let mut control: Vec<&str> = fi + .metadata + .keys() + .map(String::as_str) + .filter(|key| is_restore_control_metadata(key)) + .collect(); + control.sort_unstable(); + assert_eq!( + control, + ["X-Amz-Restore-Expiry-Days", "X-Amz-Restore-Request-Date", "x-amz-restore"], + "restore control keys in the pre-module bytes" + ); + for key in [ + metadata_keys::OBJECT_LOCK_LEGAL_HOLD, + metadata_keys::OBJECT_LOCK_MODE, + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE, + metadata_keys::SERVER_SIDE_ENCRYPTION, + metadata_keys::STORAGE_CLASS, + metadata_keys::REPLICATION_STATUS, + ] { + assert!(fi.metadata.contains_key(key), "{key:?}"); + assert!(!is_restore_control_metadata(key), "{key:?} must stay protected"); + } + } + #[test] fn restore_metadata_update_cannot_change_retention_or_user_metadata() { let mut existing = HashMap::from([ @@ -1887,7 +1924,7 @@ mod restore_metadata_update_tests { ("x-amz-object-lock-mode".to_string(), "COMPLIANCE".to_string()), ]); let mut replacement = existing.clone(); - replacement.insert(X_AMZ_RESTORE.as_str().to_string(), "ongoing-request=\"true\"".to_string()); + replacement.insert(metadata_keys::RESTORE.to_string(), "ongoing-request=\"true\"".to_string()); rustfs_utils::http::metadata_compat::insert_str( &mut replacement, SUFFIX_RESTORE_OPERATION_ID, @@ -1907,9 +1944,9 @@ mod restore_metadata_update_tests { replacement.insert("x-amz-meta-owner".to_string(), "mallory".to_string()); assert!(!restore_metadata_update_preserves_protected_metadata(&existing, &replacement)); - existing.insert(X_AMZ_RESTORE.as_str().to_string(), "ongoing-request=\"false\"".to_string()); + existing.insert(metadata_keys::RESTORE.to_string(), "ongoing-request=\"false\"".to_string()); replacement.clone_from(&existing); - replacement.remove(X_AMZ_RESTORE.as_str()); + replacement.remove(metadata_keys::RESTORE); assert!(restore_metadata_update_preserves_protected_metadata(&existing, &replacement)); } } @@ -3179,9 +3216,7 @@ pub(in crate::set_disk) fn merge_replication_metadata_lww( existing: &HashMap, opts: &ObjectOptions, ) -> bool { - use rustfs_utils::http::headers::{ - AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, AMZ_OBJECT_TAGGING, - }; + use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING; use rustfs_utils::http::metadata_compat::{ SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, SUFFIX_TAGGING_TIMESTAMP, get_str, remove_str, @@ -3193,12 +3228,12 @@ pub(in crate::set_disk) fn merge_replication_metadata_lww( ( opts.replication_retention_timestamp, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, - &[AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER], + &[metadata_keys::OBJECT_LOCK_MODE, metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE], ), ( opts.replication_legalhold_timestamp, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, - &[AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER], + &[metadata_keys::OBJECT_LOCK_LEGAL_HOLD], ), ]; @@ -3506,7 +3541,7 @@ impl SetDisks { self.pool_index, disks.len(), self.default_parity_count, - user_defined.get(AMZ_STORAGE_CLASS).map(String::as_str), + user_defined.get(metadata_keys::STORAGE_CLASS).map(String::as_str), opts.max_parity, )?; @@ -3783,10 +3818,10 @@ impl SetDisks { fi.checksum = Some(content_hash.to_bytes(&[])); } - if let Some(sc) = user_defined.get(AMZ_STORAGE_CLASS) + if let Some(sc) = user_defined.get(metadata_keys::STORAGE_CLASS) && sc == storageclass::STANDARD { - let _ = user_defined.remove(AMZ_STORAGE_CLASS); + let _ = user_defined.remove(metadata_keys::STORAGE_CLASS); } let mod_time = opts.mod_time; @@ -7631,7 +7666,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { && src_info .user_defined .keys() - .any(|key| key.eq_ignore_ascii_case(X_AMZ_RESTORE.as_str())) + .any(|key| key.eq_ignore_ascii_case(metadata_keys::RESTORE)) && restore_metadata_update_preserves_protected_metadata(&fi.metadata, src_info.user_defined.as_ref()); if let Some(dst_version_id) = dst_opts.version_id.as_deref() && !is_meta_bucketname(dst_bucket) @@ -9341,9 +9376,9 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { CONTENT_DISPOSITION, CACHE_CONTROL, EXPIRES, - X_AMZ_OBJECT_LOCK_MODE.as_str(), - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str(), - X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str(), + metadata_keys::OBJECT_LOCK_MODE, + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE, + metadata_keys::OBJECT_LOCK_LEGAL_HOLD, ] { if let Some(value) = fi.metadata.lookup(header).filter(|value| !value.is_empty()) { transition_meta.insert(header.to_ascii_lowercase(), value.to_string()); @@ -9791,7 +9826,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks { } let mut restore_commit_metadata = if let Some(expected_operation_id) = expected_operation_id { let mut metadata = HashMap::new(); - metadata.insert(X_AMZ_RESTORE.as_str().to_string(), "ongoing-request=\"false\"".to_string()); + metadata.insert(metadata_keys::RESTORE.to_string(), "ongoing-request=\"false\"".to_string()); rustfs_utils::http::metadata_compat::insert_str( &mut metadata, SUFFIX_RESTORE_OPERATION_ID, @@ -10778,9 +10813,7 @@ mod replication_lww_tests { use super::hermetic_set_disks_support::hermetic_set_disks_isolated as hermetic_set_disks; use super::*; - use rustfs_utils::http::headers::{ - AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, AMZ_OBJECT_TAGGING, - }; + use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING; use rustfs_utils::http::{ SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, SUFFIX_TAGGING_TIMESTAMP, get_str, insert_str, @@ -10958,8 +10991,11 @@ mod replication_lww_tests { let mut inbound = HashMap::new(); inbound.insert(AMZ_OBJECT_TAGGING.to_string(), "site=remote".to_string()); insert_str(&mut inbound, SUFFIX_TAGGING_TIMESTAMP, T_OLD.to_string()); - inbound.insert(AMZ_OBJECT_LOCK_MODE_LOWER.to_string(), "COMPLIANCE".to_string()); - inbound.insert(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER.to_string(), "2028-01-01T00:00:00Z".to_string()); + inbound.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()); + inbound.insert( + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), + "2028-01-01T00:00:00Z".to_string(), + ); insert_str(&mut inbound, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, T_NEW.to_string()); let opts = ObjectOptions { replication_request: true, @@ -10972,7 +11008,7 @@ mod replication_lww_tests { let info = version_info(&set_disks, bucket, object, &version_id).await; assert_eq!(info.user_tags.as_str(), "site=local", "the stale tagging category must keep local values"); assert_eq!( - info.user_defined.get(AMZ_OBJECT_LOCK_MODE_LOWER).map(String::as_str), + info.user_defined.get(metadata_keys::OBJECT_LOCK_MODE).map(String::as_str), Some("COMPLIANCE"), "the newer retention category must be applied in the same write" ); @@ -10992,12 +11028,12 @@ mod replication_lww_tests { // LWW-reachable divergence is a stale inbound ON resurrecting a hold // that was released more recently on this site.) let mut local = HashMap::new(); - local.insert(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER.to_string(), "OFF".to_string()); + local.insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "OFF".to_string()); insert_str(&mut local, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, T_LOCAL.to_string()); put_version(&set_disks, bucket, object, &version_id, &versioned_opts(&version_id, local)).await; let mut inbound = HashMap::new(); - inbound.insert(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER.to_string(), "ON".to_string()); + inbound.insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string()); insert_str(&mut inbound, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, T_OLD.to_string()); let opts = ObjectOptions { replication_request: true, @@ -11008,7 +11044,9 @@ mod replication_lww_tests { let info = version_info(&set_disks, bucket, object, &version_id).await; assert_eq!( - info.user_defined.get(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER).map(String::as_str), + info.user_defined + .get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) + .map(String::as_str), Some("OFF"), "a stale inbound legal hold must not resurrect a hold released more recently" ); @@ -11068,8 +11106,11 @@ mod replication_lww_tests { // path's eval_metadata stomped the metadata key with receiver-now // (simulated by T_NEW here). let mut inbound = HashMap::new(); - inbound.insert(AMZ_OBJECT_LOCK_MODE_LOWER.to_string(), "GOVERNANCE".to_string()); - inbound.insert(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER.to_string(), "2028-01-01T00:00:00Z".to_string()); + inbound.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "GOVERNANCE".to_string()); + inbound.insert( + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), + "2028-01-01T00:00:00Z".to_string(), + ); insert_str(&mut inbound, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, T_NEW.to_string()); let opts = ObjectOptions { replication_request: true, @@ -11084,7 +11125,10 @@ mod replication_lww_tests { Some(T_LOCAL), "the stored category timestamp must be the source-authored time, not the receiver's clock" ); - assert_eq!(info.user_defined.get(AMZ_OBJECT_LOCK_MODE_LOWER).map(String::as_str), Some("GOVERNANCE")); + assert_eq!( + info.user_defined.get(metadata_keys::OBJECT_LOCK_MODE).map(String::as_str), + Some("GOVERNANCE") + ); } #[tokio::test] @@ -11096,7 +11140,7 @@ mod replication_lww_tests { make_bucket(&disk_stores, bucket).await; let mut inbound = HashMap::new(); - inbound.insert(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER.to_string(), "OFF".to_string()); + inbound.insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "OFF".to_string()); insert_str(&mut inbound, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, T_OLD.to_string()); let mut evaluated = inbound.clone(); insert_str(&mut evaluated, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, T_NEW.to_string()); @@ -11150,10 +11194,13 @@ mod replication_lww_tests { /// plus an active COMPLIANCE retention (no retention timestamp). async fn seed_locked_version(set_disks: &Arc, bucket: &str, object: &str, version_id: &str, hold_timestamp: &str) { let mut local = HashMap::new(); - local.insert(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER.to_string(), "ON".to_string()); + local.insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string()); insert_str(&mut local, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, hold_timestamp.to_string()); - local.insert(AMZ_OBJECT_LOCK_MODE_LOWER.to_string(), "COMPLIANCE".to_string()); - local.insert(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER.to_string(), "2099-01-01T00:00:00Z".to_string()); + local.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()); + local.insert( + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), + "2099-01-01T00:00:00Z".to_string(), + ); put_version(set_disks, bucket, object, version_id, &versioned_opts(version_id, local)).await; } @@ -11162,10 +11209,13 @@ mod replication_lww_tests { /// category the source version has. fn inbound_legal_hold_release_opts(version_id: &str, timestamp: &str) -> ObjectOptions { let mut inbound = HashMap::new(); - inbound.insert(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER.to_string(), "OFF".to_string()); + inbound.insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "OFF".to_string()); insert_str(&mut inbound, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP, timestamp.to_string()); - inbound.insert(AMZ_OBJECT_LOCK_MODE_LOWER.to_string(), "COMPLIANCE".to_string()); - inbound.insert(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER.to_string(), "2099-01-01T00:00:00Z".to_string()); + inbound.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()); + inbound.insert( + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), + "2099-01-01T00:00:00Z".to_string(), + ); insert_str(&mut inbound, SUFFIX_OBJECTLOCK_RETENTION_TIMESTAMP, T_OLD.to_string()); ObjectOptions { replication_request: true, @@ -11199,13 +11249,15 @@ mod replication_lww_tests { let info = version_info(&set_disks, bucket, object, &version_id).await; assert_eq!( - info.user_defined.get(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER).map(String::as_str), + info.user_defined + .get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) + .map(String::as_str), Some("OFF"), "a newer source-side legal hold release must be applied to the locked replica" ); assert_eq!(get_str(&info.user_defined, SUFFIX_OBJECTLOCK_LEGALHOLD_TIMESTAMP).as_deref(), Some(T_NEW)); assert_eq!( - info.user_defined.get(AMZ_OBJECT_LOCK_MODE_LOWER).map(String::as_str), + info.user_defined.get(metadata_keys::OBJECT_LOCK_MODE).map(String::as_str), Some("COMPLIANCE"), "the untouched retention category must survive the write" ); @@ -11233,7 +11285,9 @@ mod replication_lww_tests { let info = version_info(&set_disks, bucket, object, &version_id).await; assert_eq!( - info.user_defined.get(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER).map(String::as_str), + info.user_defined + .get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) + .map(String::as_str), Some("ON"), "a stale inbound release must not lift a hold applied more recently on this site" ); @@ -11259,8 +11313,11 @@ mod replication_lww_tests { let mut inbound = HashMap::new(); inbound.insert(AMZ_OBJECT_TAGGING.to_string(), "k=v".to_string()); insert_str(&mut inbound, SUFFIX_TAGGING_TIMESTAMP, T_NEW.to_string()); - inbound.insert(AMZ_OBJECT_LOCK_MODE_LOWER.to_string(), "COMPLIANCE".to_string()); - inbound.insert(AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER.to_string(), "2099-01-01T00:00:00Z".to_string()); + inbound.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()); + inbound.insert( + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), + "2099-01-01T00:00:00Z".to_string(), + ); let opts = ObjectOptions { replication_request: true, replication_tagging_timestamp: Some(parse_ts(T_NEW)), @@ -11276,7 +11333,12 @@ mod replication_lww_tests { assert!(matches!(err, StorageError::PrefixAccessDenied(_, _)), "unexpected error: {err}"); let info = version_info(&set_disks, bucket, object, &version_id).await; - assert_eq!(info.user_defined.get(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER).map(String::as_str), Some("ON")); + assert_eq!( + info.user_defined + .get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) + .map(String::as_str), + Some("ON") + ); } fn default_retention_snapshot(mode: &'static str) -> Arc { @@ -11314,7 +11376,7 @@ mod replication_lww_tests { seed_local_tagged_version(&set_disks, bucket, object, &version_id).await; let seeded = version_info(&set_disks, bucket, object, &version_id).await; assert!( - !seeded.user_defined.contains_key(AMZ_OBJECT_LOCK_MODE_LOWER), + !seeded.user_defined.contains_key(metadata_keys::OBJECT_LOCK_MODE), "the seeded version must be protected by the bucket default only" ); @@ -11359,7 +11421,7 @@ mod replication_lww_tests { let version_id = Uuid::new_v4().to_string(); make_bucket(&disk_stores, bucket).await; let mut local = HashMap::new(); - local.insert(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER.to_string(), "MAYBE".to_string()); + local.insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "MAYBE".to_string()); put_version(&set_disks, bucket, object, &version_id, &versioned_opts(&version_id, local)).await; let mut reader = PutObjReader::from_vec(b"lww-body".to_vec()); @@ -11370,7 +11432,12 @@ mod replication_lww_tests { assert!(!matches!(err, StorageError::PrefixAccessDenied(_, _)), "unexpected error: {err}"); let info = version_info(&set_disks, bucket, object, &version_id).await; - assert_eq!(info.user_defined.get(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER).map(String::as_str), Some("MAYBE")); + assert_eq!( + info.user_defined + .get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) + .map(String::as_str), + Some("MAYBE") + ); } /// The bypass is scoped to authorized replication writes: the same @@ -11396,7 +11463,12 @@ mod replication_lww_tests { assert!(matches!(err, StorageError::PrefixAccessDenied(_, _)), "unexpected error: {err}"); let info = version_info(&set_disks, bucket, object, &version_id).await; - assert_eq!(info.user_defined.get(AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER).map(String::as_str), Some("ON")); + assert_eq!( + info.user_defined + .get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) + .map(String::as_str), + Some("ON") + ); } } @@ -13328,7 +13400,10 @@ mod transition_commit_failure_tests { rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_WORKER_LOCK, rustfs_utils::http::metadata_compat::RESTORE_WORKER_LOCK_PROTOCOL_V1.to_string(), ); - metadata.insert(s3s::header::X_AMZ_RESTORE.as_str().to_string(), format!("ongoing-request=\"{ongoing}\"")); + metadata.insert( + rustfs_filemeta::metadata_keys::RESTORE.to_string(), + format!("ongoing-request=\"{ongoing}\""), + ); metadata } @@ -13521,7 +13596,7 @@ mod transition_commit_failure_tests { .await .expect("failed restore cleanup should leave the transitioned object readable"); assert!( - !cleaned.user_defined.contains_key(s3s::header::X_AMZ_RESTORE.as_str()), + !cleaned.user_defined.contains_key(rustfs_filemeta::metadata_keys::RESTORE), "{point:?}: every post-snapshot failure must clean the public ongoing marker" ); assert!( @@ -13597,7 +13672,7 @@ mod transition_commit_failure_tests { ); let restore_header = restored .user_defined - .get(s3s::header::X_AMZ_RESTORE.as_str()) + .get(rustfs_filemeta::metadata_keys::RESTORE) .expect("successful multipart restore must persist restore status"); let restore_status = parse_restore_obj_status(restore_header).expect("successful restore status must be valid"); assert!(!restore_status.on_going(), "successful multipart restore must not remain in progress"); @@ -14083,7 +14158,7 @@ mod transition_commit_failure_tests { } return; } - assert!(!cleaned.user_defined.contains_key(s3s::header::X_AMZ_RESTORE.as_str())); + assert!(!cleaned.user_defined.contains_key(rustfs_filemeta::metadata_keys::RESTORE)); assert!( rustfs_utils::http::get_str(cleaned.user_defined.as_ref(), rustfs_utils::http::SUFFIX_RESTORE_OPERATION_ID,) .is_none() @@ -14189,7 +14264,7 @@ mod transition_commit_failure_tests { assert_eq!(restored.version_id, Some(version_id), "restore must preserve the selected {case} version"); let restore_header = restored .user_defined - .get(s3s::header::X_AMZ_RESTORE.as_str()) + .get(rustfs_filemeta::metadata_keys::RESTORE) .expect("restored version must carry its completed restore status"); let restore_status = parse_restore_obj_status(restore_header).expect("restore status must be valid"); assert!(!restore_status.on_going(), "restored {case} version must not remain in progress"); @@ -14849,7 +14924,7 @@ mod transition_commit_failure_tests { assert!( current_operation_b .user_defined - .contains_key(s3s::header::X_AMZ_RESTORE.as_str()), + .contains_key(rustfs_filemeta::metadata_keys::RESTORE), "stale cleanup for operation A must not remove operation B's restore header" ); assert_eq!( @@ -14884,7 +14959,7 @@ mod transition_commit_failure_tests { .await .expect("cleaned object metadata should remain readable"); assert!( - !cleaned.user_defined.contains_key(s3s::header::X_AMZ_RESTORE.as_str()), + !cleaned.user_defined.contains_key(rustfs_filemeta::metadata_keys::RESTORE), "matching cleanup must remove the restore header" ); assert!( @@ -14953,7 +15028,7 @@ mod transition_commit_failure_tests { let restore_status = parse_restore_obj_status( current .user_defined - .get(s3s::header::X_AMZ_RESTORE.as_str()) + .get(rustfs_filemeta::metadata_keys::RESTORE) .expect("restore header must remain pending"), ) .expect("restore header should remain parseable"); @@ -15017,7 +15092,7 @@ mod transition_commit_failure_tests { .await .expect("restore metadata should remain readable"); assert!( - current.user_defined.contains_key(s3s::header::X_AMZ_RESTORE.as_str()), + current.user_defined.contains_key(rustfs_filemeta::metadata_keys::RESTORE), "lost no_lock cleanup must not remove the restore header" ); assert_eq!( @@ -15235,7 +15310,7 @@ mod transition_commit_failure_tests { parse_restore_obj_status( current .user_defined - .get(s3s::header::X_AMZ_RESTORE.as_str()) + .get(rustfs_filemeta::metadata_keys::RESTORE) .expect("operation B restore header should remain pending"), ) .expect("operation B restore header should parse") @@ -15903,7 +15978,7 @@ mod transition_upload_integrity_tests { rustfs_filemeta::parse_restore_obj_status( current .user_defined - .get(s3s::header::X_AMZ_RESTORE.as_str()) + .get(rustfs_filemeta::metadata_keys::RESTORE) .expect("pending restore header should remain"), ) .expect("restore header should parse") @@ -17995,10 +18070,14 @@ mod heterogeneous_pool_put_tests { let bucket = "put-cleanup-receipt-gate-off"; let object = "object.bin"; make_bucket(&disk_stores, bucket).await; + let opts = ObjectOptions { + write_completion: WriteCompletion::TailDrained, + ..Default::default() + }; let mut first_reader = PutObjReader::from_vec(large_payload(0x41)); set_disks - .put_object(bucket, object, &mut first_reader, &ObjectOptions::default()) + .put_object(bucket, object, &mut first_reader, &opts) .await .expect("default-gate first PUT should commit"); let old_dir = current_data_dir(&disk_stores[0], bucket, object).await; @@ -18006,7 +18085,7 @@ mod heterogeneous_pool_put_tests { let _fault = cleanup_fault_injection::fail_cleanup_on(object, &[0, 1, 2, 3]); let mut second_reader = PutObjReader::from_vec(large_payload(0x42)); set_disks - .put_object(bucket, object, &mut second_reader, &ObjectOptions::default()) + .put_object(bucket, object, &mut second_reader, &opts) .await .expect("default-gate overwrite should commit"); @@ -19944,26 +20023,26 @@ mod put_object_tmp_cleanup_tests { "compliance", HashMap::from([ ( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(), ), - (X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), retain_until.clone()), + (metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until.clone()), ]), ), ( "governance", HashMap::from([ ( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::GOVERNANCE.to_string(), ), - (X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), retain_until), + (metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until), ]), ), ( "legal-hold", HashMap::from([( - X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), + metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), s3s::dto::ObjectLockLegalHoldStatus::ON.to_string(), )]), ), @@ -20264,9 +20343,9 @@ mod put_object_tmp_cleanup_tests { object, &ObjectOptions { eval_metadata: Some(HashMap::from([ - (X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), String::new()), - (X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), String::new()), - (X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), String::new()), + (metadata_keys::OBJECT_LOCK_MODE.to_string(), String::new()), + (metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), String::new()), + (metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), String::new()), ])), ..version_opts.clone() }, @@ -20333,7 +20412,7 @@ mod put_object_tmp_cleanup_tests { version_id: Some(destination_version.clone()), versioned: true, eval_metadata: Some(HashMap::from([( - X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), + metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), s3s::dto::ObjectLockLegalHoldStatus::ON.to_string(), )])), ..Default::default() @@ -21294,11 +21373,11 @@ mod delete_objects_lock_gating_tests { let retain_until = OffsetDateTime::now_utc() + Duration::from_secs(60 * 60 * 24 * 30); let mut user_defined = HashMap::new(); user_defined.insert( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(), ); user_defined.insert( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until .format(&time::format_description::well_known::Rfc3339) .expect("retain-until date should format"), diff --git a/crates/ecstore/src/set_disk/replication.rs b/crates/ecstore/src/set_disk/replication.rs index c7d3bff4e..8964a9420 100644 --- a/crates/ecstore/src/set_disk/replication.rs +++ b/crates/ecstore/src/set_disk/replication.rs @@ -14,13 +14,12 @@ use super::{ Error, FileInfo, NamespaceLockFence, ObjectInfo, ObjectOptions, OffsetDateTime, Result, SetDisks, StorageError, - UpdateMetadataOpts, Uuid, X_AMZ_RESTORE, get_raw_etag, restore_operation_id_from_metadata, + UpdateMetadataOpts, Uuid, get_raw_etag, restore_operation_id_from_metadata, }; use crate::bucket::lifecycle::lifecycle; use crate::core::pools::DecommissionCapacityAdmission; -use rustfs_filemeta::RestoreStatusOps; -use rustfs_utils::http::headers::{AMZ_RESTORE_EXPIRY_DAYS, AMZ_RESTORE_REQUEST_DATE}; -use s3s::dto::{RestoreStatus, Timestamp}; +use rustfs_filemeta::metadata_keys; +use rustfs_filemeta::{RestoreStatus, RestoreStatusOps}; #[cfg(all(test, feature = "test-util"))] use std::sync::Arc; @@ -213,10 +212,10 @@ impl SetDisks { let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), opts.transition.restore_request.days.unwrap_or(1)); fi.metadata.insert( - X_AMZ_RESTORE.as_str().to_string(), + metadata_keys::RESTORE.to_string(), RestoreStatus { is_restore_in_progress: Some(false), - restore_expiry_date: Some(Timestamp::from(restore_expiry)), + restore_expiry_date: Some(restore_expiry), } .to_string(), ); @@ -317,9 +316,9 @@ impl SetDisks { return Ok(()); } ensure_restore_metadata_lock_held(bucket, object, opts, "restore_cleanup_metadata")?; - fi.metadata.remove(X_AMZ_RESTORE.as_str()); - fi.metadata.remove(AMZ_RESTORE_EXPIRY_DAYS); - fi.metadata.remove(AMZ_RESTORE_REQUEST_DATE); + fi.metadata.remove(metadata_keys::RESTORE); + fi.metadata.remove(metadata_keys::RESTORE_EXPIRY_DAYS); + fi.metadata.remove(metadata_keys::RESTORE_REQUEST_DATE); rustfs_utils::http::metadata_compat::remove_str( &mut fi.metadata, rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID, diff --git a/crates/ecstore/src/set_disk/transition_matrix_tests.rs b/crates/ecstore/src/set_disk/transition_matrix_tests.rs index 9b5912ce7..bc14f95df 100644 --- a/crates/ecstore/src/set_disk/transition_matrix_tests.rs +++ b/crates/ecstore/src/set_disk/transition_matrix_tests.rs @@ -145,7 +145,7 @@ async fn transition_and_restore_reclaim_prior_metadata_generations() { let restore_status = parse_restore_obj_status( restored_info .user_defined - .get(s3s::header::X_AMZ_RESTORE.as_str()) + .get(rustfs_filemeta::metadata_keys::RESTORE) .expect("completed restore header should be present"), ) .expect("completed restore header should parse"); diff --git a/crates/ecstore/src/store/init.rs b/crates/ecstore/src/store/init.rs index 6264b4a3a..3022eec30 100644 --- a/crates/ecstore/src/store/init.rs +++ b/crates/ecstore/src/store/init.rs @@ -5937,7 +5937,7 @@ mod tests { metadata_acknowledged_target, &ObjectOptions { eval_metadata: Some(HashMap::from([( - s3s::header::X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), + rustfs_filemeta::metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), s3s::dto::ObjectLockLegalHoldStatus::OFF.to_string(), )])), ..target_version_opts.clone() @@ -5966,7 +5966,7 @@ mod tests { assert_eq!( metadata_acknowledged .user_defined - .get(s3s::header::X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()) + .get(rustfs_filemeta::metadata_keys::OBJECT_LOCK_LEGAL_HOLD) .map(String::as_str), Some("OFF") ); @@ -5996,9 +5996,9 @@ mod tests { ("governance-target.bin", s3s::dto::ObjectLockRetentionMode::GOVERNANCE), ] { let retained_metadata = HashMap::from([ - (s3s::header::X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), mode.to_string()), + (rustfs_filemeta::metadata_keys::OBJECT_LOCK_MODE.to_string(), mode.to_string()), ( - s3s::header::X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + rustfs_filemeta::metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until.clone(), ), ]); @@ -12606,6 +12606,24 @@ mod tests { }) .await .expect("tier free-version recovery should complete"); + wait_for_expiry_workers_idle(&store).await; + } + + /// Unlocked poll for exact metadata absence while asynchronous free-version + /// cleanup removes the per-disk copies. Mid-cleanup, fewer than a read + /// quorum of disks may still hold the record, so that transient result + /// means "not converged yet"; every other error still fails the test. + #[cfg(feature = "test-util")] + async fn exact_metadata_absent_during_cleanup(store: &crate::store::ECStore, bucket: &str, object: &str) -> bool { + match store.pools[0] + .get_disks_by_key(object) + .load_file_info_versions_exact(bucket, object) + .await + { + Ok(metadata) => metadata.is_none(), + Err(StorageError::InsufficientReadQuorum(_, _)) => false, + Err(error) => panic!("{object} cleanup metadata should remain readable: {error:?}"), + } } #[cfg(feature = "test-util")] @@ -12702,7 +12720,7 @@ mod tests { .await .expect("restored transitioned source should remain readable"); assert!( - restored.user_defined.contains_key(s3s::header::X_AMZ_RESTORE.as_str()), + restored.user_defined.contains_key(rustfs_filemeta::metadata_keys::RESTORE), "restore completion metadata must be present before the delete regression" ); } @@ -15807,20 +15825,7 @@ mod tests { ); tokio::time::timeout(Duration::from_secs(30), async { loop { - let metadata_absent = { - // Synchronize with cleanup so the snapshot cannot span per-disk marker removal. - let mut read_opts = ObjectOptions::default(); - let _guards = store - .acquire_all_physical_object_read_locks("batch_transitioned_delete_test", bucket, causal, &mut read_opts) - .await - .expect("causal batch cleanup observation should acquire object read locks"); - store.pools[0] - .get_disks_by_key(causal) - .load_file_info_versions_exact(bucket, causal) - .await - .expect("causal batch cleanup metadata should remain readable") - .is_none() - }; + let metadata_absent = exact_metadata_absent_during_cleanup(&store, bucket, causal).await; if metadata_absent && backend.remove_versions().await.len() >= 2 { return; } @@ -15899,24 +15904,7 @@ mod tests { ); tokio::time::timeout(Duration::from_secs(30), async { loop { - let metadata_absent = { - let mut read_opts = ObjectOptions::default(); - let _guards = store - .acquire_all_physical_object_read_locks( - "batch_transitioned_delete_test", - bucket, - versioned_causal, - &mut read_opts, - ) - .await - .expect("versioned causal batch cleanup observation should acquire object read locks"); - store.pools[0] - .get_disks_by_key(versioned_causal) - .load_file_info_versions_exact(bucket, versioned_causal) - .await - .expect("versioned causal batch cleanup metadata should remain readable") - .is_none() - }; + let metadata_absent = exact_metadata_absent_during_cleanup(&store, bucket, versioned_causal).await; if metadata_absent && backend.remove_versions().await.len() == 3 { return; } @@ -17067,11 +17055,11 @@ mod tests { &ObjectOptions { user_defined: HashMap::from([ ( - s3s::header::X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + rustfs_filemeta::metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(), ), ( - s3s::header::X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + rustfs_filemeta::metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), "2099-01-01T00:00:00Z".to_string(), ), ]), diff --git a/crates/ecstore/src/store/object.rs b/crates/ecstore/src/store/object.rs index c229cf710..c5fbe2d26 100644 --- a/crates/ecstore/src/store/object.rs +++ b/crates/ecstore/src/store/object.rs @@ -1687,6 +1687,15 @@ fn should_create_delete_marker_for_missing_object(opts: &ObjectOptions) -> bool (opts.versioned || opts.version_suspended) && opts.version_id.is_none() && !opts.delete_marker && !opts.data_movement } +fn latest_versioned_delete_creates_distinct_marker(opts: &ObjectOptions) -> bool { + opts.versioned + && !opts.version_suspended + && opts.version_id.is_none() + && !opts.delete_marker + && !opts.data_movement + && !opts.replication_request +} + #[cfg(any(test, feature = "test-util"))] struct DeleteAfterObjectLockSnapshotBarrierState { bucket: String, @@ -4894,7 +4903,8 @@ impl ECStore { return Ok(ObjectInfo::default()); } - let creates_latest_marker = should_create_delete_marker_for_missing_object(&opts); + let creates_latest_marker = latest_versioned_delete_creates_distinct_marker(&opts) + || (opts.version_suspended && should_create_delete_marker_for_missing_object(&opts)); let mut gopts = delete_pool_lookup_opts(&opts, true); if creates_latest_marker { // An unwritable source still owns its current version. Hiding it @@ -7815,6 +7825,57 @@ mod tests { assert!(matches!(err, Error::MethodNotAllowed)); } + #[tokio::test] + async fn latest_versioned_delete_creates_a_new_marker_after_a_current_marker() { + let ctx = Arc::new(crate::runtime::instance::InstanceContext::new()); + let (_dirs, set_disks) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await; + let store = Arc::new(new_prepared_reader_test_store_with_ctx(&[set_disks], ctx).await); + crate::bucket::metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await; + let bucket = "versioned-delete-marker-identity"; + let object = "object.bin"; + let opts = ObjectOptions { + versioned: true, + ..Default::default() + }; + + store + .make_bucket(bucket, &MakeBucketOptions::default()) + .await + .expect("bucket should be created"); + + let first = store + .handle_delete_object(bucket, object, opts.clone()) + .await + .expect("first versioned delete should create a marker"); + let second = store + .handle_delete_object(bucket, object, opts) + .await + .expect("second versioned delete should create a marker"); + + assert!(first.delete_marker); + assert!(second.delete_marker); + assert_ne!( + first.version_id, second.version_id, + "each latest versioned delete must persist a distinct delete marker" + ); + + let versions = store.pools[0] + .get_disks_by_key(object) + .load_file_info_versions_exact(bucket, object) + .await + .expect("delete-marker metadata should decode") + .expect("delete-marker metadata should be persisted"); + let marker_ids = versions + .versions + .iter() + .filter(|version| version.deleted) + .filter_map(|version| version.version_id) + .collect::>(); + assert!(marker_ids.contains(&first.version_id.expect("first marker should have a version id"))); + assert!(marker_ids.contains(&second.version_id.expect("second marker should have a version id"))); + assert_eq!(marker_ids.len(), 2, "both delete markers must remain durable versions"); + } + #[tokio::test] async fn versioned_delete_quorum_failure_rolls_back_and_retries_require_quorum() { for marker_copies in [0, 2, 4] { @@ -7941,6 +8002,39 @@ mod tests { } } + #[test] + fn latest_versioned_delete_marker_creation_excludes_specialized_deletes() { + assert!(latest_versioned_delete_creates_distinct_marker(&ObjectOptions { + versioned: true, + ..Default::default() + })); + assert!(!latest_versioned_delete_creates_distinct_marker(&ObjectOptions { + versioned: true, + version_id: Some(Uuid::new_v4().to_string()), + ..Default::default() + })); + assert!(!latest_versioned_delete_creates_distinct_marker(&ObjectOptions { + versioned: true, + delete_marker: true, + ..Default::default() + })); + assert!(!latest_versioned_delete_creates_distinct_marker(&ObjectOptions { + versioned: true, + data_movement: true, + ..Default::default() + })); + assert!(!latest_versioned_delete_creates_distinct_marker(&ObjectOptions { + versioned: true, + replication_request: true, + ..Default::default() + })); + assert!(!latest_versioned_delete_creates_distinct_marker(&ObjectOptions { + versioned: true, + version_suspended: true, + ..Default::default() + })); + } + #[test] fn should_create_delete_marker_for_missing_object_allows_latest_versioned_delete() { let opts = ObjectOptions { diff --git a/crates/ecstore/src/store/utils.rs b/crates/ecstore/src/store/utils.rs index a39bca8f3..cfc2d0d8a 100644 --- a/crates/ecstore/src/store/utils.rs +++ b/crates/ecstore/src/store/utils.rs @@ -15,7 +15,8 @@ use crate::config::storageclass::STANDARD; use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET}; use regex::Regex; -use rustfs_utils::http::headers::{AMZ_OBJECT_TAGGING, AMZ_STORAGE_CLASS}; +use rustfs_filemeta::metadata_keys; +use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING; use std::collections::HashMap; use std::io::{Error, Result}; use std::sync::LazyLock; @@ -32,8 +33,8 @@ pub fn clean_metadata(metadata: &mut HashMap) { } pub fn remove_standard_storage_class(metadata: &mut HashMap) { - if metadata.get(AMZ_STORAGE_CLASS) == Some(&STANDARD.to_string()) { - metadata.remove(AMZ_STORAGE_CLASS); + if metadata.get(metadata_keys::STORAGE_CLASS) == Some(&STANDARD.to_string()) { + metadata.remove(metadata_keys::STORAGE_CLASS); } } diff --git a/crates/filemeta/Cargo.toml b/crates/filemeta/Cargo.toml index ef0d49c3b..ae173577a 100644 --- a/crates/filemeta/Cargo.toml +++ b/crates/filemeta/Cargo.toml @@ -48,7 +48,6 @@ rustfs-config = { workspace = true, features = ["constants"] } byteorder = { workspace = true } tracing.workspace = true thiserror.workspace = true -s3s = { workspace = true, features = ["minio"] } regex.workspace = true arc-swap.workspace = true sha2.workspace = true @@ -56,6 +55,9 @@ base64-simd.workspace = true [dev-dependencies] criterion = { workspace = true, features = ["html_reports"] } +# The metacache tests use tokio::time; s3s used to enable the feature via +# feature unification before filemeta dropped it (backlog#1735 A3c). +tokio = { workspace = true, features = ["time"] } tempfile = { workspace = true } proptest = "1" diff --git a/crates/filemeta/src/fileinfo.rs b/crates/filemeta/src/fileinfo.rs index 4e0661eb8..cd938adae 100644 --- a/crates/filemeta/src/fileinfo.rs +++ b/crates/filemeta/src/fileinfo.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +use crate::metadata_keys; use crate::{Error, ReplicationState, ReplicationStatusType, Result, TRANSITION_COMPLETE, VersionPurgeStatusType}; use bytes::Bytes; use rmp_serde::Serializer; @@ -22,8 +23,6 @@ use rustfs_utils::http::{ contains_key_str, get_consistent_str, get_str, has_internal_suffix, insert_str, is_encryption_metadata_key, starts_with_ignore_ascii_case, }; -use s3s::dto::{RestoreStatus, Timestamp}; -use s3s::header::X_AMZ_RESTORE; use serde::de::{self, MapAccess, SeqAccess, Visitor, value::MapAccessDeserializer}; use serde::ser::SerializeMap; use serde::{Deserialize, Serialize}; @@ -1420,6 +1419,18 @@ pub struct FilesInfo { pub is_truncated: bool, } +/// Parsed restore status of a restored object: the value persisted under +/// [`metadata_keys::RESTORE`]. filemeta owns this type so the persisted format +/// does not depend on an HTTP library DTO (backlog#1735 A3c); the field names +/// match the former `s3s::dto::RestoreStatus` so the rendered bytes are unchanged. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct RestoreStatus { + /// `ongoing-request`; `None` reads as not in progress. + pub is_restore_in_progress: Option, + /// `expiry-date`; required to render a finished restore. + pub restore_expiry_date: Option, +} + pub trait RestoreStatusOps { fn expiry(&self) -> Option; fn on_going(&self) -> bool; @@ -1433,7 +1444,7 @@ impl RestoreStatusOps for RestoreStatus { if self.on_going() { return None; } - self.restore_expiry_date.clone().map(OffsetDateTime::from) + self.restore_expiry_date } fn on_going(&self) -> bool { @@ -1459,9 +1470,7 @@ impl RestoreStatusOps for RestoreStatus { } format!( "ongoing-request=\"false\", expiry-date=\"{}\"", - OffsetDateTime::from(self.restore_expiry_date.clone().unwrap()) - .format(&Rfc3339) - .unwrap() + self.restore_expiry_date.unwrap().format(&Rfc3339).unwrap() ) } @@ -1471,9 +1480,7 @@ impl RestoreStatusOps for RestoreStatus { } format!( "ongoing-request=\"false\", expiry-date=\"{}\"", - OffsetDateTime::from(self.restore_expiry_date.clone().unwrap()) - .format(&RFC1123) - .unwrap() + self.restore_expiry_date.unwrap().format(&RFC1123).unwrap() ) } } @@ -1534,7 +1541,7 @@ pub fn parse_restore_obj_status(restore_hdr: &str) -> Result { let expiry = parse_restore_expiry_date(expiry_tokens[1].trim_matches('"'))?; return Ok(RestoreStatus { is_restore_in_progress: Some(false), - restore_expiry_date: Some(Timestamp::from(expiry)), + restore_expiry_date: Some(expiry), }); } _ => (), @@ -1543,7 +1550,7 @@ pub fn parse_restore_obj_status(restore_hdr: &str) -> Result { } pub fn is_restored_object_on_disk(meta: &HashMap) -> bool { - if let Some(restore_hdr) = meta.get(X_AMZ_RESTORE.as_str()) + if let Some(restore_hdr) = meta.get(metadata_keys::RESTORE) && let Ok(restore_status) = parse_restore_obj_status(restore_hdr) { return restore_status.on_disk(); @@ -2833,7 +2840,7 @@ mod tests { fn restore_status_round_trips_through_both_formats() { let status = RestoreStatus { is_restore_in_progress: Some(false), - restore_expiry_date: Some(Timestamp::from(datetime!(2030-06-15 07:08:09 UTC))), + restore_expiry_date: Some(datetime!(2030-06-15 07:08:09 UTC)), }; for rendered in [RestoreStatusOps::to_string(&status), status.to_string2()] { let parsed = parse_restore_obj_status(&rendered).unwrap_or_else(|e| panic!("{rendered} must parse: {e}")); @@ -2841,6 +2848,47 @@ mod tests { } } + /// backlog#1735 A3c: filemeta's own `RestoreStatus` must persist the same + /// bytes the s3s DTO did. The restore value in the pre-`metadata_keys` + /// fixture was rendered by the s3s-backed writer; parsing and rendering it + /// again must reproduce it exactly. + #[test] + fn restore_status_rerenders_pre_module_fixture_bytes() { + let fi = crate::FileMeta::load(&crate::test_data::create_pre_metadata_keys_xlmeta().expect("decode fixture hex")) + .expect("load fixture xl.meta") + .into_fileinfo("bucket", "object", "0b1e5a3a-1735-4a3a-8000-00000000a3a0", false, false, false) + .expect("fixture version to FileInfo"); + let stored = fi.metadata.get(metadata_keys::RESTORE).expect("fixture restore value"); + assert_eq!(stored, "ongoing-request=\"false\", expiry-date=\"9999-01-01T00:00:00Z\""); + + let parsed = parse_restore_obj_status(stored).expect("fixture restore value parses"); + assert_eq!( + parsed, + RestoreStatus { + is_restore_in_progress: Some(false), + restore_expiry_date: Some(datetime!(9999-01-01 00:00:00 UTC)), + } + ); + assert_eq!(RestoreStatusOps::to_string(&parsed), *stored); + assert_eq!( + parsed.to_string2(), + "ongoing-request=\"false\", expiry-date=\"Fri, 01 Jan 9999 00:00:00 GMT\"" + ); + + let ongoing = RestoreStatus { + is_restore_in_progress: Some(true), + restore_expiry_date: Some(datetime!(2030-06-15 07:08:09 UTC)), + }; + assert_eq!(RestoreStatusOps::to_string(&ongoing), "ongoing-request=\"true\""); + assert_eq!( + parse_restore_obj_status("ongoing-request=\"true\"").expect("in-progress form parses"), + RestoreStatus { + is_restore_in_progress: Some(true), + restore_expiry_date: None, + } + ); + } + /// A restored object migrated from MinIO must still be recognised as /// on-disk: `is_restored_object_on_disk` fails open, and /// `MetaObject::uses_data_dir` uses it to decide whether a data dir is @@ -2849,7 +2897,7 @@ mod tests { fn minio_restored_object_is_recognised_as_on_disk() { let mut meta = HashMap::new(); meta.insert( - X_AMZ_RESTORE.as_str().to_string(), + metadata_keys::RESTORE.to_string(), "ongoing-request=\"false\", expiry-date=\"Fri, 01 Jan 9999 00:00:00 GMT\"".to_string(), ); assert!(is_restored_object_on_disk(&meta)); diff --git a/crates/filemeta/src/filemeta.rs b/crates/filemeta/src/filemeta.rs index 79dc69c0e..086d6805d 100644 --- a/crates/filemeta/src/filemeta.rs +++ b/crates/filemeta/src/filemeta.rs @@ -17,22 +17,18 @@ use crate::replication::{ }; use crate::{ ErasureAlgo, ErasureInfo, Error, FileInfo, FileInfoVersions, InlineData, NULL_VERSION_ID, ObjectPartInfo, RawFileInfo, - ReplicationState, ReplicationStatusType, Result, VersionPurgeStatusType, is_restored_object_on_disk, + ReplicationState, ReplicationStatusType, Result, VersionPurgeStatusType, is_restored_object_on_disk, metadata_keys, replication_statuses_map, version_purge_statuses_map, }; use byteorder::ByteOrder; use bytes::Bytes; -use rustfs_utils::http::headers::{ - AMZ_META_UNENCRYPTED_CONTENT_LENGTH, AMZ_META_UNENCRYPTED_CONTENT_MD5, AMZ_RESTORE_EXPIRY_DAYS, AMZ_RESTORE_REQUEST_DATE, - AMZ_STORAGE_CLASS, -}; +use rustfs_utils::http::headers::{AMZ_META_UNENCRYPTED_CONTENT_LENGTH, AMZ_META_UNENCRYPTED_CONTENT_MD5}; use rustfs_utils::http::{ - AMZ_BUCKET_REPLICATION_STATUS, MINIO_INTERNAL_PREFIX, RUSTFS_INTERNAL_PREFIX, SUFFIX_CRC, SUFFIX_DATA_MOV, SUFFIX_HEALING, - SUFFIX_PURGESTATUS, SUFFIX_REPLICA_STATUS, SUFFIX_REPLICA_TIMESTAMP, SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX, + MINIO_INTERNAL_PREFIX, RUSTFS_INTERNAL_PREFIX, SUFFIX_CRC, SUFFIX_DATA_MOV, SUFFIX_HEALING, SUFFIX_PURGESTATUS, + SUFFIX_REPLICA_STATUS, SUFFIX_REPLICA_TIMESTAMP, SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX, SUFFIX_REPLICATION_RESET, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TIMESTAMP, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK, contains_key_str, has_internal_suffix, insert_bytes, is_internal_key, remove_bytes, }; -use s3s::header::X_AMZ_RESTORE; use serde::{Deserialize, Serialize}; #[cfg(test)] use std::cell::Cell; @@ -1239,34 +1235,6 @@ mod test { /// `parse_restore_obj_status` (fileinfo.rs). const RESTORED_ON_DISK: &str = "ongoing-request=\"false\", expiry-date=\"9999-01-01T00:00:00Z\""; - /// backlog#1733 (P9-01 §4.3/§7.6, g-key-001): pin the five `s3s::header` - /// constants that double as **persisted metadata map keys**. They are not - /// just HTTP header names — they are stored inside xl.meta (`meta_user`) - /// and read back by fail-open code, so a silent drift produces zero - /// HTTP-visible errors while: - /// - /// 1. **WORM silently dissolves** — `get_object_retention_meta` - /// (ecstore objectlock.rs) returns an empty retention when the lock keys - /// are unreadable, making every compliance-locked object deletable. - /// 2. **Live data dirs can be reclaimed** — `MetaObject::uses_data_dir` - /// falls back to `is_restored_object_on_disk`, which returns `false` - /// when `x-amz-restore` is unreadable, so a restored object's data dir - /// is judged unused. - /// - /// Any migration replacing these constants must keep the literals byte-stable. - #[test] - fn persisted_metadata_keys_are_byte_stable() { - use s3s::header::{ - X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, - X_AMZ_SERVER_SIDE_ENCRYPTION, - }; - assert_eq!(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str(), "x-amz-object-lock-legal-hold"); - assert_eq!(X_AMZ_OBJECT_LOCK_MODE.as_str(), "x-amz-object-lock-mode"); - assert_eq!(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str(), "x-amz-object-lock-retain-until-date"); - assert_eq!(X_AMZ_RESTORE.as_str(), "x-amz-restore"); - assert_eq!(X_AMZ_SERVER_SIDE_ENCRYPTION.as_str(), "x-amz-server-side-encryption"); - } - /// backlog#1733 g-key-003: a restored-to-local object must keep its data /// dir. The restore marker lives under the pinned `x-amz-restore` key; if /// the key ever drifts this flips to `false` and the data dir becomes @@ -1303,11 +1271,11 @@ mod test { #[test] fn restore_marker_roundtrips_through_parser() { let mut meta = HashMap::new(); - meta.insert(X_AMZ_RESTORE.as_str().to_string(), RESTORED_ON_DISK.to_string()); + meta.insert(metadata_keys::RESTORE.to_string(), RESTORED_ON_DISK.to_string()); assert!(crate::is_restored_object_on_disk(&meta)); // An in-progress restore is not "on disk". - meta.insert(X_AMZ_RESTORE.as_str().to_string(), "ongoing-request=\"true\"".to_string()); + meta.insert(metadata_keys::RESTORE.to_string(), "ongoing-request=\"true\"".to_string()); assert!(!crate::is_restored_object_on_disk(&meta)); } @@ -2404,7 +2372,7 @@ mod test { /// with its transition metadata (and thus the remote tier copy) intact. #[test] fn test_delete_version_expire_restored_keeps_transitioned_version() { - use rustfs_utils::http::headers::{AMZ_RESTORE, AMZ_RESTORE_EXPIRY_DAYS, AMZ_RESTORE_REQUEST_DATE}; + use crate::metadata_keys::{RESTORE, RESTORE_EXPIRY_DAYS, RESTORE_REQUEST_DATE}; let mut fm = FileMeta::new(); let vid = Uuid::new_v4(); @@ -2418,12 +2386,12 @@ mod test { fi.transitioned_objname = "remote/obj".to_string(); fi.transition_tier = "COLDTIER".to_string(); fi.metadata.insert( - AMZ_RESTORE.to_string(), + RESTORE.to_string(), "ongoing-request=\"false\", expiry-date=\"Fri, 17 Jul 2026 00:00:00 GMT\"".to_string(), ); - fi.metadata.insert(AMZ_RESTORE_EXPIRY_DAYS.to_string(), "1".to_string()); + fi.metadata.insert(RESTORE_EXPIRY_DAYS.to_string(), "1".to_string()); fi.metadata - .insert(AMZ_RESTORE_REQUEST_DATE.to_string(), "Thu, 16 Jul 2026 00:00:00 GMT".to_string()); + .insert(RESTORE_REQUEST_DATE.to_string(), "Thu, 16 Jul 2026 00:00:00 GMT".to_string()); rustfs_utils::http::insert_str(&mut fi.metadata, SUFFIX_RESTORE_OPERATION_ID, Uuid::from_u128(1).to_string()); rustfs_utils::http::insert_str( &mut fi.metadata, @@ -2443,9 +2411,9 @@ mod test { assert_eq!(fm.versions.len(), 1, "the version must survive restored-copy expiry"); let after = fm.into_fileinfo("vol", "restored.bin", "", false, false, true).unwrap(); - assert!(!after.metadata.contains_key(AMZ_RESTORE), "x-amz-restore must be stripped"); - assert!(!after.metadata.contains_key(AMZ_RESTORE_EXPIRY_DAYS)); - assert!(!after.metadata.contains_key(AMZ_RESTORE_REQUEST_DATE)); + assert!(!after.metadata.contains_key(RESTORE), "x-amz-restore must be stripped"); + assert!(!after.metadata.contains_key(RESTORE_EXPIRY_DAYS)); + assert!(!after.metadata.contains_key(RESTORE_REQUEST_DATE)); assert!( rustfs_utils::http::get_str(&after.metadata, SUFFIX_RESTORE_OPERATION_ID).is_none(), "expired restore must not retain its operation generation" diff --git a/crates/filemeta/src/filemeta/inline_data.rs b/crates/filemeta/src/filemeta/inline_data.rs index 98d3bb28f..e1154e5c3 100644 --- a/crates/filemeta/src/filemeta/inline_data.rs +++ b/crates/filemeta/src/filemeta/inline_data.rs @@ -144,7 +144,6 @@ impl FileMeta { #[cfg(test)] mod tests { use super::*; - use s3s::header::X_AMZ_RESTORE; use time::format_description::well_known::Rfc3339; use time::{Duration, OffsetDateTime}; @@ -170,7 +169,7 @@ mod tests { data_dir, HashMap::from([ ("etag".to_string(), format!("etag-{version_id}")), - (X_AMZ_RESTORE.as_str().to_string(), restore_header), + (metadata_keys::RESTORE.to_string(), restore_header), ]), ) } diff --git a/crates/filemeta/src/filemeta/version.rs b/crates/filemeta/src/filemeta/version.rs index 554b2373b..0b3ca8b1d 100644 --- a/crates/filemeta/src/filemeta/version.rs +++ b/crates/filemeta/src/filemeta/version.rs @@ -2610,7 +2610,7 @@ impl MetaObject { continue; } - if k == AMZ_STORAGE_CLASS && v == "STANDARD" { + if k == metadata_keys::STORAGE_CLASS && v == "STANDARD" { continue; } @@ -2622,7 +2622,7 @@ impl MetaObject { continue; } - if k.eq_ignore_ascii_case(AMZ_STORAGE_CLASS) && v == b"STANDARD" { + if k.eq_ignore_ascii_case(metadata_keys::STORAGE_CLASS) && v == b"STANDARD" { continue; } @@ -2642,7 +2642,7 @@ impl MetaObject { let st = v.composite_replication_status(); if !st.is_empty() { - metadata.insert(AMZ_BUCKET_REPLICATION_STATUS.to_string(), st.to_string()); + metadata.insert(metadata_keys::REPLICATION_STATUS.to_string(), st.to_string()); } } @@ -2762,9 +2762,9 @@ impl MetaObject { } pub fn remove_restore_hdrs(&mut self) { - self.meta_user.remove(X_AMZ_RESTORE.as_str()); - self.meta_user.remove(AMZ_RESTORE_EXPIRY_DAYS); - self.meta_user.remove(AMZ_RESTORE_REQUEST_DATE); + self.meta_user.remove(metadata_keys::RESTORE); + self.meta_user.remove(metadata_keys::RESTORE_EXPIRY_DAYS); + self.meta_user.remove(metadata_keys::RESTORE_REQUEST_DATE); remove_bytes(&mut self.meta_sys, SUFFIX_RESTORE_OPERATION_ID); remove_bytes(&mut self.meta_sys, SUFFIX_RESTORE_WORKER_LOCK); } diff --git a/crates/filemeta/src/lib.rs b/crates/filemeta/src/lib.rs index 16d6d3a71..65eaccc72 100644 --- a/crates/filemeta/src/lib.rs +++ b/crates/filemeta/src/lib.rs @@ -20,6 +20,7 @@ mod filemeta_inline; mod metacache; mod replication; +pub mod metadata_keys; pub mod test_data; pub use error::*; diff --git a/crates/filemeta/src/metadata_keys.rs b/crates/filemeta/src/metadata_keys.rs new file mode 100644 index 000000000..3b8c70ef4 --- /dev/null +++ b/crates/filemeta/src/metadata_keys.rs @@ -0,0 +1,258 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Object-metadata map keys persisted inside xl.meta (`MetaObject::meta_user`). +//! +//! These strings are part of the on-disk format, not HTTP header names. They +//! happen to spell S3 header names, and some were historically taken from +//! `s3s::header` or duplicated in `rustfs_utils::http::headers`, which let an +//! HTTP-library or header-table edit silently change what is written to and +//! looked up in xl.meta. Map lookups are exact, so every byte (including ASCII +//! case) is load-bearing and existing xl.meta cannot be rewritten in bulk: +//! +//! - the object-lock keys unreadable => retention reads back as "no lock" and +//! WORM objects become deletable; +//! - [`RESTORE`] unreadable => `MetaObject::uses_data_dir` reports a restored +//! object's data dir as unused, making live data reclaimable. +//! +//! The literals are pinned by tests below against both a literal table and +//! xl.meta bytes captured from the code that predates this module. + +/// Object-lock legal hold (`ON`/`OFF`). +pub const OBJECT_LOCK_LEGAL_HOLD: &str = "x-amz-object-lock-legal-hold"; +/// Object-lock retention mode (`GOVERNANCE`/`COMPLIANCE`). +pub const OBJECT_LOCK_MODE: &str = "x-amz-object-lock-mode"; +/// Object-lock retain-until date. +pub const OBJECT_LOCK_RETAIN_UNTIL_DATE: &str = "x-amz-object-lock-retain-until-date"; +/// Restore status of a transitioned object restored to local storage. +pub const RESTORE: &str = "x-amz-restore"; +/// Requested restore duration. Persisted in this mixed case. +pub const RESTORE_EXPIRY_DAYS: &str = "X-Amz-Restore-Expiry-Days"; +/// Restore request date. Persisted in this mixed case. +pub const RESTORE_REQUEST_DATE: &str = "X-Amz-Restore-Request-Date"; +/// Server-side encryption algorithm. +pub const SERVER_SIDE_ENCRYPTION: &str = "x-amz-server-side-encryption"; +/// Storage class. +pub const STORAGE_CLASS: &str = "x-amz-storage-class"; +/// Composite replication status. Persisted in this mixed case. +pub const REPLICATION_STATUS: &str = "X-Amz-Replication-Status"; + +#[cfg(test)] +mod tests { + use super::*; + use crate::test_data::create_pre_metadata_keys_xlmeta; + use crate::{FileMeta, FileMetaVersion, MetaObject, is_restored_object_on_disk}; + use std::collections::HashMap; + + /// Every persisted key with its exact on-disk spelling, written out as an + /// independent literal so a change to a constant cannot also change the + /// expectation. + const PINNED: [(&str, &str); 9] = [ + (OBJECT_LOCK_LEGAL_HOLD, "x-amz-object-lock-legal-hold"), + (OBJECT_LOCK_MODE, "x-amz-object-lock-mode"), + (OBJECT_LOCK_RETAIN_UNTIL_DATE, "x-amz-object-lock-retain-until-date"), + (RESTORE, "x-amz-restore"), + (RESTORE_EXPIRY_DAYS, "X-Amz-Restore-Expiry-Days"), + (RESTORE_REQUEST_DATE, "X-Amz-Restore-Request-Date"), + (SERVER_SIDE_ENCRYPTION, "x-amz-server-side-encryption"), + (STORAGE_CLASS, "x-amz-storage-class"), + (REPLICATION_STATUS, "X-Amz-Replication-Status"), + ]; + + /// Values stored under each key in the pre-module fixture. + const FIXTURE_VALUES: [(&str, &str); 9] = [ + ("x-amz-object-lock-legal-hold", "ON"), + ("x-amz-object-lock-mode", "COMPLIANCE"), + ("x-amz-object-lock-retain-until-date", "2099-01-01T00:00:00Z"), + ("x-amz-restore", "ongoing-request=\"false\", expiry-date=\"9999-01-01T00:00:00Z\""), + ("X-Amz-Restore-Expiry-Days", "7"), + ("X-Amz-Restore-Request-Date", "Thu, 16 Jul 2026 00:00:00 GMT"), + ("x-amz-server-side-encryption", "AES256"), + ("x-amz-storage-class", "GLACIER"), + ("X-Amz-Replication-Status", "COMPLETED"), + ]; + + const FIXTURE_VERSION_ID: &str = "0b1e5a3a-1735-4a3a-8000-00000000a3a0"; + + fn fixture_object() -> MetaObject { + let fm = FileMeta::load(&create_pre_metadata_keys_xlmeta().expect("decode fixture hex")).expect("load fixture xl.meta"); + assert_eq!(fm.versions.len(), 1); + FileMetaVersion::try_from(fm.versions[0].meta.as_slice()) + .expect("decode fixture version") + .object + .expect("fixture version is an object") + } + + fn fixture_metadata() -> HashMap { + let fm = FileMeta::load(&create_pre_metadata_keys_xlmeta().expect("decode fixture hex")).expect("load fixture xl.meta"); + fm.into_fileinfo("bucket", "object", FIXTURE_VERSION_ID, false, false, false) + .expect("fixture version to FileInfo") + .metadata + } + + /// Replaces the byte at `idx` with a different one of the same class: + /// ASCII letters flip case (lookups are case-sensitive), anything else + /// becomes `_`. + fn mutate_at(key: &str, idx: usize) -> String { + let mut bytes = key.as_bytes().to_vec(); + let b = bytes[idx]; + bytes[idx] = if b.is_ascii_lowercase() { + b.to_ascii_uppercase() + } else if b.is_ascii_uppercase() { + b.to_ascii_lowercase() + } else { + b'_' + }; + String::from_utf8(bytes).expect("ascii mutation stays utf-8") + } + + #[test] + fn persisted_metadata_keys_are_pinned_byte_for_byte() { + for (key, literal) in PINNED { + assert_eq!(key.as_bytes(), literal.as_bytes(), "persisted key {literal:?} drifted"); + } + let fixture_keys: Vec<&str> = FIXTURE_VALUES.iter().map(|(k, _)| *k).collect(); + let pinned_keys: Vec<&str> = PINNED.iter().map(|(_, l)| *l).collect(); + assert_eq!(fixture_keys, pinned_keys, "fixture table must cover every pinned key"); + } + + /// Migration-period cross-check: the constants must equal the historical + /// `rustfs_utils` sources callers used before this module existed. The + /// `s3s::header` half was dropped with filemeta's `s3s` dependency (A3c); + /// `PINNED` and the pre-module fixture keep pinning those bytes. + #[test] + fn persisted_metadata_keys_match_their_historical_sources() { + use rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS; + use rustfs_utils::http::headers::{ + AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, AMZ_RESTORE, + AMZ_RESTORE_EXPIRY_DAYS, AMZ_RESTORE_REQUEST_DATE, AMZ_STORAGE_CLASS, + }; + assert_eq!(OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER); + assert_eq!(OBJECT_LOCK_MODE, AMZ_OBJECT_LOCK_MODE_LOWER); + assert_eq!(OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER); + assert_eq!(RESTORE, AMZ_RESTORE); + assert_eq!(RESTORE_EXPIRY_DAYS, AMZ_RESTORE_EXPIRY_DAYS); + assert_eq!(RESTORE_REQUEST_DATE, AMZ_RESTORE_REQUEST_DATE); + assert_eq!(STORAGE_CLASS, AMZ_STORAGE_CLASS); + assert_eq!(REPLICATION_STATUS, AMZ_BUCKET_REPLICATION_STATUS); + } + + /// The fixture bytes were written by the code that predates this module, + /// with each key taken from its historical source. Every key must appear + /// verbatim in the raw bytes and read back through the new constants. + #[test] + fn pre_module_xlmeta_reads_back_every_persisted_key() { + let raw = create_pre_metadata_keys_xlmeta().expect("decode fixture hex"); + for (key, _) in PINNED { + assert!( + raw.windows(key.len()).any(|w| w == key.as_bytes()), + "fixture bytes must contain persisted key {key:?} verbatim" + ); + } + + let obj = fixture_object(); + let metadata = fixture_metadata(); + for ((key, _), (fixture_key, value)) in PINNED.iter().zip(FIXTURE_VALUES) { + assert_eq!(*key, fixture_key); + assert_eq!(obj.meta_user.get(*key).map(String::as_str), Some(value), "meta_user[{key:?}]"); + assert_eq!(metadata.get(*key).map(String::as_str), Some(value), "FileInfo.metadata[{key:?}]"); + } + + assert!(is_restored_object_on_disk(&metadata), "restore marker must read back as on disk"); + assert!(obj.uses_data_dir(), "restored object's data dir must stay in use"); + } + + /// Removing restore headers from the pre-module object must drop exactly + /// the restore keys and flip `uses_data_dir`, proving the remover and the + /// reader address the same persisted bytes. + #[test] + fn remove_restore_hdrs_strips_pre_module_restore_keys() { + let mut obj = fixture_object(); + obj.remove_restore_hdrs(); + for key in [RESTORE, RESTORE_EXPIRY_DAYS, RESTORE_REQUEST_DATE] { + assert!(!obj.meta_user.contains_key(key), "{key:?} must be removed"); + } + for key in [ + OBJECT_LOCK_LEGAL_HOLD, + OBJECT_LOCK_MODE, + OBJECT_LOCK_RETAIN_UNTIL_DATE, + SERVER_SIDE_ENCRYPTION, + STORAGE_CLASS, + REPLICATION_STATUS, + ] { + assert!(obj.meta_user.contains_key(key), "{key:?} must survive restore cleanup"); + } + assert!(!is_restored_object_on_disk(&obj.meta_user)); + assert!(!obj.uses_data_dir()); + } + + /// Writing through the new constants must produce the same persisted key + /// set as the pre-module bytes, so a rollback reads new objects too. + #[test] + fn new_writes_persist_the_same_keys_as_pre_module_bytes() { + let mut fi = FileMeta::load(&create_pre_metadata_keys_xlmeta().expect("decode fixture hex")) + .expect("load fixture xl.meta") + .into_fileinfo("bucket", "object", FIXTURE_VERSION_ID, false, false, false) + .expect("fixture version to FileInfo"); + fi.metadata = PINNED + .iter() + .zip(FIXTURE_VALUES) + .map(|((key, _), (_, value))| (key.to_string(), value.to_string())) + .collect(); + fi.metadata + .insert("etag".to_string(), "d41d8cd98f00b204e9800998ecf8427e".to_string()); + + let mut fm = FileMeta::new(); + fm.add_version(fi).expect("add version"); + let rewritten = FileMeta::load(&fm.marshal_msg().expect("marshal")).expect("reload"); + let rewritten = FileMetaVersion::try_from(rewritten.versions[0].meta.as_slice()) + .expect("decode rewritten version") + .object + .expect("rewritten version is an object"); + + assert_eq!(rewritten.meta_user, fixture_object().meta_user); + } + + /// Per-character mutation: a constant that differs from the persisted key + /// by any single byte (including ASCII case) finds nothing in pre-module + /// xl.meta, and a restore marker stored under such a key is not honored. + /// This is the drift the pins above turn into a test failure. + #[test] + fn single_character_key_mutation_misses_pre_module_data() { + let metadata = fixture_metadata(); + let mut mutations = 0usize; + for (key, _) in PINNED { + for idx in 0..key.len() { + let mutated = mutate_at(key, idx); + assert_ne!(mutated, key); + assert!( + !metadata.contains_key(&mutated), + "mutation {mutated:?} of {key:?} must not match pre-module data" + ); + mutations += 1; + } + } + assert_eq!(mutations, PINNED.iter().map(|(k, _)| k.len()).sum::()); + + let marker = metadata.get(RESTORE).expect("fixture restore marker").clone(); + for idx in 0..RESTORE.len() { + let meta = HashMap::from([(mutate_at(RESTORE, idx), marker.clone())]); + assert!( + !is_restored_object_on_disk(&meta), + "restore marker under {:?} must not count as on disk", + mutate_at(RESTORE, idx) + ); + } + } +} diff --git a/crates/filemeta/src/test_data.rs b/crates/filemeta/src/test_data.rs index 3ccb7b0d8..95a552597 100644 --- a/crates/filemeta/src/test_data.rs +++ b/crates/filemeta/src/test_data.rs @@ -171,6 +171,14 @@ pub fn create_minio_large_object_xlmeta() -> Result> { decode_hex_fixture(include_str!("../tests/fixtures/minio/object_large_bin.xlmeta.hex")) } +/// xl.meta for one restored, object-locked object, written by the code that +/// predates `metadata_keys` with every persisted key taken from its historical +/// source (`s3s::header` / `rustfs_utils::http`). Never regenerate it: it is +/// the old-bytes evidence that the key authority stayed byte-identical. +pub fn create_pre_metadata_keys_xlmeta() -> Result> { + decode_hex_fixture(include_str!("../tests/fixtures/persisted_metadata_keys_pre_a3a.hex")) +} + fn write_legacy_time(wr: &mut Vec, ts: OffsetDateTime) { wr.push(MSGPACK_EXT8); wr.push(12); diff --git a/crates/filemeta/tests/fixtures/persisted_metadata_keys_pre_a3a.hex b/crates/filemeta/tests/fixtures/persisted_metadata_keys_pre_a3a.hex new file mode 100644 index 000000000..52f4d9809 --- /dev/null +++ b/crates/filemeta/tests/fixtures/persisted_metadata_keys_pre_a3a.hex @@ -0,0 +1 @@ +584c322001000300c6000002aa030301c42697c4100b1e5a3a17354a3a800000000000a3a0d318c29c112e760000c40405541d2b01020204c5027c83a45479706501a556324f626ade0011a24944c4100b1e5a3a17354a3a800000000000a3a0a444446972c4100b1e5a3a17354a3a800000000000d1d0a64563416c676f01a345634d04a345634e02a745634253697a65ce00100000a74563496e64657801a645634469737496010203040506a84353756d416c676f01a8506172744e756d7390a9506172744554616773c0a95061727453697a657390aa506172744153697a6573c0a453697a65ce00010000a54d54696d65cf18c29c112e760000a74d657461537973c0a74d6574615573728aa465746167d9206434316438636439386630306232303465393830303939386563663834323765d923782d616d7a2d6f626a6563742d6c6f636b2d72657461696e2d756e74696c2d64617465b4323039392d30312d30315430303a30303a30305abc782d616d7a2d7365727665722d736964652d656e6372797074696f6ea6414553323536b8582d416d7a2d5265706c69636174696f6e2d537461747573a9434f4d504c45544544bc782d616d7a2d6f626a6563742d6c6f636b2d6c6567616c2d686f6c64a24f4eb9582d416d7a2d526573746f72652d4578706972792d44617973a137ad782d616d7a2d726573746f7265d93b6f6e676f696e672d726571756573743d2266616c7365222c206578706972792d646174653d22393939392d30312d30315430303a30303a30305a22ba582d416d7a2d526573746f72652d526571756573742d44617465bd5468752c203136204a756c20323032362030303a30303a303020474d54b6782d616d7a2d6f626a6563742d6c6f636b2d6d6f6465aa434f4d504c49414e4345b3782d616d7a2d73746f726167652d636c617373a7474c4143494552a17600ce35759b93 diff --git a/crates/heal/tests/admin_pool_set_contract_test.rs b/crates/heal/tests/admin_pool_set_contract_test.rs index 201d55f3f..dd161eeb1 100644 --- a/crates/heal/tests/admin_pool_set_contract_test.rs +++ b/crates/heal/tests/admin_pool_set_contract_test.rs @@ -358,11 +358,13 @@ async fn crash_child(root: PathBuf, replay: bool) { && processed > 0 { assert!(processed < 64, "crash must interrupt a partial traversal"); - std::fs::write( - root.join("acknowledged.json"), - serde_json::to_vec(&value["outcome"]).expect("acknowledged outcome"), - ) - .expect("signal committed work"); + // The parent may kill us as soon as the final path exists. + let mut acknowledged = tempfile::NamedTempFile::new_in(&root).expect("acknowledgement staging file"); + serde_json::to_writer(acknowledged.as_file_mut(), &value["outcome"]).expect("write acknowledged outcome"); + acknowledged.as_file().sync_all().expect("sync acknowledged outcome"); + acknowledged + .persist(root.join("acknowledged.json")) + .expect("signal committed work"); // Freeze the current-thread executor at an acknowledged boundary // until the parent kills the process without running destructors. loop { diff --git a/crates/heal/tests/mrf_partial_write_test.rs b/crates/heal/tests/mrf_partial_write_test.rs index dc57cfdd4..75f2c9fde 100644 --- a/crates/heal/tests/mrf_partial_write_test.rs +++ b/crates/heal/tests/mrf_partial_write_test.rs @@ -51,27 +51,31 @@ async fn partial_write_persistence_failure_is_reported_and_retained_for_retry() .await .expect("initial responsibility must commit"); assert!(snapshot_contains("old.bin").await); - // Linux pins the disk root with a directory descriptor, so renaming that - // root does not interrupt I/O. Block the metadata volume below it instead. - let metadata_roots: Vec<_> = env.disk_paths.iter().map(|path| path.join(RUSTFS_META_BUCKET)).collect(); - for path in &metadata_roots { - tokio::fs::rename(path, path.with_extension("offline")) + let snapshot = inspect_local_committed_snapshot(SNAPSHOT_LIMIT) + .await + .expect("initial committed snapshot should validate") + .expect("initial responsibility should have a committed snapshot"); + // Block only the successor manifest, preserving the committed anchor. + // Removing each empty blocker restores writes in one operation, so the + // consumer cannot recreate a directory between removal and restoration. + let manifest_path = format!(".heal-mrf-commit.{}.bin", 1 - snapshot.slot()); + let manifest_blockers: Vec<_> = env + .disk_paths + .iter() + .map(|path| path.join(RUSTFS_META_BUCKET).join(&manifest_path)) + .collect(); + for path in &manifest_blockers { + tokio::fs::create_dir(path) .await - .expect("detach journal disk"); - tokio::fs::write(path, b"unwritable journal root") - .await - .expect("prevent journal writes"); + .expect("block successor checkpoint manifest"); } assert_eq!( persist_partial_write_intent("partial-persistence", "new.bin", None, scope).await, Err(MrfDurableAdmissionError::Persistence), "failed checkpoint publication must not be acknowledged as durable success" ); - for path in &metadata_roots { - tokio::fs::remove_file(path).await.expect("remove journal fault"); - tokio::fs::rename(path.with_extension("offline"), path) - .await - .expect("restore journal disk"); + for path in &manifest_blockers { + tokio::fs::remove_dir(path).await.expect("remove checkpoint manifest blocker"); } for disk in env.ecstore.pools[0].get_disks(0).disks.read().await.iter().flatten() { disk.reset_health_for_store_init_retry(); diff --git a/crates/io-metrics/src/lib.rs b/crates/io-metrics/src/lib.rs index 9de1fd285..25fccf1f7 100644 --- a/crates/io-metrics/src/lib.rs +++ b/crates/io-metrics/src/lib.rs @@ -292,6 +292,7 @@ pub use process_lock_metrics::{ snapshot_process_platform_stats, }; pub use s3_api_metrics::{S3OperationMetricSnapshot, init_s3_metrics, record_s3_op, s3_op_metrics_snapshot}; +pub use s3_http_metrics::{S3HttpCompletionObserver, S3HttpCompletionObserverEnabled, install_s3_http_completion_observer}; pub use sampler::{ ProcessResourceSnapshot, ProcessSampler, ProcessStatusSnapshot, ProcessSystemSnapshot, snapshot_process_platform, snapshot_process_resource, snapshot_process_resource_and_system, snapshot_process_resource_and_system_with, diff --git a/crates/io-metrics/src/s3_http_metrics.rs b/crates/io-metrics/src/s3_http_metrics.rs index 283d01a2e..1cef441d7 100644 --- a/crates/io-metrics/src/s3_http_metrics.rs +++ b/crates/io-metrics/src/s3_http_metrics.rs @@ -20,6 +20,13 @@ use rustfs_s3_ops::S3Operation; use std::cell::Cell; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{LazyLock, OnceLock}; +use std::time::{Duration, Instant}; + +/// Receives a finished external request's first dispatched operation, its +/// latency, and whether it produced a 2xx response. The server injects this +/// so the leaf metrics crate never depends on a trace-bus implementation. +pub type S3HttpCompletionObserver = fn(S3Operation, Duration, bool); +pub type S3HttpCompletionObserverEnabled = fn() -> bool; const METRIC: &str = "rustfs_s3_http_requests_total"; const METHODS: [&str; 10] = [ @@ -28,6 +35,7 @@ const METHODS: [&str; 10] = [ const OUTCOMES: [&str; 8] = ["1xx", "2xx", "3xx", "4xx", "5xx", "unknown", "service_error", "cancelled"]; const UNKNOWN_OPERATION: usize = S3Operation::ALL.len(); static COUNTERS: LazyLock = LazyLock::new(HttpOutcomeCounters::new); +static COMPLETION_OBSERVER: OnceLock<(S3HttpCompletionObserverEnabled, S3HttpCompletionObserver)> = OnceLock::new(); tokio::task_local! { static CURRENT_OPERATION: Cell; @@ -110,6 +118,7 @@ pub(crate) fn observe_s3_http_operation(op: S3Operation) { pub struct S3HttpRequestGuard { method: usize, operation: usize, + completion: Option<(S3HttpCompletionObserver, Instant)>, finished: bool, } @@ -122,10 +131,20 @@ impl S3HttpRequestGuard { Self { method: METHODS.iter().position(|known| *known == method).unwrap_or(METHODS.len() - 1), operation: UNKNOWN_OPERATION, + completion: COMPLETION_OBSERVER + .get() + .and_then(|(enabled, observer)| enabled().then_some((*observer, Instant::now()))), finished: false, } } + /// Report the request to `observer` when it finishes with a dispatched S3 + /// operation. Latency is measured from this call. + pub fn with_completion_observer(mut self, observer: S3HttpCompletionObserver) -> Self { + self.completion = Some((observer, Instant::now())); + self + } + /// Attribute existing operation instrumentation without changing S3 /// handlers or propagating metric labels through storage/RPC contracts. pub fn in_scope(&mut self, f: impl FnOnce() -> T) -> T { @@ -151,11 +170,18 @@ impl S3HttpRequestGuard { fn finish(&mut self, outcome: usize) { if !self.finished { COUNTERS.record(self.method, self.operation, outcome); + if let Some(((observer, started_at), operation)) = self.completion.take().zip(S3Operation::ALL.get(self.operation)) { + observer(*operation, started_at.elapsed(), outcome == 1); + } self.finished = true; } } } +pub fn install_s3_http_completion_observer(enabled: S3HttpCompletionObserverEnabled, observer: S3HttpCompletionObserver) { + let _ = COMPLETION_OBSERVER.set((enabled, observer)); +} + impl Drop for S3HttpRequestGuard { fn drop(&mut self) { self.finish(7); @@ -172,6 +198,32 @@ mod tests { use metrics::with_local_recorder; use metrics_util::debugging::DebuggingRecorder; + #[test] + fn completion_observer_sees_each_dispatched_request_once() { + static SEEN: std::sync::Mutex> = std::sync::Mutex::new(Vec::new()); + fn observe(operation: S3Operation, _duration: Duration, succeeded: bool) { + SEEN.lock().expect("observer log").push((operation, succeeded)); + } + + let mut ok = S3HttpRequestGuard::new("GET").with_completion_observer(observe); + ok.in_scope(|| observe_s3_http_operation(S3Operation::GetObject)); + ok.response(200); + drop(ok); + + let mut failed = S3HttpRequestGuard::new("PUT").with_completion_observer(observe); + failed.in_scope(|| observe_s3_http_operation(S3Operation::PutObject)); + failed.response(503); + + // Rejected before S3 dispatch: counted, but there is no operation to report. + let mut undispatched = S3HttpRequestGuard::new("GET").with_completion_observer(observe); + undispatched.service_error(); + + assert_eq!( + *SEEN.lock().expect("observer log"), + [(S3Operation::GetObject, true), (S3Operation::PutObject, false)] + ); + } + #[test] fn outcome_counters_distinguish_partial_and_complete_write_failure() { let counters = HttpOutcomeCounters::new(); diff --git a/crates/lifecycle/src/evaluator.rs b/crates/lifecycle/src/evaluator.rs index d492b4645..7f9089610 100644 --- a/crates/lifecycle/src/evaluator.rs +++ b/crates/lifecycle/src/evaluator.rs @@ -206,12 +206,12 @@ mod tests { use std::sync::Arc; use rustfs_scanner_metrics::metrics::IlmAction; + use rustfs_storage_api::metadata_keys; use s3s::dto::{ BucketLifecycleConfiguration, DefaultRetention, ExpirationStatus, LifecycleExpiration, LifecycleRule, NoncurrentVersionExpiration, ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule, Transition, TransitionStorageClass, }; - use s3s::header::{X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE}; use time::OffsetDateTime; use uuid::Uuid; @@ -251,7 +251,7 @@ mod tests { for object in &mut objects { object .user_defined - .insert(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), "ON".to_string()); + .insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string()); } let locked = evaluator .eval(&objects) @@ -485,7 +485,7 @@ mod tests { fn locked_current_object_opts(replication_status: ReplicationStatusType) -> ObjectOpts { let mut user_defined = HashMap::new(); - user_defined.insert(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), "ON".to_string()); + user_defined.insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string()); ObjectOpts { user_defined, @@ -507,10 +507,10 @@ mod tests { .expect("future retain-until date should format"); let mut user_defined = HashMap::new(); user_defined.insert( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(), ); - user_defined.insert(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), retain_until); + user_defined.insert(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until); ObjectOpts { user_defined, @@ -717,7 +717,7 @@ mod tests { .eval(&version_group( Some(Uuid::nil()), ReplicationStatusType::Completed, - HashMap::from([(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), "ON".to_string())]), + HashMap::from([(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string())]), )) .await .expect("locked null-version lifecycle evaluation should fail closed without aborting evaluation"); diff --git a/crates/lifecycle/src/object_lock.rs b/crates/lifecycle/src/object_lock.rs index c2049dbd2..57e1b5d46 100644 --- a/crates/lifecycle/src/object_lock.rs +++ b/crates/lifecycle/src/object_lock.rs @@ -14,8 +14,8 @@ use std::collections::HashMap; +use rustfs_storage_api::metadata_keys; use s3s::dto::{ObjectLockConfiguration, ObjectLockRetentionMode}; -use s3s::header::{X_AMZ_OBJECT_LOCK_LEGAL_HOLD, X_AMZ_OBJECT_LOCK_MODE, X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE}; use time::{OffsetDateTime, format_description}; pub fn is_object_locked_by_metadata(user_defined: &HashMap, is_delete_marker: bool) -> bool { @@ -24,13 +24,13 @@ pub fn is_object_locked_by_metadata(user_defined: &HashMap, is_d } if user_defined - .get(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()) + .get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) .is_some_and(|value| value.eq_ignore_ascii_case("ON")) { return true; } - let Some(mode) = user_defined.get(X_AMZ_OBJECT_LOCK_MODE.as_str()) else { + let Some(mode) = user_defined.get(metadata_keys::OBJECT_LOCK_MODE) else { return false; }; if !is_retention_mode(mode) { @@ -38,7 +38,7 @@ pub fn is_object_locked_by_metadata(user_defined: &HashMap, is_d } user_defined - .get(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str()) + .get(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE) .and_then(|value| OffsetDateTime::parse(value, &format_description::well_known::Iso8601::DEFAULT).ok()) .is_some_and(|retain_until| retain_until.unix_timestamp() > OffsetDateTime::now_utc().unix_timestamp()) } @@ -85,22 +85,22 @@ pub fn is_object_locked( } fn has_explicit_lock_metadata(user_defined: &HashMap) -> bool { - user_defined.contains_key(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()) - || user_defined.contains_key(X_AMZ_OBJECT_LOCK_MODE.as_str()) - || user_defined.contains_key(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str()) + user_defined.contains_key(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) + || user_defined.contains_key(metadata_keys::OBJECT_LOCK_MODE) + || user_defined.contains_key(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE) } fn explicit_lock_metadata_is_well_formed(user_defined: &HashMap) -> bool { if user_defined - .get(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str()) + .get(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) .is_some_and(|value| !value.eq_ignore_ascii_case("ON") && !value.eq_ignore_ascii_case("OFF")) { return false; } match ( - user_defined.get(X_AMZ_OBJECT_LOCK_MODE.as_str()), - user_defined.get(X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str()), + user_defined.get(metadata_keys::OBJECT_LOCK_MODE), + user_defined.get(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE), ) { (None, None) => true, (Some(mode), Some(retain_until)) => { @@ -142,7 +142,7 @@ mod tests { #[test] fn is_object_locked_by_metadata_preserves_object_lock_parser_behavior() { let mut user_defined = HashMap::new(); - user_defined.insert(X_AMZ_OBJECT_LOCK_LEGAL_HOLD.as_str().to_string(), "ON".to_string()); + user_defined.insert(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string()); assert!(is_object_locked_by_metadata(&user_defined, false)); assert!(!is_object_locked_by_metadata(&user_defined, true)); @@ -231,11 +231,11 @@ mod tests { let config = default_retention_config(30); let mut user_defined = HashMap::new(); user_defined.insert( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), ObjectLockRetentionMode::GOVERNANCE.to_string(), ); user_defined.insert( - X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), (OffsetDateTime::now_utc() - Duration::days(1)) .format(&format_description::well_known::Iso8601::DEFAULT) .expect("expired retention date should format"), @@ -249,7 +249,7 @@ mod tests { let config = default_retention_config(1); let mut user_defined = HashMap::new(); user_defined.insert( - X_AMZ_OBJECT_LOCK_MODE.as_str().to_string(), + metadata_keys::OBJECT_LOCK_MODE.to_string(), ObjectLockRetentionMode::GOVERNANCE.to_string(), ); let created = OffsetDateTime::now_utc() - Duration::days(2); diff --git a/crates/lock/src/fast_lock/manager.rs b/crates/lock/src/fast_lock/manager.rs index fecb2d92f..8bbada477 100644 --- a/crates/lock/src/fast_lock/manager.rs +++ b/crates/lock/src/fast_lock/manager.rs @@ -341,6 +341,16 @@ impl FastObjectLockManager { self.shards.iter().map(|shard| shard.lock_count()).sum() } + /// Return aggregate live holder and waiter counts without collecting lock names. + pub fn current_lock_counts(&self) -> (u64, u64) { + self.shards.iter().map(|shard| shard.current_lock_counts()).fold( + (0, 0), + |(holders, waiters), (shard_holders, shard_waiters)| { + (holders.saturating_add(shard_holders), waiters.saturating_add(shard_waiters)) + }, + ) + } + /// Get pool statistics from all shards pub fn get_pool_stats(&self) -> Vec<(u64, u64, u64, usize)> { self.shards.iter().map(|shard| shard.pool_stats()).collect() diff --git a/crates/lock/src/fast_lock/shard.rs b/crates/lock/src/fast_lock/shard.rs index 5da4d355b..402c49e1b 100644 --- a/crates/lock/src/fast_lock/shard.rs +++ b/crates/lock/src/fast_lock/shard.rs @@ -717,6 +717,16 @@ impl LockShard { objects.values().filter(|state| state.is_locked()).count() } + pub(crate) fn current_lock_counts(&self) -> (u64, u64) { + self.objects + .read() + .values() + .map(|state| state.atomic_state.holder_and_waiter_counts()) + .fold((0, 0), |(holders, waiters), (state_holders, state_waiters)| { + (holders.saturating_add(state_holders), waiters.saturating_add(state_waiters)) + }) + } + /// Adaptive cleanup based on current load pub fn adaptive_cleanup(&self) -> usize { let current_load = self.current_load_factor(); diff --git a/crates/lock/src/fast_lock/state.rs b/crates/lock/src/fast_lock/state.rs index da6611a58..4ef012248 100644 --- a/crates/lock/src/fast_lock/state.rs +++ b/crates/lock/src/fast_lock/state.rs @@ -307,6 +307,13 @@ impl AtomicLockState { ((state & WRITERS_WAITING_MASK) >> WRITERS_WAITING_SHIFT) as u16 } + pub(crate) fn holder_and_waiter_counts(&self) -> (u64, u64) { + let state = self.state.load(Ordering::Acquire); + let holders = u64::from(self.readers_count(state)) + u64::from((state & WRITER_FLAG_MASK) != 0); + let waiters = u64::from(self.readers_waiting(state)) + u64::from(self.writers_waiting(state)); + (holders, waiters) + } + #[cfg(test)] pub fn readers_waiting_count(&self) -> u16 { let state = self.state.load(Ordering::Acquire); diff --git a/crates/lock/src/fast_lock/tests.rs b/crates/lock/src/fast_lock/tests.rs index 7a2543a84..08bbe3639 100644 --- a/crates/lock/src/fast_lock/tests.rs +++ b/crates/lock/src/fast_lock/tests.rs @@ -111,6 +111,40 @@ mod fast_lock_tests { assert!(guard.is_released(), "Guard should be marked as released"); } + #[tokio::test] + async fn current_lock_counts_report_live_holders_and_waiters() { + let manager = Arc::new(create_test_manager()); + let key = ObjectKey::new("test-bucket", "contended-object"); + let holder = manager + .acquire_read_lock(key.clone(), Arc::::from("holder")) + .await + .expect("read holder"); + let second_holder = manager + .acquire_read_lock(key.clone(), Arc::::from("second-holder")) + .await + .expect("second read holder"); + + let waiter_manager = manager.clone(); + let waiter = tokio::spawn(async move { waiter_manager.acquire_write_lock(key, Arc::::from("waiter")).await }); + tokio::time::timeout(Duration::from_secs(1), async { + loop { + if manager.current_lock_counts() == (2, 1) { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("writer waiter should be observed"); + + drop(holder); + drop(second_holder); + let acquired = waiter.await.expect("waiter task").expect("waiting writer should acquire"); + assert_eq!(manager.current_lock_counts(), (1, 0)); + drop(acquired); + assert_eq!(manager.current_lock_counts(), (0, 0)); + } + #[tokio::test] async fn test_lock_auto_release_on_drop() { let manager = create_test_manager(); diff --git a/crates/lock/src/lib.rs b/crates/lock/src/lib.rs index be1f1c68d..faade2a83 100644 --- a/crates/lock/src/lib.rs +++ b/crates/lock/src/lib.rs @@ -273,6 +273,11 @@ pub fn get_global_lock_manager() -> Arc { GLOBAL_LOCK_MANAGER.get_or_init(|| Arc::new(GlobalLockManager::new())).clone() } +/// Return the global lock manager only when it is already initialized. +pub fn get_initialized_global_lock_manager() -> Option> { + GLOBAL_LOCK_MANAGER.get().cloned() +} + /// Get the global shared FastLock manager instance (legacy) /// /// This function is deprecated. Use get_global_lock_manager() instead. diff --git a/crates/protocols/Cargo.toml b/crates/protocols/Cargo.toml index 7f8170d9c..24cd74d7f 100644 --- a/crates/protocols/Cargo.toml +++ b/crates/protocols/Cargo.toml @@ -102,7 +102,7 @@ swift = [ "dep:hex-simd", "dep:ipnetwork", "dep:rustfs-trusted-proxies", - "dep:astral-tokio-tar", + "dep:rustfs-tokio-tar", "dep:base64-simd", "dep:async-compression", ] @@ -166,7 +166,7 @@ sha1 = { workspace = true, optional = true } hex-simd = { workspace = true, optional = true } ipnetwork = { workspace = true, optional = true } rustfs-trusted-proxies = { workspace = true, optional = true } -astral-tokio-tar = { workspace = true, optional = true } +rustfs-tokio-tar = { workspace = true, optional = true } base64-simd = { workspace = true, optional = true } async-compression = { workspace = true, optional = true, features = ["tokio", "gzip", "bzip2"] } diff --git a/crates/protocols/src/swift/bulk.rs b/crates/protocols/src/swift/bulk.rs index 199b1296c..9ef5382a1 100644 --- a/crates/protocols/src/swift/bulk.rs +++ b/crates/protocols/src/swift/bulk.rs @@ -633,7 +633,7 @@ mod tests { /// Tests for the `extract_tar_entries` async function. /// /// Conditionally compiled with the `swift` feature, which gates the - /// `tokio_tar` (astral-tokio-tar) and `async_compression` dependencies. + /// `tokio_tar` (rustfs-tokio-tar) and `async_compression` dependencies. #[cfg(feature = "swift")] mod tar_extraction { use super::*; diff --git a/crates/rio/Cargo.toml b/crates/rio/Cargo.toml index 278d2373e..c30b83a86 100644 --- a/crates/rio/Cargo.toml +++ b/crates/rio/Cargo.toml @@ -86,7 +86,6 @@ base64-simd.workspace = true sha1.workspace = true sha2.workspace = true xxhash-rust = { workspace = true, features = ["xxh64", "xxh3"] } -s3s = { workspace = true, features = ["minio"] } hex-simd.workspace = true [dev-dependencies] diff --git a/crates/rio/src/hash_reader.rs b/crates/rio/src/hash_reader.rs index 49efc1d6b..36ef1a7e6 100644 --- a/crates/rio/src/hash_reader.rs +++ b/crates/rio/src/hash_reader.rs @@ -90,11 +90,11 @@ use crate::ChecksumType; use crate::Sha256Hasher; use crate::compress_index::{Index, TryGetIndex}; use crate::get_content_checksum; +use crate::trailer::SharedTrailerSource; use crate::{DynReader, EtagReader, EtagResolvable, HardLimitReader, HashReaderDetector, WarpReader, boxed_reader, wrap_reader}; use http::HeaderMap; use pin_project_lite::pin_project; -use s3s::TrailingHeaders; use std::collections::HashMap; use std::io::Cursor; use std::io::Write; @@ -117,8 +117,8 @@ pub trait HashReaderMut { fn set_actual_size(&mut self, actual_size: i64); fn content_hash(&self) -> &Option; fn content_sha256(&self) -> &Option; - fn get_trailer(&self) -> Option<&TrailingHeaders>; - fn set_trailer(&mut self, trailer: Option); + fn get_trailer(&self) -> Option<&SharedTrailerSource>; + fn set_trailer(&mut self, trailer: Option); } pin_project! { @@ -137,7 +137,8 @@ pin_project! { content_sha256_hasher: Option, checksum_on_finish: bool, - trailer_s3s: Option, + // Read only at EOF; see the timing contract in `crate::trailer`. + trailer: Option, } @@ -183,7 +184,7 @@ impl HashReader { content_sha256: sha256hex.clone(), content_sha256_hasher: sha256hex.map(|_| Sha256Hasher::new()), checksum_on_finish: false, - trailer_s3s: None, + trailer: None, }) } @@ -223,7 +224,7 @@ impl HashReader { content_sha256: sha256hex.clone(), content_sha256_hasher: sha256hex.map(|_| Sha256Hasher::new()), checksum_on_finish: false, - trailer_s3s: None, + trailer: None, }) } @@ -293,7 +294,7 @@ impl HashReader { content_hash, content_hasher, checksum_on_finish: false, - trailer_s3s: existing_hash_reader.get_trailer().cloned(), + trailer: existing_hash_reader.get_trailer().cloned(), }) } else { if size > 0 { @@ -321,7 +322,7 @@ impl HashReader { content_sha256: sha256hex.clone(), content_sha256_hasher: sha256hex.map(|_| Sha256Hasher::new()), checksum_on_finish: false, - trailer_s3s: None, + trailer: None, }) } } @@ -352,10 +353,12 @@ impl HashReader { self.actual_size } - pub fn add_checksum_from_s3s( + /// Records the request checksum declared in `headers`. For a trailing + /// checksum, `trailer` supplies its value once the body has been read. + pub fn add_checksum( &mut self, headers: &HeaderMap, - trailing_headers: Option, + trailer: Option, ignore_value: bool, ) -> Result<(), std::io::Error> { let cs = get_content_checksum(headers)?; @@ -366,7 +369,7 @@ impl HashReader { if let Some(checksum) = cs { if checksum.checksum_type.trailing() { - self.trailer_s3s = trailing_headers; + self.trailer = trailer; } self.content_hash = Some(checksum.clone()); @@ -447,14 +450,7 @@ impl HashReader { } if checksum.checksum_type.trailing() { - if let Some(trailer) = self.trailer_s3s.as_ref() - && let Some(Some(checksum_str)) = trailer.read(|headers| { - checksum - .checksum_type - .key() - .and_then(|key| headers.get(key).and_then(|value| value.to_str().ok().map(|s| s.to_string()))) - }) - { + if let Some(checksum_str) = trailing_checksum_value(self.trailer.as_ref(), checksum.checksum_type) { map.insert(checksum.checksum_type.to_string(), checksum_str); } return map; @@ -468,6 +464,14 @@ impl HashReader { } } +/// Returns the received trailer value for `checksum_type`. `Pending` and +/// `Missing` both yield `None`, which leaves a trailing checksum unverified +/// and therefore rejected at EOF. +fn trailing_checksum_value(trailer: Option<&SharedTrailerSource>, checksum_type: ChecksumType) -> Option { + let key = checksum_type.key()?; + trailer?.lookup(key).into_present() +} + impl HashReaderMut for HashReader { fn into_inner(self) -> DynReader { self.inner @@ -514,12 +518,12 @@ impl HashReaderMut for HashReader { &self.content_sha256 } - fn get_trailer(&self) -> Option<&TrailingHeaders> { - self.trailer_s3s.as_ref() + fn get_trailer(&self) -> Option<&SharedTrailerSource> { + self.trailer.as_ref() } - fn set_trailer(&mut self, trailer: Option) { - self.trailer_s3s = trailer; + fn set_trailer(&mut self, trailer: Option) { + self.trailer = trailer; } } @@ -572,13 +576,8 @@ impl AsyncRead for HashReader { // check content hasher if let (Some(hasher), Some(expected_content_hash)) = (this.content_hasher, this.content_hash) { if expected_content_hash.checksum_type.trailing() - && let Some(trailer) = this.trailer_s3s.as_ref() - && let Some(Some(checksum_str)) = trailer.read(|headers| { - expected_content_hash - .checksum_type - .key() - .and_then(|key| headers.get(key).and_then(|value| value.to_str().ok().map(|s| s.to_string()))) - }) + && let Some(checksum_str) = + trailing_checksum_value(this.trailer.as_ref(), expected_content_hash.checksum_type) { expected_content_hash.encoded = checksum_str; expected_content_hash.raw = base64_simd::STANDARD @@ -1073,3 +1072,190 @@ mod tests { } } } + +#[cfg(test)] +mod trailer_tests { + use super::*; + use crate::{TrailerSource, TrailerValue, is_checksum_mismatch}; + use http::HeaderValue; + use std::sync::{Arc, Mutex}; + use tokio::io::AsyncReadExt; + + const TYPED: [ChecksumType; 5] = [ + ChecksumType::CRC32, + ChecksumType::CRC32C, + ChecksumType::SHA1, + ChecksumType::SHA256, + ChecksumType::CRC64_NVME, + ]; + + /// Mirrors an aws-chunked decoder handle: `None` until the trailer section + /// has been published. + #[derive(Default)] + struct FakeTrailer(Mutex>>); + + impl TrailerSource for FakeTrailer { + fn lookup(&self, name: &str) -> TrailerValue { + match self.0.lock().expect("trailer lock").as_ref() { + None => TrailerValue::Pending, + Some(fields) => fields.get(name).cloned().map_or(TrailerValue::Missing, TrailerValue::Present), + } + } + } + + /// Body that publishes `fields` (when set) only once it reaches EOF. + struct PublishOnEof { + body: Cursor>, + trailer: Arc, + fields: Option>, + } + + impl AsyncRead for PublishOnEof { + fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll> { + let this = &mut *self; + let before = buf.filled().len(); + let result = Pin::new(&mut this.body).poll_read(cx, buf); + if matches!(result, Poll::Ready(Ok(()))) + && buf.filled().len() == before + && buf.remaining() > 0 + && let Some(fields) = this.fields.clone() + { + *this.trailer.0.lock().expect("trailer lock") = Some(fields); + } + result + } + } + + fn encoded(checksum_type: ChecksumType, data: &[u8]) -> String { + let mut hasher = checksum_type.hasher().expect("typed checksum hasher"); + hasher.write_all(data).expect("hash data"); + base64_simd::STANDARD.encode_to_string(hasher.finalize()) + } + + fn trailing_request(checksum_type: ChecksumType) -> HeaderMap { + let mut headers = HeaderMap::new(); + headers.insert( + "x-amz-trailer", + HeaderValue::from_static(checksum_type.key().expect("typed checksum key")), + ); + headers + } + + /// `published`: `None` never publishes (still `Pending` at EOF); + /// `Some(None)` publishes a trailer without the field; `Some(Some(v))` + /// publishes `v` under the checksum's key. + fn reader(checksum_type: ChecksumType, data: &[u8], published: Option>) -> (HashReader, Arc) { + let trailer = Arc::new(FakeTrailer::default()); + let key = checksum_type.key().expect("typed checksum key"); + let fields = published.map(|value| value.into_iter().map(|v| (key.to_string(), v)).collect()); + let body = PublishOnEof { + body: Cursor::new(data.to_vec()), + trailer: trailer.clone(), + fields, + }; + let size = data.len() as i64; + let mut reader = HashReader::from_stream(body, size, size, None, None, false).expect("hash reader"); + reader + .add_checksum(&trailing_request(checksum_type), Some(trailer.clone()), false) + .expect("declared trailing checksum"); + (reader, trailer) + } + + async fn read_all(reader: &mut HashReader) -> std::io::Result> { + let mut out = Vec::new(); + reader.read_to_end(&mut out).await.map(|_| out) + } + + fn assert_checksum_mismatch(err: &std::io::Error) { + let inner = err.get_ref().expect("typed error source"); + assert!(is_checksum_mismatch(inner), "expected ChecksumMismatch, got {err:?}"); + } + + #[tokio::test] + async fn trailing_checksum_published_at_eof_is_verified_and_reported() { + let data = b"trailing checksum payload"; + for checksum_type in TYPED { + let value = encoded(checksum_type, data); + let (mut reader, trailer) = reader(checksum_type, data, Some(Some(value.clone()))); + let key = checksum_type.key().expect("typed checksum key"); + assert_eq!( + trailer.lookup(key), + TrailerValue::Pending, + "{checksum_type}: attached before the body ends" + ); + + assert_eq!(read_all(&mut reader).await.expect("verified body"), data); + assert_eq!( + reader.content_crc(), + HashMap::from([(checksum_type.to_string(), value)]), + "{checksum_type}: verified trailer value is reported for persistence" + ); + } + } + + #[tokio::test] + async fn trailing_checksum_mismatch_is_rejected() { + let data = b"trailing checksum payload"; + for checksum_type in TYPED { + let wrong = encoded(checksum_type, b"different payload"); + let (mut reader, _) = reader(checksum_type, data, Some(Some(wrong.clone()))); + let err = read_all(&mut reader).await.expect_err("mismatched trailer must fail"); + assert_checksum_mismatch(&err); + let mismatch = err + .get_ref() + .and_then(|e| e.downcast_ref::()) + .expect("mismatch details"); + let wrong_raw = base64_simd::STANDARD.decode_to_vec(&wrong).expect("valid base64"); + assert_eq!(mismatch.want, hex_simd::encode_to_string(wrong_raw, hex_simd::AsciiCase::Lower)); + } + } + + #[tokio::test] + async fn trailing_checksum_missing_or_still_pending_at_eof_is_rejected() { + let data = b"trailing checksum payload"; + for published in [None, Some(None)] { + let (mut reader, _) = reader(ChecksumType::CRC32C, data, published.clone()); + let err = read_all(&mut reader).await.expect_err("unverifiable trailer must fail"); + assert_checksum_mismatch(&err); + assert!(reader.content_crc().is_empty(), "{published:?}: nothing to persist"); + } + + let size = data.len() as i64; + let mut reader = HashReader::from_stream(Cursor::new(data.to_vec()), size, size, None, None, false).expect("hash reader"); + reader + .add_checksum(&trailing_request(ChecksumType::CRC32C), None, false) + .expect("declared trailing checksum"); + let err = read_all(&mut reader).await.expect_err("no trailer source must fail"); + assert_checksum_mismatch(&err); + assert!(reader.content_crc().is_empty()); + } + + #[tokio::test] + async fn malformed_trailing_checksum_is_an_error_not_a_panic() { + let data = b"trailing checksum payload"; + for value in ["not base64!", ""] { + let (mut reader, _) = reader(ChecksumType::SHA256, data, Some(Some(value.to_string()))); + let err = read_all(&mut reader).await.expect_err("malformed trailer must fail"); + assert_eq!(err.kind(), std::io::ErrorKind::Other, "{value:?}: {err:?}"); + } + } + + #[tokio::test] + async fn wrapping_hash_reader_keeps_the_trailer_source() { + let data = b"trailing checksum payload"; + let value = encoded(ChecksumType::CRC32, data); + let size = data.len() as i64; + + let (inner, _) = reader(ChecksumType::CRC32, data, Some(Some(value.clone()))); + let mut outer = HashReader::new(Box::new(inner), size, size, None, None, false).expect("outer hash reader"); + assert!(outer.get_trailer().is_some(), "wrapping must carry the trailer source"); + assert_eq!(read_all(&mut outer).await.expect("verified body"), data); + assert_eq!(outer.content_crc(), HashMap::from([("CRC32".to_string(), value)])); + + let wrong = encoded(ChecksumType::CRC32, b"different payload"); + let (inner, _) = reader(ChecksumType::CRC32, data, Some(Some(wrong))); + let mut outer = HashReader::new(Box::new(inner), size, size, None, None, false).expect("outer hash reader"); + let err = read_all(&mut outer).await.expect_err("mismatch through a wrapper must fail"); + assert_checksum_mismatch(&err); + } +} diff --git a/crates/rio/src/lib.rs b/crates/rio/src/lib.rs index af874ce08..43e2d706b 100644 --- a/crates/rio/src/lib.rs +++ b/crates/rio/src/lib.rs @@ -119,6 +119,9 @@ pub use hardlimit_reader::HardLimitReader; mod hash_reader; pub use hash_reader::*; +mod trailer; +pub use trailer::{SharedTrailerSource, TrailerSource, TrailerValue}; + mod tee_reader; pub use tee_reader::{ DEFAULT_TEE_MAX_DRAIN_BYTES, TeeDrainLimitExceeded, TeeOptions, TeePrimary, TeeSecondary, TeeStream, tee_reader, diff --git a/crates/rio/src/trailer.rs b/crates/rio/src/trailer.rs new file mode 100644 index 000000000..6ec523548 --- /dev/null +++ b/crates/rio/src/trailer.rs @@ -0,0 +1,66 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Framework-neutral source of HTTP trailer fields, such as the +//! `x-amz-checksum-*` trailers of an aws-chunked upload. +//! +//! Timing contract: +//! 1. A source is attached to a `HashReader` while request headers are +//! processed, before any trailer can have arrived, so it starts `Pending`. +//! 2. The body decoder that owns the source publishes the trailer section +//! after the final chunk has been decoded and before the decoded body +//! reports EOF. +//! 3. `HashReader` reads the trailer only once its inner reader reports EOF, +//! and `HashReader::content_crc` is only meaningful after that point. +//! +//! Once a lookup has returned anything other than [`TrailerValue::Pending`], +//! the source must never report `Pending` again. +//! +//! `HashReader` currently treats `Pending` at EOF the same as `Missing`: the +//! declared trailing checksum fails verification at EOF and is omitted from +//! `content_crc`. + +use std::sync::Arc; + +/// Outcome of looking up one trailer field. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum TrailerValue { + /// The trailer section has not been received yet. + Pending, + /// The trailer section was received without a usable value for the field + /// (absent, or not visible ASCII). + Missing, + /// The field value exactly as received. + Present(String), +} + +impl TrailerValue { + /// Returns the received value, if any. + pub fn into_present(self) -> Option { + match self { + Self::Present(value) => Some(value), + Self::Pending | Self::Missing => None, + } + } +} + +/// Read-only view of a request's trailer section. See the module docs for +/// the timing contract implementations must follow. +pub trait TrailerSource: Send + Sync { + /// Looks up the lower-case field `name` in the trailer section. + fn lookup(&self, name: &str) -> TrailerValue; +} + +/// Shared handle to a [`TrailerSource`]. Clones observe the same trailer. +pub type SharedTrailerSource = Arc; diff --git a/crates/signer/src/lib.rs b/crates/signer/src/lib.rs index d802b0f32..8daa0366f 100644 --- a/crates/signer/src/lib.rs +++ b/crates/signer/src/lib.rs @@ -32,4 +32,5 @@ pub use request_signature_v4::sign_v4; pub use request_signature_v4::sign_v4_trailer; pub use request_signature_v4::try_pre_sign_v4; pub use request_signature_v4::try_sign_v4; +pub use request_signature_v4::try_sign_v4_headers; pub use request_signature_v4::try_sign_v4_trailer; diff --git a/crates/signer/src/request_signature_v4.rs b/crates/signer/src/request_signature_v4.rs index 1c82b9662..cd21f2008 100644 --- a/crates/signer/src/request_signature_v4.rs +++ b/crates/signer/src/request_signature_v4.rs @@ -679,6 +679,19 @@ pub fn try_sign_v4( .map_err(|failure| failure.error) } +pub fn try_sign_v4_headers( + parts: request::Parts, + content_len: i64, + access_key_id: &str, + secret_access_key: &str, + session_token: &str, + location: &str, +) -> SignResult { + let request = request::Request::from_parts(parts, Body::empty()); + try_sign_v4(request, content_len, access_key_id, secret_access_key, session_token, location) + .map(|request| request.into_parts().0.headers) +} + pub fn sign_v4_trailer( req: request::Request, access_key_id: &str, diff --git a/crates/storage-api/src/lib.rs b/crates/storage-api/src/lib.rs index 5ba4dc186..3315b0044 100644 --- a/crates/storage-api/src/lib.rs +++ b/crates/storage-api/src/lib.rs @@ -101,6 +101,10 @@ pub use object::{VersionMarker, WalkOptions, WalkVersionsSortOrder}; pub use observability::{ MemorySamplingState, ObservabilitySnapshot, ObservabilitySnapshotProvider, PlatformSupport, UserspaceProfilingCapability, }; +/// Object-metadata keys persisted in xl.meta `meta_user`. filemeta owns the +/// on-disk spelling; contract consumers that read or write persisted object +/// metadata (lifecycle object-lock checks) take the keys from here. +pub use rustfs_filemeta::metadata_keys; pub use topology::{ DiskCapabilities, TopologyCapabilities, TopologyDisk, TopologyLabels, TopologyPool, TopologySet, TopologySnapshot, TopologySnapshotProvider, diff --git a/crates/zip/Cargo.toml b/crates/zip/Cargo.toml index 312d2928d..c0b21eb0e 100644 --- a/crates/zip/Cargo.toml +++ b/crates/zip/Cargo.toml @@ -50,7 +50,7 @@ tokio = { workspace = true, features = ["io-util", "macros", "rt"] } thiserror = { workspace = true } [dev-dependencies] -astral-tokio-tar = { workspace = true } +rustfs-tokio-tar = { workspace = true } futures = { workspace = true } serde = { workspace = true, features = ["derive"] } serde_json = { workspace = true } diff --git a/deny.toml b/deny.toml index cfecab82f..cd645812d 100644 --- a/deny.toml +++ b/deny.toml @@ -36,18 +36,13 @@ unknown-registry = "deny" unknown-git = "deny" allow-registry = ["https://github.com/rust-lang/crates.io-index"] allow-git = [ - # Temporary tokio-tar fork pinned to the reviewed parser limits, - # cancellation safety, and error-fusing change while Snowball is - # prototyped against tar-codec and Swift retains its current reader. - # owner: cxymds review: 2026-10 - "https://github.com/cxymds/tokio-tar.git", # Official s3s repository. Temporarily pinned to the merged generic REST # SigV4 payload-checksum fix until it is available in a crates.io release. # owner: marshawcoco review: 2026-10 - "https://github.com/s3s-project/s3s.git", + # "https://github.com/s3s-project/s3s.git", # RustFS fork carrying presigned expiry and constant-time authentication fixes. # owner: rustfs-maintainers review: 2026-10 - "https://github.com/rustfs/s3s.git", + # "https://github.com/rustfs/s3s.git", ] [bans] diff --git a/docs/architecture/compat-cleanup-register.md b/docs/architecture/compat-cleanup-register.md index 275168f78..2643f7a1a 100644 --- a/docs/architecture/compat-cleanup-register.md +++ b/docs/architecture/compat-cleanup-register.md @@ -18,7 +18,6 @@ - `backlog-2263` legacy heal MRF inspection: retained per-record journals remain readable while committed-snapshot ownership and writer activation are staged. Remove legacy import only after all supported direct-upgrade and rollback readers understand committed snapshots and migration tooling confirms that no retained or restorable legacy journal requires it. This does not enable a new writer or change the automatic legacy consumer. - `backlog-1337` legacy restore orphan recovery: releases that predate the restore worker-lock marker can leave a valid operation-id and `ongoing-request="true"` after cancellation or process failure, with no durable liveness proof. New servers allow an exact, non-nil legacy generation to be superseded only when its consistently parsed request date is at least 24 hours old. Remove the clock-based legacy fallback after the minimum supported direct-upgrade release writes the v1 worker-lock marker on every restore and operators have resolved every retained pre-v1 ongoing generation. - `backlog-2133-tier-delete-chunk-parent` bounded tier-delete dispatch compatibility: prefixes at or below the legacy manifest limit keep the byte-compatible v1 single-manifest protocol, while larger prefixes place a chunk-parent sentinel at the original deterministic root path and use operation-scoped child manifests. Older binaries reject the sentinel and child paths, preserving the v6 sole-owner downgrade fence instead of starting a competing local delete. Remove the v1 reader and fail-closed mixed-version sentinel only after every supported rollback release validates the parent/child protocol and migration tooling confirms that no retained v1 dispatch manifest remains. -- `tokio-tar-extension-limits` bounded archive parser hardening: Snowball extraction depends on precedence-resolved MinIO PAX metadata; per-entry and cumulative extension limits; a physical-entry limit; cancellation-safe parsing and ownership of large streamed members; fused streams after errors; and compatibility with minio-go streams that omit the two-block terminator. Swift bulk extraction also uses the same fork. Keep the reviewed pin while the Snowball path is prototyped against tar-codec/tar-framing. Remove it only after a released API exposes the effective allowed vendor records, RustFS provides a cancellation-safe handoff for borrowed member payloads, footerless input is accepted solely when authenticated request framing proves EOF immediately after a complete member, the existing resource-limit, cancellation, error-fuse, and real minio-go fixtures pass against the replacement, and Swift no longer depends on the fork. - `backlog-2102` rc.2/rc.3 empty scanner usage floor recovery: old DeleteBucket cleanup could synthesize an empty incomplete v2 usage primary/backup before leadership added an epoch, while newer scanners require a durable authoritative baseline identity. New scanners recognize only that exact serialized empty-fence shape, preserve its epoch through a CAS-protected recovery marker, and rebuild namespace coverage without treating zero usage as authoritative. Remove this recovery path and marker after rc.2 and rc.3 are no longer supported direct-upgrade sources. - `backlog-2122` rc.1-rc.3 non-empty scanner usage floor recovery: leadership fencing in those releases can stamp scanner_epoch onto a real bucket-usage snapshot before any scanner cycle completed, leaving a non-empty floor with no scanner_cycle and no authoritative baseline identity. New scanners recognize only this consistent incomplete fenced shape, preserve the epoch through the CAS-protected recovery marker, and rebuild namespace coverage without treating the old usage data as authoritative. Remove this recovery path after rc.1, rc.2, and rc.3 are no longer supported direct-upgrade sources. - `s3gate-metadata-xml` persisted bucket XML migration: mixed-version site-replication peers, retained `.metadata.bin` objects, and backup archives can all carry XML written by the s3s codec, so the gateway migration must keep the legacy codec available until every stored form has crossed a verified rewrite boundary. Remove the legacy s3s parser and serializer only after the minimum supported direct-upgrade release reads and writes every persisted XML configuration family through the gateway codec, every supported mixed-version site-replication topology has completed its writer upgrade, and migration tooling has verified or rewritten every retained bucket metadata object and restorable backup archive. @@ -51,6 +50,7 @@ - `backlog-2097-tier-mutation-v4-error-text` tier-mutation Prepare rejection classification: a v3 server rejects a v4 request before store/runtime dispatch with the FailedPrecondition status and an authenticated, byte-exact unsupported-version message. A v4 coordinator recognizes only that exact code/message/requested-version tuple as definitely not persisted and fails the mutation without sending that peer an incompatible Abort; Unimplemented, near-text, missing/unknown failure classes, timeouts, and every other transport outcome remain ambiguous and stay in identity-bound Abort fanout. There is no automatic v3 retry. New servers retain v3 request/proof decoding for older coordinators, while operators must pause tier edit/remove/clear during a mixed v3/v4 rollout. Remove the text classifier after the minimum supported RustFS peer version returns the signed v4 PreDispatchRejected failure class. - `backlog-2097-tier-delete-journal-v6` tier-delete sole-owner recovery: v6 distinguishes transactions that may replace the xl.meta free-version owner and therefore require the all-pool live-source proof. v5-and-older readers reject and retain v6 records during rolling upgrades instead of performing an unsafe remote delete. Keep v1-v5 readers for upgrade recovery and keep the downgrade prohibition while any v6 record exists; retire the fence only after every supported rollback release understands and enforces v6 proof semantics. - `multipart-compression-default-off-window` staged multipart disk-compression rollout: releases before the resumable legacy decompressor fail transient reads of compressed objects under mid-payload suspension, so multipart uploads advertise the compression marker only when RUSTFS_COMPRESSION_MULTIPART_ENABLED is set in addition to RUSTFS_COMPRESSION_ENABLED, keeping rolling upgrades from creating new compressed multipart objects while pre-fix nodes may still serve reads. Flip the default to enabled (and retire the extra switch) after the minimum supported direct-upgrade release ships the resumable decompressor. +- `s3gate-trailer-adapter` aws-chunked trailer handle bridge: s3s decodes aws-chunked request bodies and publishes their x-amz-checksum-* trailers through its own handle, while rio consumes trailers only through its framework-neutral TrailerSource trait. The RustFS application crate adapts the s3s handle at the object write path so rio carries no s3s dependency. Remove the adapter after the gateway stack replaces s3s as the request body decoder and publishes trailers through its own TrailerSource implementation. ## Review Checklist diff --git a/docs/testing/ci-timing.md b/docs/testing/ci-timing.md new file mode 100644 index 000000000..2b001d197 --- /dev/null +++ b/docs/testing/ci-timing.md @@ -0,0 +1,9 @@ +# CI timing samples + +Collect a bounded sample of completed PR CI runs created in the last seven days with `python3 scripts/ci_timing_report.py --limit 30 --output /tmp/ci-timing.json`. This requires an authenticated `gh` CLI with Actions read access. The JSON retains run SHA, attempt, job and step timestamps so another reviewer can reproduce the summary with `--input /tmp/ci-timing.json --output /tmp/ci-timing-summary.json` without GitHub access. + +The report separates job creation-to-start wait, job execution, and individual step durations. The successful-code sample excludes documentation-only runs and successful PR-closure cancellation handlers. It uses the selected attempt's start for reruns, includes parallel jobs in the runner-minute sum, and leaves missing timestamps unknown. It does not estimate compiler time from a combined build-and-test step or treat runner minutes as wall time or a bill. + +A small recent sample may contain no complete successful code runs. In that case its median is absent, not zero. Keep cancelled and failed counts visible; do not replace the sample with only green runs when evaluating changes. + +Before reducing a PR lane, compare a proposed path classifier in shadow mode against the existing full selection. Keep that lane required until equivalent nightly evidence is complete and current for the same source policy. Timing data alone does not establish functional coverage, escaped-regression rate or quarantine health. The 30–45 minute PR target in backlog #2483 remains an experiment to measure, not an acceptance result from this tool. diff --git a/docs/testing/functional-chain.md b/docs/testing/functional-chain.md new file mode 100644 index 000000000..f567cb1fc --- /dev/null +++ b/docs/testing/functional-chain.md @@ -0,0 +1,29 @@ +# Functional chain evidence + +The functional-chain driver calls twelve reusable suite workflows sequentially in one Actions run. A suite failure does not suppress later suites. `complete-chain` requires every suite job and its evidence artifact to succeed. The root uses its own concurrency group so an active chain can finish; the suites retain the shared VM lock used by standalone tests. + +## Candidate identity + +The driver resolves one successful `nightly-gnu.yml` attempt from main. Automatic runs select the triggering scheduled attempt. Manual runs require an explicit build run ID and attempt. Resolution verifies the artifact name, run association, ZIP size and digest, single JSON member, and immutable package URL containing run, attempt and package checksum. + +Schema 2 distinguishes the workflow SHA from the actual build SHA and source ref. A release build triggered by a main workflow remains release evidence. Legacy schema 1 is accepted only when its source SHA equals the producer workflow SHA. The consumer does not resolve the source branch again, so branch movement cannot silently select a different package. + +All suites use the same candidate package and checksum. Installers with checksum support receive `PACKAGE_SHA256`, or `TO_SHA256` for upgrade, and verify the package before installation. Table and fault-tolerance installers have no checksum option, so `scripts/prepare_functional_package.py` first downloads and verifies the package on the runner. It transfers those bytes to a root-owned cache on each configured node, checks the SHA256 again before atomic publication, and supplies a `file://` URL to the existing installer. The cache is isolated by chain run and attempt and removed after the suite, including failed attempts. Fetch, hash, transfer, or cleanup failure prevents valid chain evidence. + +The private test repository is checked out at `.config/functional-script-revision.txt`; change that pin only to a reviewed, merged revision. Standalone table and fault-tolerance runs retain their existing package inputs and report policy. + +## Completion and reruns + +The required order is upgrade, S3, KMS, tier, storage, heal, pool expansion, security, replication, fault tolerance, table, and performance. Fault-tolerance evidence requires a final summary matching its per-probe verdicts; its existing non-strict known-divergence verdicts are reported as unsupported cases. + +Every suite records its chain run/attempt, workflow SHA, private pin, candidate identity, report hash and execution counts. A missing/empty report, zero passing executions, failed or unfinished case, failed test/report step, cancelled job, or mismatched private checkout invalidates the evidence. Uploading the suite proof requires successful proof generation. + +The final job checks all twelve expected suite results and all twelve proof files against the same envelope. It emits `functional-chain-complete--` only after those checks pass. Failed partial reruns cannot combine an old successful lane's proof with a new attempt. Use **Re-run all jobs** for a new complete acceptance attempt. + +## Health publication + +`functional-chain-health.yml` inspects recent main-branch chain runs hourly. It validates complete evidence against the exact producer artifact again and checks the private pin from the chain's workflow commit. Its JSON separates the latest attempt from the last complete success for each source. A later failure preserves historical success without turning the new failure green. + +Evidence expires 36 hours after the producer attempt started. The dashboard also treats collection older than two hours as stale. Workflow enablement, owner, source identities, evidence version and expiry are visible. An invalid legacy chain-driver success is not complete evidence, and release success cannot authorize moving main PR coverage to nightly. + +The dashboard's `src/chain-health.json` must exist before enabling publication. Updates use the read blob SHA and reject unsupported, null or newer existing state. The companion dashboard view is required to display this data. Until a real same-candidate chain completes, these workflow and script checks do not satisfy backlog #2481 or unblock coverage migration in #2483. diff --git a/flake.lock b/flake.lock index 9fc362d30..12ca3e340 100644 --- a/flake.lock +++ b/flake.lock @@ -2,11 +2,11 @@ "nodes": { "nixpkgs": { "locked": { - "lastModified": 1788549839, - "narHash": "sha256-kOrCcSIA6w9J1hX5DqHy2k9pDTJymExTsbV74U9UtCA=", + "lastModified": 1789073787, + "narHash": "sha256-xfX/toC2QV707s06GbP4II/TxYF0fNQj7s5/LClNDKc=", "owner": "NixOS", "repo": "nixpkgs", - "rev": "17de0b976395537756f30a3e78f2f06e5cec89ed", + "rev": "aff8a0b28396750446e5537a96461bc4facdb287", "type": "github" }, "original": { @@ -29,11 +29,11 @@ ] }, "locked": { - "lastModified": 1788591095, - "narHash": "sha256-Vh+BeLWfbTT9AecazIsQ/Tkg/RzJeX3lEduANf256WA=", + "lastModified": 1789196581, + "narHash": "sha256-yJr1Bt4fKkKIpPYbsKGqJ0VFdoDnURgRwuffmBQ2WzY=", "owner": "oxalica", "repo": "rust-overlay", - "rev": "c361047d3a538f547f1617bb6b410411929ac9cc", + "rev": "228ecefb6329d5a531b77b46b581a2f0c26ee056", "type": "github" }, "original": { diff --git a/protocol/agent/v1/fixtures/bundle/MANIFEST.sha256 b/protocol/agent/v1/fixtures/bundle/MANIFEST.sha256 index 0cbb72354..2b0fd7e86 100644 --- a/protocol/agent/v1/fixtures/bundle/MANIFEST.sha256 +++ b/protocol/agent/v1/fixtures/bundle/MANIFEST.sha256 @@ -2,3 +2,4 @@ afe10983a0de23cf2e1400399bb8a757de90cff2c6120d90cd83b0d28bc2cad2 error-codes.json 22133a5cbd5cd36588987d3540c9cadde063faa0ea529dc913f19ed2dc253dea manifest-signing.json 04aedbaacdac72fa2a0df22edf6a8c402645a972713778c6031d6c643976a14c reject-vectors.json +7d1a84f83f1075401e23b0c5fcf27b86ed669bcadd080079544419409cb1e905 typed-offline-import-vectors.json diff --git a/protocol/agent/v1/fixtures/bundle/typed-offline-import-vectors.json b/protocol/agent/v1/fixtures/bundle/typed-offline-import-vectors.json new file mode 100644 index 000000000..432e40d43 --- /dev/null +++ b/protocol/agent/v1/fixtures/bundle/typed-offline-import-vectors.json @@ -0,0 +1,550 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "bundle", + "fixture": "typed-offline-import-vectors", + "description": "Signed receiver-side compatibility vectors for exact typed offline diagnostic archives. These are contract fixtures, not evidence that a producer exists or ran.", + "domainSeparationTags": { + "manifest": "rustfs-support-bundle-v1", + "diagnosticEnvelope": "rustfs-diagnostic-envelope-v1" + }, + "evaluationTime": "2026-09-13T12:00:00Z", + "authorizedBundle": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000003", + "publicKey": "BN18AJq1Zx2KyCPAKIgx9QY4qcr-1RBOxBj8BACTGF4rRtUSgSWSw2nJP8Y11Bziw6REtEy90FT003jCkQChuBU", + "keyId": "3192932cae8bbc8a5734a3605fa0e4430d8291de0fea389d28e831faf3ae8aa2" + }, + "vectors": [ + { + "toolId": "health.check", + "classification": "L1", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMmMiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJGeWk1V2V2NDZNWmdOQ2Y3ODdPSnBhWTlXRkM1aS11SzJQcnlFRE9wWGJjIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjUsInNoYTI1NiI6IjQ5ODQ0ZGE3MzgxN2FjNDhhZmY2NGY0NzNhODczMWQxMWIxYjYyNmZiNDRjZDA3NDk4ZTc4ZDY2YzcwZGJkMDciLCJjbGFzc2lmaWNhdGlvbiI6IkwxIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiJmZjRhNDA5OTY0OGE4M2E3NGEwMmU1OWUxNzkzMThmMTQ0YzM3NDNlNjhmNTM2YjI4ZGM0NjVjY2IxNjJiYmNjIiwiY2xhc3NpZmljYXRpb24iOiJMMSJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjUzLCJzaGEyNTYiOiI2ZGZkNjk3OWZhZTY5NjU3OTIzNTE1ZDI3NDlkOWI4MTNhNjFmMjY5YjMzZWVmZWNhZjFjMjM0MjYwMDQ0Nzk3IiwiY2xhc3NpZmljYXRpb24iOiJMMSJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiQW5nNmEwMm0xZXBCS2ppbWEweFo3RGpqMUtiV2ktakhxdUowY2lOZS0yZG5aaU00cWxMa3U5TG5ZSHB1Q29vZXB4ZXdIMjViTlJfZDhhMzJBLVJyZUEifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjQiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjOCIsInRvb2xJZCI6ImhlYWx0aC5jaGVjayIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDEiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiJGeWk1V2V2NDZNWmdOQ2Y3ODdPSnBhWTlXRkM1aS11SzJQcnlFRE9wWGJjIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2NTMsInNoYTI1NiI6IjZkZmQ2OTc5ZmFlNjk2NTc5MjM1MTVkMjc0OWQ5YjgxM2E2MWYyNjliMzNlZWZlY2FmMWMyMzQyNjAwNDQ3OTcifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiVjRzWURmeU5zc19ySDZLanUtSGFWZVF6RklObFFLclU0ZDRJRXhGZVRNbGcxcGY1dFNBNU5JLTI3TVRsZGR0akZPS2xSenlfbmRHWHV2dGwtNzNaRUEifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjQiLCJ0b29sSWQiOiJoZWFsdGguY2hlY2siLCJjYXBhYmlsaXR5IjoiaGVhbHRoLmNoZWNrQDEiLCJvdXRjb21lIjoiU1VDQ0VFREVEIiwicmVhc29uQ29kZSI6IkNPTVBMRVRFIiwiZHVyYXRpb25NaWxsaXMiOjEwMDAsInByb3ZlbmFuY2UiOnsicmVwb3NpdG9yeSI6InJ1c3Rmcy9ydXN0ZnMiLCJzb3VyY2VDb21taXQiOiJhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhIiwiZXhlY3V0YWJsZVNoYTI1NiI6ImJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmIiLCJydXN0ZnNWZXJzaW9uIjoiMS4wLjAtcmMuNiIsIm9zRmFtaWx5IjoiTElOVVgiLCJhcmNoaXRlY3R1cmUiOiJhYXJjaDY0IiwiYnVpbGRGZWF0dXJlcyI6W119LCJjb3ZlcmFnZSI6eyJyZXF1ZXN0ZWRVbml0cyI6MSwiY29tcGxldGVkVW5pdHMiOjEsInVuaXQiOiJDSEVDSyJ9LCJkYXRhIjp7ImNhdGFsb2dWZXJzaW9uIjoxLCJjaGVja3MiOlt7ImNoZWNrSWQiOiJjYXBhY2l0eS5leGhhdXN0ZWQiLCJvdXRjb21lIjoiRkFJTCIsInJlc291cmNlQWxpYXMiOiJyZXNvdXJjZS0xIn1dfX0=" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjQiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjOCIsInRvb2xJZCI6ImhlYWx0aC5jaGVjayIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDEiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiJGeWk1V2V2NDZNWmdOQ2Y3ODdPSnBhWTlXRkM1aS11SzJQcnlFRE9wWGJjIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2NTMsInNoYTI1NiI6IjZkZmQ2OTc5ZmFlNjk2NTc5MjM1MTVkMjc0OWQ5YjgxM2E2MWYyNjliMzNlZWZlY2FmMWMyMzQyNjAwNDQ3OTcifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiQkN0Tks5YVpOUHRGZlp0LTE3TUZBVkVwRFQ5NFZFOF9uckF2NEdyaTd0c0N1cFM1OHo0NDVNM0tGTnFDN1NVQ3FhWS1ra0dkeFFZTnhKZTJaMUlzV0EifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "performance.network", + "classification": "L1", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMmQiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiI2RlA0WWtqRU1zS3dpbXNPUWMzXzVLcG9tdDFFSl84V3F2VF9UXzY2X0lRIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMzIsInNoYTI1NiI6ImFhOGQyNDFlZjAyNjVlYWU1NWYwNzIwMzMxZWY2MzQ4OWVhNDE1MmVkZmUzNWRmYzVmOWYxNTgzZTdkZjAwM2EiLCJjbGFzc2lmaWNhdGlvbiI6IkwxIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiJlZjRlZDE1ZGIxOTIzMjdmYjNmYmM2NmQyZTY0OWVjZTVhNDBlMjVlM2ExZjEyMjQ0NjdmNGUxYjNlZjlhNWFjIiwiY2xhc3NpZmljYXRpb24iOiJMMSJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjM3LCJzaGEyNTYiOiI1NDM0NTlkNGE2MTUwZDNlNmI4YTMzODQ1MDNmY2MyZTFhMmMxNmNhZmExODdlMDBhZWM1MGY1NmNjMTA3MWQyIiwiY2xhc3NpZmljYXRpb24iOiJMMSJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiQnNlNk1GM19hSHMyQlJKUU9LRENLLU1JQ0R2ZEFZbHQ0VVp1c2Vlbm1EOWh4NkthNXNPVVE5SGRqQXkteFFlSEFUNU9PTV9XM3VXS1FnNllCV1JEUWcifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjUiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjOSIsInRvb2xJZCI6InBlcmZvcm1hbmNlLm5ldHdvcmsiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwxIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoiNkZQNFlrakVNc0t3aW1zT1FjM181S3BvbXQxRUpfOFdxdlRfVF82Nl9JUSIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NjM3LCJzaGEyNTYiOiI1NDM0NTlkNGE2MTUwZDNlNmI4YTMzODQ1MDNmY2MyZTFhMmMxNmNhZmExODdlMDBhZWM1MGY1NmNjMTA3MWQyIn19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiX1FuLVh0QnUzdXZ2MjBrMFlMVDhra01saEdFekxFRXZ3MEVFZUVGTVFvVlhLSHVnaTV1Ujl1dHUxMS0tNF9ObTZJT1FCcEJBbDVQd2txTW5nM0JOdVEifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjUiLCJ0b29sSWQiOiJwZXJmb3JtYW5jZS5uZXR3b3JrIiwiY2FwYWJpbGl0eSI6InBlcmZvcm1hbmNlLm5ldHdvcmtAMSIsIm91dGNvbWUiOiJTVUNDRUVERUQiLCJyZWFzb25Db2RlIjoiQ09NUExFVEUiLCJkdXJhdGlvbk1pbGxpcyI6MTAwMCwicHJvdmVuYW5jZSI6eyJyZXBvc2l0b3J5IjoicnVzdGZzL3J1c3RmcyIsInNvdXJjZUNvbW1pdCI6ImFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWEiLCJleGVjdXRhYmxlU2hhMjU2IjoiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYiIsInJ1c3Rmc1ZlcnNpb24iOiIxLjAuMC1yYy42Iiwib3NGYW1pbHkiOiJMSU5VWCIsImFyY2hpdGVjdHVyZSI6ImFhcmNoNjQiLCJidWlsZEZlYXR1cmVzIjpbXX0sImNvdmVyYWdlIjp7InJlcXVlc3RlZFVuaXRzIjoxLCJjb21wbGV0ZWRVbml0cyI6MSwidW5pdCI6IldJTkRPVyJ9LCJkYXRhIjp7InRyYW5zZmVycmVkQnl0ZXMiOjEwNDg1NzYsImR1cmF0aW9uTWlsbGlzIjoxMDAwLCJlcnJvckNvdW50IjowLCJwZWVyQ291bnQiOjJ9fQ==" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjUiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjOSIsInRvb2xJZCI6InBlcmZvcm1hbmNlLm5ldHdvcmsiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwxIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoiNkZQNFlrakVNc0t3aW1zT1FjM181S3BvbXQxRUpfOFdxdlRfVF82Nl9JUSIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NjM3LCJzaGEyNTYiOiI1NDM0NTlkNGE2MTUwZDNlNmI4YTMzODQ1MDNmY2MyZTFhMmMxNmNhZmExODdlMDBhZWM1MGY1NmNjMTA3MWQyIn19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiV05CZ2trLUtFWlF3ay01MzRBdzBOYTZyZWtIUmVCQjU0Vmt3Y1NfUkVnMThJemdFTVBZSndQYjFTS2Q2OXhvZ2FlQXF6MzVwNnFCNl9ONkE5QUZvNHcifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "performance.drive", + "classification": "L1", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMmUiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJkWEFjV19EcW81a2praW4xa0llOW9ybU1Xb3VYYTBScC1zV2Q4WFZCN0ZJIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMzAsInNoYTI1NiI6ImY2YWI5NTVhYmI0MmQ2MDZiMjcwNmFhZTc3ZmIzYjZkZTAwZTFhNDNmM2NkMTQzYzA1MmM1YjcwZDYxYmFkNDciLCJjbGFzc2lmaWNhdGlvbiI6IkwxIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiI0OWM3YzhlYjM0YTkwNjZkNWJkMmJiNTk0N2QzNDNhYmNmODA3YTNjODgzNGI4NzliOTE5YmVmZmM3MGJlODZmIiwiY2xhc3NpZmljYXRpb24iOiJMMSJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjQ3LCJzaGEyNTYiOiJiOWExMWM0MzY5NDZhYjllYjcwZGQzNmYyMWMxZDk1ODMxZTQwYTE1NjEzNmQ2OTI5NTljODI5YmZhNzQxZTYzIiwiY2xhc3NpZmljYXRpb24iOiJMMSJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiWjVXVExJUGpRX0xUTGhYczhBZ2duTHJ1UHAwYUlGajI1LUlwdzIxVW16b1BDanA3QTdidTBlaVNHeV9wSkZDbTFvUWVuX18tZTdiMzhzZXdVcnFfVGcifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjYiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjYSIsInRvb2xJZCI6InBlcmZvcm1hbmNlLmRyaXZlIiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMSIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6ImRYQWNXX0RxbzVramtpbjFrSWU5b3JtTVdvdVhhMFJwLXNXZDhYVkI3RkkiLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjY0Nywic2hhMjU2IjoiYjlhMTFjNDM2OTQ2YWI5ZWI3MGRkMzZmMjFjMWQ5NTgzMWU0MGExNTYxMzZkNjkyOTU5YzgyOWJmYTc0MWU2MyJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiRW1fSXpZNEpaMnJtT3ZfeUhKZXdwMXV0S3lveVNoR2hDMU4xTUloUzNEOW9faTV0T0owMXZfZWJxR0twLU5Kc2pUc1NhMDBDV0F2SmVTMm9CN09ZLUEifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjYiLCJ0b29sSWQiOiJwZXJmb3JtYW5jZS5kcml2ZSIsImNhcGFiaWxpdHkiOiJwZXJmb3JtYW5jZS5kcml2ZUAxIiwib3V0Y29tZSI6IlNVQ0NFRURFRCIsInJlYXNvbkNvZGUiOiJDT01QTEVURSIsImR1cmF0aW9uTWlsbGlzIjoxMDAwLCJwcm92ZW5hbmNlIjp7InJlcG9zaXRvcnkiOiJydXN0ZnMvcnVzdGZzIiwic291cmNlQ29tbWl0IjoiYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYSIsImV4ZWN1dGFibGVTaGEyNTYiOiJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiIiwicnVzdGZzVmVyc2lvbiI6IjEuMC4wLXJjLjYiLCJvc0ZhbWlseSI6IkxJTlVYIiwiYXJjaGl0ZWN0dXJlIjoiYWFyY2g2NCIsImJ1aWxkRmVhdHVyZXMiOltdfSwiY292ZXJhZ2UiOnsicmVxdWVzdGVkVW5pdHMiOjEsImNvbXBsZXRlZFVuaXRzIjoxLCJ1bml0IjoiV0lORE9XIn0sImRhdGEiOnsicmVhZEJ5dGVzIjoxMDQ4NTc2LCJ3cml0ZUJ5dGVzIjoxMDQ4NTc2LCJpb0NvdW50Ijo1MTIsImR1cmF0aW9uTWlsbGlzIjoxMDAwLCJlcnJvckNvdW50IjowfX0=" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjYiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjYSIsInRvb2xJZCI6InBlcmZvcm1hbmNlLmRyaXZlIiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMSIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6ImRYQWNXX0RxbzVramtpbjFrSWU5b3JtTVdvdVhhMFJwLXNXZDhYVkI3RkkiLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjY0Nywic2hhMjU2IjoiYjlhMTFjNDM2OTQ2YWI5ZWI3MGRkMzZmMjFjMWQ5NTgzMWU0MGExNTYxMzZkNjkyOTU5YzgyOWJmYTc0MWU2MyJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoicUlRaUNXZHZSS2ZwNTI3bVpVQmFQZXppWTcwYlo2SG5PdG0zUEtYVjdRbDJ5VDNyRFdRTGJPQVdZUWxiVEQ3cTItVV92dE5KcWtwWTRUbzB0a0sxLUEifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "performance.object", + "classification": "L1", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMmYiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJkU2ozbnVaWTFGYXUyTDRlQ04yaDVDT3gwai1peGFpR1h4Q19pcnNCT0hVIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMzEsInNoYTI1NiI6IjY5N2EyZDVhODEyOTM5MWRjMmU4NTdjZjUzYWIwNzRjMTljMWFjMTMyNjIzZjcwZjAxYWE0NzBkNTY4MDEwNDIiLCJjbGFzc2lmaWNhdGlvbiI6IkwxIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiI3ZTExY2MxNmUwMmMwNWQ0NDMxNTA2ZDU4NjBlZTQ0MjZmMWI4NWFkNDljMDM4MmM4YTEwNzFmNjRiNGE2NDRkIiwiY2xhc3NpZmljYXRpb24iOiJMMSJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjcwLCJzaGEyNTYiOiJmMGY1MDY0ZjNjNGRhYTYwZWJkNjVlMDQ3NDA5MTcxY2M1ODc0YWQ5NDlhMjBjMDMyNWZlNzJiYzc0MGRhNDI2IiwiY2xhc3NpZmljYXRpb24iOiJMMSJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiNlN2Y2wzYy1wUnlBRzNGOWF3WmxDa2UxVm4wSl9VUVliWFNPUTVuOC1Xb1R0RmtUQm5laFBCY2R1M3V4a2dUZEROTi1pMWNhcWQ0OFFhZ3VpTzNnMlEifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjciLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjYiIsInRvb2xJZCI6InBlcmZvcm1hbmNlLm9iamVjdCIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDEiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiJkU2ozbnVaWTFGYXUyTDRlQ04yaDVDT3gwai1peGFpR1h4Q19pcnNCT0hVIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2NzAsInNoYTI1NiI6ImYwZjUwNjRmM2M0ZGFhNjBlYmQ2NWUwNDc0MDkxNzFjYzU4NzRhZDk0OWEyMGMwMzI1ZmU3MmJjNzQwZGE0MjYifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoidWhqYmxaeUNmOUlvZW1BUmUxZ0U0ZllLc1Y5ZjMtYnFuWERJMnJRWlZDeEE1aDU2OU54ZlNfNURBUjRHcm9nVUNjSk9vMU1JNG82ckdya043UDRTVFEifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjciLCJ0b29sSWQiOiJwZXJmb3JtYW5jZS5vYmplY3QiLCJjYXBhYmlsaXR5IjoicGVyZm9ybWFuY2Uub2JqZWN0QDEiLCJvdXRjb21lIjoiU1VDQ0VFREVEIiwicmVhc29uQ29kZSI6IkNPTVBMRVRFIiwiZHVyYXRpb25NaWxsaXMiOjEwMDAsInByb3ZlbmFuY2UiOnsicmVwb3NpdG9yeSI6InJ1c3Rmcy9ydXN0ZnMiLCJzb3VyY2VDb21taXQiOiJhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhIiwiZXhlY3V0YWJsZVNoYTI1NiI6ImJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmIiLCJydXN0ZnNWZXJzaW9uIjoiMS4wLjAtcmMuNiIsIm9zRmFtaWx5IjoiTElOVVgiLCJhcmNoaXRlY3R1cmUiOiJhYXJjaDY0IiwiYnVpbGRGZWF0dXJlcyI6W119LCJjb3ZlcmFnZSI6eyJyZXF1ZXN0ZWRVbml0cyI6MSwiY29tcGxldGVkVW5pdHMiOjEsInVuaXQiOiJXSU5ET1cifSwiZGF0YSI6eyJvcGVyYXRpb24iOiJHRVRfT0JKRUNUIiwidHJhbnNmZXJyZWRCeXRlcyI6MTA0ODU3NiwiY29tcGxldGVkT3BlcmF0aW9ucyI6MSwiZHVyYXRpb25NaWxsaXMiOjEwMDAsImVycm9yQ291bnQiOjB9fQ==" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjciLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjYiIsInRvb2xJZCI6InBlcmZvcm1hbmNlLm9iamVjdCIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDEiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiJkU2ozbnVaWTFGYXUyTDRlQ04yaDVDT3gwai1peGFpR1h4Q19pcnNCT0hVIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2NzAsInNoYTI1NiI6ImYwZjUwNjRmM2M0ZGFhNjBlYmQ2NWUwNDc0MDkxNzFjYzU4NzRhZDk0OWEyMGMwMzI1ZmU3MmJjNzQwZGE0MjYifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiazRaS0FXWDZlVUdkaEt5SkRyY0dmSVZxeVcxUUNnVTVGTkpjWHRfX3NVc01BTHhTdlIxcnNzLW1xWnV3ZWxBMXFRYWRVTFlzWHNVR0pWTzZuT2oxeEEifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "performance.client", + "classification": "L1", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMzAiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiIxRGhlc0NMTTIxV0RZVGRCd0pfTEFuQVpZVGFYTXRWY2lvdV9Zb0dKY0RRIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMzEsInNoYTI1NiI6IjQxZTc1NzhlODJlMjQwYWYyMWUwMWI2NWU2ZGE0YWFhMzBjNzI2MjI3MTE4OWQzNzFhOWE1NDRjYThjYjA5NTYiLCJjbGFzc2lmaWNhdGlvbiI6IkwxIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiIwYWE2OTRiNzk0OTg5MjdmNzBmYzAxODRmMDQ0ODc0ZDI3ODhiOGYwZmZlMzM0ZTM2ODUyYjRmY2I1ODAxYWU5IiwiY2xhc3NpZmljYXRpb24iOiJMMSJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjcwLCJzaGEyNTYiOiIyZmYwZWZlMjJiYzIxMTJhZWMxM2NkNWM0OWRmZjUxNTFjYTU0NTk5ZmNhZTFkMzdiNmExOTA3ZTM4NDc0ZWEzIiwiY2xhc3NpZmljYXRpb24iOiJMMSJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiTDVlVFhUVmQ4SFdUZm42T2FvUWdqRDcyRkZlbUc0MW9YVGx4MVMxQWdqWW1kQnB0MFJNd2s1MFV2Zzg5SGNmZ2gtWlpkc25BOWlBVHBManNQb1V2S0EifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjgiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjYyIsInRvb2xJZCI6InBlcmZvcm1hbmNlLmNsaWVudCIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDEiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiIxRGhlc0NMTTIxV0RZVGRCd0pfTEFuQVpZVGFYTXRWY2lvdV9Zb0dKY0RRIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2NzAsInNoYTI1NiI6IjJmZjBlZmUyMmJjMjExMmFlYzEzY2Q1YzQ5ZGZmNTE1MWNhNTQ1OTlmY2FlMWQzN2I2YTE5MDdlMzg0NzRlYTMifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiV3RPaC1KcEQ5Q002VEtSLWZFWlVvdFdQQzVyQmt1Q3lsdHphRzZlOFJjOUhSNmRmSVNKSDVaUWxvaGRGVHp5dXJLLWJ3bEJVOU1meFFpVTRvcUZxcGcifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjgiLCJ0b29sSWQiOiJwZXJmb3JtYW5jZS5jbGllbnQiLCJjYXBhYmlsaXR5IjoicGVyZm9ybWFuY2UuY2xpZW50QDEiLCJvdXRjb21lIjoiU1VDQ0VFREVEIiwicmVhc29uQ29kZSI6IkNPTVBMRVRFIiwiZHVyYXRpb25NaWxsaXMiOjEwMDAsInByb3ZlbmFuY2UiOnsicmVwb3NpdG9yeSI6InJ1c3Rmcy9ydXN0ZnMiLCJzb3VyY2VDb21taXQiOiJhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhIiwiZXhlY3V0YWJsZVNoYTI1NiI6ImJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmIiLCJydXN0ZnNWZXJzaW9uIjoiMS4wLjAtcmMuNiIsIm9zRmFtaWx5IjoiTElOVVgiLCJhcmNoaXRlY3R1cmUiOiJhYXJjaDY0IiwiYnVpbGRGZWF0dXJlcyI6W119LCJjb3ZlcmFnZSI6eyJyZXF1ZXN0ZWRVbml0cyI6MSwiY29tcGxldGVkVW5pdHMiOjEsInVuaXQiOiJXSU5ET1cifSwiZGF0YSI6eyJvcGVyYXRpb24iOiJQVVRfT0JKRUNUIiwidHJhbnNmZXJyZWRCeXRlcyI6MTA0ODU3NiwiY29tcGxldGVkT3BlcmF0aW9ucyI6MSwiZHVyYXRpb25NaWxsaXMiOjEwMDAsImVycm9yQ291bnQiOjB9fQ==" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjgiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjYyIsInRvb2xJZCI6InBlcmZvcm1hbmNlLmNsaWVudCIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDEiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiIxRGhlc0NMTTIxV0RZVGRCd0pfTEFuQVpZVGFYTXRWY2lvdV9Zb0dKY0RRIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2NzAsInNoYTI1NiI6IjJmZjBlZmUyMmJjMjExMmFlYzEzY2Q1YzQ5ZGZmNTE1MWNhNTQ1OTlmY2FlMWQzN2I2YTE5MDdlMzg0NzRlYTMifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoid0dKbTYxb3R3OURHTll6bkFJREZUSm5qakc4UHV6WnZkcF9GMXBQWmJrMFJVQkxKOE5oVG9KMkpzY1d3SGdvOV9CbjZJU1BVaWVSbFpkZFVDUTJ6ZXcifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "performance.siteReplication", + "classification": "L2", + "expectedDisposition": "REJECT_SITE_REPLICATION_TARGETS_UNAVAILABLE", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMzEiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJ6eTVtZExyd1dwcUxNS2pCUWhta2FTQ3Y0cWE1czNwNVQzcGtZSjd6VFUwIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwNDAsInNoYTI1NiI6IjRjYzkwODMxMmRlNDZlMjJlNmQ0NTI2YjIxNmFhZTEzOGRiYzE3OWYyNmQxNWYyZmUxMGU0NTg4MWJkYWE1MDkiLCJjbGFzc2lmaWNhdGlvbiI6IkwyIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiJmNjdkYmI4ZmQ4MjhmZjRjNmI2NmIyZWNhZDMyNTg0MDE0Y2ExYmEzZDdiOGVjOTI1YTQzMWIyNzg1ZWJlNzZiIiwiY2xhc3NpZmljYXRpb24iOiJMMiJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6Njg2LCJzaGEyNTYiOiI0MjQxZjRjY2UwZTI1ODU1ZjQ3ZTVhYzZiMWM4Y2UyNzI0ZGMyMTMzNmQzNTA2YTQ2MDkwMWQ1ZGVjY2ViNjdlIiwiY2xhc3NpZmljYXRpb24iOiJMMiJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiTU5WNXZGeFlSdXhMcWF4S3NneVpJbzRfVGFranh6T3NwUjlaMDJzc1J6WlNOXzZVbXA0emRCSzh4aEEzWW9ldU1DNlhoMm11cnZsMEVDX0Jzd1hDSWcifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjkiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjZCIsInRvb2xJZCI6InBlcmZvcm1hbmNlLnNpdGVSZXBsaWNhdGlvbiIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDIiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiJ6eTVtZExyd1dwcUxNS2pCUWhta2FTQ3Y0cWE1czNwNVQzcGtZSjd6VFUwIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2ODYsInNoYTI1NiI6IjQyNDFmNGNjZTBlMjU4NTVmNDdlNWFjNmIxYzhjZTI3MjRkYzIxMzM2ZDM1MDZhNDYwOTAxZDVkZWNjZWI2N2UifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiU2EwMUFGMkx4Si1PSnZmUldMZGtDbEdpNEtnXzVDQmM0dUp6RHpKdEttQS0tN1JuTVJiOVFYVGtMek5yZ3RwX1hOWC1uY0JjWU1iZEhidkhnblFPUXcifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjkiLCJ0b29sSWQiOiJwZXJmb3JtYW5jZS5zaXRlUmVwbGljYXRpb24iLCJjYXBhYmlsaXR5IjoicGVyZm9ybWFuY2Uuc2l0ZVJlcGxpY2F0aW9uQDEiLCJvdXRjb21lIjoiU1VDQ0VFREVEIiwicmVhc29uQ29kZSI6IkNPTVBMRVRFIiwiZHVyYXRpb25NaWxsaXMiOjEwMDAsInByb3ZlbmFuY2UiOnsicmVwb3NpdG9yeSI6InJ1c3Rmcy9ydXN0ZnMiLCJzb3VyY2VDb21taXQiOiJhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhIiwiZXhlY3V0YWJsZVNoYTI1NiI6ImJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmIiLCJydXN0ZnNWZXJzaW9uIjoiMS4wLjAtcmMuNiIsIm9zRmFtaWx5IjoiTElOVVgiLCJhcmNoaXRlY3R1cmUiOiJhYXJjaDY0IiwiYnVpbGRGZWF0dXJlcyI6W119LCJjb3ZlcmFnZSI6eyJyZXF1ZXN0ZWRVbml0cyI6MSwiY29tcGxldGVkVW5pdHMiOjEsInVuaXQiOiJXSU5ET1cifSwiZGF0YSI6eyJyZXBsaWNhdGVkQnl0ZXMiOjEwNDg1NzYsImNvbmZpcm1lZE9iamVjdHMiOjEsImR1cmF0aW9uTWlsbGlzIjoxMDAwLCJtYXhPYnNlcnZlZExhZ01pbGxpcyI6MjUwLCJlcnJvckNvdW50IjowfX0=" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNjkiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjZCIsInRvb2xJZCI6InBlcmZvcm1hbmNlLnNpdGVSZXBsaWNhdGlvbiIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDIiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiJ6eTVtZExyd1dwcUxNS2pCUWhta2FTQ3Y0cWE1czNwNVQzcGtZSjd6VFUwIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2ODYsInNoYTI1NiI6IjQyNDFmNGNjZTBlMjU4NTVmNDdlNWFjNmIxYzhjZTI3MjRkYzIxMzM2ZDM1MDZhNDYwOTAxZDVkZWNjZWI2N2UifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiX2xxQ2pPUzdkaFgwU2k4Uno3ZDhKVzVvbVozTWhocE9uak0tNXd3MWd6VnhERnNmdkxFWUQ3YmQ0WDlQQ3B0czVqeVhfUzhPWFRZUW9SZjBrN19sX2cifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "logs.capture", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMzIiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiIxdmhnS3pqRUFIRmhVRnoyTktzSjNrQm9IaW5LVHZ4WGo4Q2RaanJQU3NnIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjUsInNoYTI1NiI6ImFlZDM4YzdmOWRhNzQ1ZjM1NjJjN2VlMjQ1MGQ1ZDNmYjFhMmY2M2Q3Nzc5MzU3ZjVjODlkNjUxMWMxMWU2YTAiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiI2ZmMyZGNlNTlhYzRlZjU0Y2ZhNjY1YTBiYTUzZjVhOTBiMDI4ZDI3OTI2MjcyYjM5NjdjODE4ZWMzNjNiN2ZkIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjQ3LCJzaGEyNTYiOiJiODRkOWZlNjgwYmMyMDg0NTlmMTZmOTAxYzhiN2UyYzc2MGE3MDM1MDJmOWMzMjQ0ZWRiOTc0NjFlZWYzMDhmIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiOFJHallYWjJoYzRlZUllcU9IOVJ1alB2R2NRT3FpcVFPeHdMSlNSZVNQSV9yV3VOV3kwWVNtRXhxZmFrSElpejF0a001UXFPdEdSMklUcS1zZlpDaUEifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmEiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjZSIsInRvb2xJZCI6ImxvZ3MuY2FwdHVyZSIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDMiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiIxdmhnS3pqRUFIRmhVRnoyTktzSjNrQm9IaW5LVHZ4WGo4Q2RaanJQU3NnIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2NDcsInNoYTI1NiI6ImI4NGQ5ZmU2ODBiYzIwODQ1OWYxNmY5MDFjOGI3ZTJjNzYwYTcwMzUwMmY5YzMyNDRlZGI5NzQ2MWVlZjMwOGYifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiMmJHcDZoV3RYbllzbDV0Qkp6Nkt0UEpINWJQc0sxZnJ4YkpMVFRvOHRaVmNaMlJMT0F3aUlOQkxBbGMyZW1fUTZETzhXVWRrNHY4bmVkQjdOMndzc0EifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmEiLCJ0b29sSWQiOiJsb2dzLmNhcHR1cmUiLCJjYXBhYmlsaXR5IjoibG9ncy5jYXB0dXJlQDEiLCJvdXRjb21lIjoiU1VDQ0VFREVEIiwicmVhc29uQ29kZSI6IkNPTVBMRVRFIiwiZHVyYXRpb25NaWxsaXMiOjEwMDAsInByb3ZlbmFuY2UiOnsicmVwb3NpdG9yeSI6InJ1c3Rmcy9ydXN0ZnMiLCJzb3VyY2VDb21taXQiOiJhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhIiwiZXhlY3V0YWJsZVNoYTI1NiI6ImJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmIiLCJydXN0ZnNWZXJzaW9uIjoiMS4wLjAtcmMuNiIsIm9zRmFtaWx5IjoiTElOVVgiLCJhcmNoaXRlY3R1cmUiOiJhYXJjaDY0IiwiYnVpbGRGZWF0dXJlcyI6W119LCJjb3ZlcmFnZSI6eyJyZXF1ZXN0ZWRVbml0cyI6MSwiY29tcGxldGVkVW5pdHMiOjEsInVuaXQiOiJXSU5ET1cifSwiZGF0YSI6eyJldmVudHMiOlt7Im9mZnNldE1pbGxpcyI6MTAsInNldmVyaXR5IjoiRVJST1IiLCJldmVudElkIjoiRFJJVkVfVU5BVkFJTEFCTEUifV0sImRyb3BwZWRFdmVudENvdW50IjowfX0=" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmEiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjZSIsInRvb2xJZCI6ImxvZ3MuY2FwdHVyZSIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDMiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiIxdmhnS3pqRUFIRmhVRnoyTktzSjNrQm9IaW5LVHZ4WGo4Q2RaanJQU3NnIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2NDcsInNoYTI1NiI6ImI4NGQ5ZmU2ODBiYzIwODQ1OWYxNmY5MDFjOGI3ZTJjNzYwYTcwMzUwMmY5YzMyNDRlZGI5NzQ2MWVlZjMwOGYifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoid0NEQTdxcVBWRkNfZ3JtSVQ4RTBHRGpLcFBpSmhvXzdKVUs0Z3Ixb0FGbzZZcG9TNzZWbEx0X1FHcDlqd0phYjkwR2pDZW9qRzJsWEpoNEpQWlFXT3cifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "profile.cpu", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMzMiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJDWEVkdTlCRVh3YTgyZXFsbWZsbUpSQ1J0SFlRei1HR2dRV2dWa3hzQ2FjIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjQsInNoYTI1NiI6Ijk4NmU1YTdkYzkxZTJlZTRkN2U1YTI0NGE3NTljNzk4Yjg1YjA2M2YyNzI3N2UxYmUzNzU0NDk4MjMwYjRlYWYiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiIxZWY5ODE2OTU0OWY4M2MwNzFiZjY0NmNlODU0MWQwY2Y5YjFiN2NjMmUzOTkxMzU0NjNlNzlhNjk1MGM2NGNkIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjQ1LCJzaGEyNTYiOiI1NjkxMjYxODgwMjFhZGQ3MmNlM2JmY2M1Y2E4MDUyZTdlZGQ1YjZhYWRjMTUzZTU1OTJmNmRhOTU5NjViZDhlIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiZkktT1lGUUVoNnB3bFZMRVoxdHhEcWY4WG9qaC1RUGxINlk0Y1Nfd1JHUjdtOVJrTXViTjd0TEF1dEtST0wzLTNtdmozZG84NGZ1LU5UWnl1Smc3S3cifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmIiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjZiIsInRvb2xJZCI6InByb2ZpbGUuY3B1Iiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMyIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6IkNYRWR1OUJFWHdhODJlcWxtZmxtSlJDUnRIWVF6LUdHZ1FXZ1ZreHNDYWMiLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjY0NSwic2hhMjU2IjoiNTY5MTI2MTg4MDIxYWRkNzJjZTNiZmNjNWNhODA1MmU3ZWRkNWI2YWFkYzE1M2U1NTkyZjZkYTk1OTY1YmQ4ZSJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiR2EzcVR1T0tQV3I0UUxMWlFPeVAwQ3FCbFBaU1BSSEFraTFJaTBmajdjY2pwNjQyLWsxeDUxQzRjT00zeWdldmx6NGlXY0RJUEZGdWFiQWJ2OTdpNmcifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmIiLCJ0b29sSWQiOiJwcm9maWxlLmNwdSIsImNhcGFiaWxpdHkiOiJwcm9maWxlLmNwdUAxIiwib3V0Y29tZSI6IlNVQ0NFRURFRCIsInJlYXNvbkNvZGUiOiJDT01QTEVURSIsImR1cmF0aW9uTWlsbGlzIjoxMDAwLCJwcm92ZW5hbmNlIjp7InJlcG9zaXRvcnkiOiJydXN0ZnMvcnVzdGZzIiwic291cmNlQ29tbWl0IjoiYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYSIsImV4ZWN1dGFibGVTaGEyNTYiOiJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiIiwicnVzdGZzVmVyc2lvbiI6IjEuMC4wLXJjLjYiLCJvc0ZhbWlseSI6IkxJTlVYIiwiYXJjaGl0ZWN0dXJlIjoiYWFyY2g2NCIsImJ1aWxkRmVhdHVyZXMiOltdfSwiY292ZXJhZ2UiOnsicmVxdWVzdGVkVW5pdHMiOjEsImNvbXBsZXRlZFVuaXRzIjoxLCJ1bml0IjoiV0lORE9XIn0sImRhdGEiOnsic2FtcGxlUGVyaW9kTWljcm9zIjoxMDAwMCwic2FtcGxlcyI6W3sic3ltYm9sSWQiOiJzeW1ib2wtMSIsInNhbXBsZUNvdW50IjozfV0sImRyb3BwZWRTYW1wbGVDb3VudCI6MH19" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmIiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBjZiIsInRvb2xJZCI6InByb2ZpbGUuY3B1Iiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMyIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6IkNYRWR1OUJFWHdhODJlcWxtZmxtSlJDUnRIWVF6LUdHZ1FXZ1ZreHNDYWMiLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjY0NSwic2hhMjU2IjoiNTY5MTI2MTg4MDIxYWRkNzJjZTNiZmNjNWNhODA1MmU3ZWRkNWI2YWFkYzE1M2U1NTkyZjZkYTk1OTY1YmQ4ZSJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoidVJ6N28xcGp1VDBrdTdpeExyYmQ4M1QzQ3RIc29MRTVSMWRNbE1fMUJ4d1hYRHlBQ3pOQzd0LWlUd3ZNTkRfUjJwUmdOY0ZDMXA3eTNCLTZLeDJWM2cifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "profile.memory", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMzQiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiI2Zm1rS1lzTWlfRzU2UDZoRkRXdERFbTctc1BvcnJwVlhXWkljOWFpdlNrIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjcsInNoYTI1NiI6IjY1MzNkMWVjNzZlN2ZiNjBlM2Q1NDkyY2NlNzg3NGIzNzE1Mjk5ZjUzODQwNmM4YjJiZmQxZjZmMTY2YThmZTMiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiI4M2QwYzI5YTM3MmIxYTg0ZDQ2OWU0NTQxMTdkYTk0YjUyNGMzZTk0ZTc2ZjhhYTgxZDRlMGU5ZjJmZGU5YWE1IiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjUwLCJzaGEyNTYiOiI2OTg3YjQwM2VjNzg5MDQyMjcyOGRmY2VmMDZjNzU3YWMyMDAyM2RjYjY5NWE1OGQxMDA5ZmM4YmRlM2FlZDNmIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoieVR1RVJfMlMyMWN1R3hPSEFQZThTYVMwYWdkcHFkNm51R0lKOFp4a0RVd2lCbWlaektoZ1FZS2cxOVRwYkM5S0lxQ2FlUmswLVotSVEyZFhMdWt0cXcifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmMiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkMCIsInRvb2xJZCI6InByb2ZpbGUubWVtb3J5Iiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMyIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6IjZmbWtLWXNNaV9HNTZQNmhGRFd0REVtNy1zUG9ycnBWWFdaSWM5YWl2U2siLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjY1MCwic2hhMjU2IjoiNjk4N2I0MDNlYzc4OTA0MjI3MjhkZmNlZjA2Yzc1N2FjMjAwMjNkY2I2OTVhNThkMTAwOWZjOGJkZTNhZWQzZiJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiOTNLVkRYWFB1YnlxcVlXOVVVMGVyc0tEc29QX1dhOGx1d0U5OU1DT2stVWJpRno4cXJLb0dEMEp0MmhSd2JHcGJPaDBzcEplVGlubDNnR1JnLVZVencifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmMiLCJ0b29sSWQiOiJwcm9maWxlLm1lbW9yeSIsImNhcGFiaWxpdHkiOiJwcm9maWxlLm1lbW9yeUAxIiwib3V0Y29tZSI6IlNVQ0NFRURFRCIsInJlYXNvbkNvZGUiOiJDT01QTEVURSIsImR1cmF0aW9uTWlsbGlzIjoxMDAwLCJwcm92ZW5hbmNlIjp7InJlcG9zaXRvcnkiOiJydXN0ZnMvcnVzdGZzIiwic291cmNlQ29tbWl0IjoiYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYSIsImV4ZWN1dGFibGVTaGEyNTYiOiJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiIiwicnVzdGZzVmVyc2lvbiI6IjEuMC4wLXJjLjYiLCJvc0ZhbWlseSI6IkxJTlVYIiwiYXJjaGl0ZWN0dXJlIjoiYWFyY2g2NCIsImJ1aWxkRmVhdHVyZXMiOltdfSwiY292ZXJhZ2UiOnsicmVxdWVzdGVkVW5pdHMiOjEsImNvbXBsZXRlZFVuaXRzIjoxLCJ1bml0IjoiV0lORE9XIn0sImRhdGEiOnsic2NvcGUiOiJBTExPQ0FUSU9OX0FHR1JFR0FURVMiLCJhbGxvY2F0ZWRCeXRlcyI6NDA5NiwiYWxsb2NhdGlvbkNvdW50IjoyLCJzYW1wbGVQZXJpb2RNaWNyb3MiOjEwMDAwfX0=" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmMiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkMCIsInRvb2xJZCI6InByb2ZpbGUubWVtb3J5Iiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMyIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6IjZmbWtLWXNNaV9HNTZQNmhGRFd0REVtNy1zUG9ycnBWWFdaSWM5YWl2U2siLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjY1MCwic2hhMjU2IjoiNjk4N2I0MDNlYzc4OTA0MjI3MjhkZmNlZjA2Yzc1N2FjMjAwMjNkY2I2OTVhNThkMTAwOWZjOGJkZTNhZWQzZiJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiWElKU3h4UFRFbWhXLWdYR3NaWGlySXFUMi0tblFfU3gyNHVzQl83dmxrUTBZUWpqb3BjakIzeFgwRTZNWkkxenRsaXhNUWtkT2tBMnV1Q0FIZ01RTUEifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "profile.threads", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMzUiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJxMTIwQy1NN0xJZmNmUUpZWURKNUFQWktjZGhCNlVPSHUyMzR0U3NiXzA4IiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjgsInNoYTI1NiI6ImUzODIyZTAyY2Y0ZDBhZTA1ZjJhOTQ4MTMzYTRmZTg4ZDM5MjNjNzA0ODVhYjk1MTdmMDI3NmViMTRjM2M3NzciLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiI4OTc5Zjk1ZTU3MzM1ZTA5NzgwNWI2NGVlOTM1MDgzMTYyOWVlMDg3MTY5MTYxY2I1Mzk1NGNkYzljZjJhOWM4IiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjIyLCJzaGEyNTYiOiJhYTExZTI5MjgzOTRjZjA1MTBiNjAyZDY0MzA4NGMxYWYwMGM5NmNiM2Q3Y2E0YzMwY2MyZDhjNDM1ZTUyYzc5IiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiZ0RXcVJoTE5SR1hpUjNlZXlIdTdHQXRfTjFKa0NFNXVSUmNnUVFldENxSUJKZU4xdTc2ZnNRUng2d0JOd3RoMnFSb0w3SVNjUWR5ZHVlT3NUM1Z2aVEifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmQiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkMSIsInRvb2xJZCI6InByb2ZpbGUudGhyZWFkcyIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDMiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiJxMTIwQy1NN0xJZmNmUUpZWURKNUFQWktjZGhCNlVPSHUyMzR0U3NiXzA4IiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2MjIsInNoYTI1NiI6ImFhMTFlMjkyODM5NGNmMDUxMGI2MDJkNjQzMDg0YzFhZjAwYzk2Y2IzZDdjYTRjMzBjYzJkOGM0MzVlNTJjNzkifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiQ3RMalBZSk40VWRSSlhDT1pVSFNIZkd3TF9IeHBVYWlDLThyOGt3cHUtcDZ6U2VvR2wwRTlRMHFXVXBGYlAwdFM4Nnk0aUdPQkd3TUs5Y0RGQV9oNmcifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmQiLCJ0b29sSWQiOiJwcm9maWxlLnRocmVhZHMiLCJjYXBhYmlsaXR5IjoicHJvZmlsZS50aHJlYWRzQDEiLCJvdXRjb21lIjoiU1VDQ0VFREVEIiwicmVhc29uQ29kZSI6IkNPTVBMRVRFIiwiZHVyYXRpb25NaWxsaXMiOjEwMDAsInByb3ZlbmFuY2UiOnsicmVwb3NpdG9yeSI6InJ1c3Rmcy9ydXN0ZnMiLCJzb3VyY2VDb21taXQiOiJhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhIiwiZXhlY3V0YWJsZVNoYTI1NiI6ImJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmIiLCJydXN0ZnNWZXJzaW9uIjoiMS4wLjAtcmMuNiIsIm9zRmFtaWx5IjoiTElOVVgiLCJhcmNoaXRlY3R1cmUiOiJhYXJjaDY0IiwiYnVpbGRGZWF0dXJlcyI6W119LCJjb3ZlcmFnZSI6eyJyZXF1ZXN0ZWRVbml0cyI6MSwiY29tcGxldGVkVW5pdHMiOjEsInVuaXQiOiJXSU5ET1cifSwiZGF0YSI6eyJzY29wZSI6IlRPS0lPX1JVTlRJTUUiLCJzdGF0ZXMiOlt7InN0YXRlIjoiV0FJVElORyIsInRocmVhZENvdW50IjoyfV19fQ==" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmQiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkMSIsInRvb2xJZCI6InByb2ZpbGUudGhyZWFkcyIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDMiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiJxMTIwQy1NN0xJZmNmUUpZWURKNUFQWktjZGhCNlVPSHUyMzR0U3NiXzA4IiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo2MjIsInNoYTI1NiI6ImFhMTFlMjkyODM5NGNmMDUxMGI2MDJkNjQzMDg0YzFhZjAwYzk2Y2IzZDdjYTRjMzBjYzJkOGM0MzVlNTJjNzkifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoieFF3MXRfYUljMnlWeFgyRVhMQmdkaGZjaWRsMkh4S2lGX2YyU1kzWVBTc3RwUDVVVUM3TktMck95SGxKWlFZQWdRRkh2YXZ6T1BVN1JnSkZXUlp2OWcifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "inspect.object", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMzYiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiI1UnV1eUxzbjBvdkJISHZ0cUU4S0dKWUxuX2o0Nlc4U1NIZEZoMWRpZ0s4IiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjcsInNoYTI1NiI6ImRiYjQ5YjhjZWNkMDM1MTFjOTY5MjE1OTdjYmRiODBlNWQ0ZWQ1ZTQzMmMzNDAyZDdjNDQ1ZjNjMzZlNjIwMDAiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiJlZDIwZmNlMGNlZjhjZGIzNDZiNjA3YjZkNjFkNGUxMzQzNmUwMTQyNjAxMjEyZTcwODM0YTkwOGIyZWI1NTdiIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NzY0LCJzaGEyNTYiOiI0OTE2NTk4NjY3NjAzMTNhN2VlZWM0NzZiYjFiYjUzNTY3ZjA3NGU4NGUzNGQ3Y2NkOWU1MDQxOTk2OTJhMGFiIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiRklGd05yT2FSN0ZUbENRQjhyZlhmVzVSZmFEMjBTTVp5cTJMYkdtVHg1OUJFSzhMWi1SNnMzMk40ZGgwLVl5bXBPckNiZkxJY2Z6RFVISXRyVE85RkEifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmUiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkMiIsInRvb2xJZCI6Imluc3BlY3Qub2JqZWN0Iiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMyIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6IjVSdXV5THNuMG92QkhIdnRxRThLR0pZTG5fajQ2VzhTU0hkRmgxZGlnSzgiLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjc2NCwic2hhMjU2IjoiNDkxNjU5ODY2NzYwMzEzYTdlZWVjNDc2YmIxYmI1MzU2N2YwNzRlODRlMzRkN2NjZDllNTA0MTk5NjkyYTBhYiJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoia2Y0cV95X1pFbW5SQjNvNkNYeHdBMG1UVGhkRVRZZDJQNHkwV3pGMllRNTZvTl80WERvczRqMU1JQ1ozOE55MTBxb01lZUxCYl82Q0lOV3haYXRfZFEifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmUiLCJ0b29sSWQiOiJpbnNwZWN0Lm9iamVjdCIsImNhcGFiaWxpdHkiOiJpbnNwZWN0Lm9iamVjdEAxIiwib3V0Y29tZSI6IlNVQ0NFRURFRCIsInJlYXNvbkNvZGUiOiJDT01QTEVURSIsImR1cmF0aW9uTWlsbGlzIjoxMCwicHJvdmVuYW5jZSI6eyJyZXBvc2l0b3J5IjoicnVzdGZzL3J1c3RmcyIsInNvdXJjZUNvbW1pdCI6ImFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWEiLCJleGVjdXRhYmxlU2hhMjU2IjoiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYiIsInJ1c3Rmc1ZlcnNpb24iOiIxLjAuMCIsIm9zRmFtaWx5IjoiTElOVVgiLCJhcmNoaXRlY3R1cmUiOiJ4ODZfNjQiLCJidWlsZEZlYXR1cmVzIjpbImluc3BlY3QiXX0sImNvdmVyYWdlIjp7InJlcXVlc3RlZFVuaXRzIjoxLCJjb21wbGV0ZWRVbml0cyI6MSwidW5pdCI6IkNIRUNLIn0sImRhdGEiOnsic2NvcGUiOiJMT0NBTF9PQkpFQ1RfU1VNTUFSWSIsImZpbmRpbmdzIjpbeyJyZXNvdXJjZUFsaWFzIjoib2JqZWN0LTEiLCJydWxlSWQiOiJTSEFSRF9BVkFJTEFCSUxJVFkiLCJvdXRjb21lIjoiUEFTUyIsInJlYXNvbiI6IkFWQUlMQUJMRSIsIm1pc3NpbmdTaGFyZENvdW50IjowLCJjb3JydXB0U2hhcmRDb3VudCI6MCwicmVjb25zdHJ1Y3Rpb24iOiJOT1RfTkVFREVEIn1dfX0=" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmUiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkMiIsInRvb2xJZCI6Imluc3BlY3Qub2JqZWN0Iiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMyIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6IjVSdXV5THNuMG92QkhIdnRxRThLR0pZTG5fajQ2VzhTU0hkRmgxZGlnSzgiLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjc2NCwic2hhMjU2IjoiNDkxNjU5ODY2NzYwMzEzYTdlZWVjNDc2YmIxYmI1MzU2N2YwNzRlODRlMzRkN2NjZDllNTA0MTk5NjkyYTBhYiJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiRUxlUUlOcHR4d3M5X0hkN0ktMVZqVVlQRHA2ODZnSE5XNThndmpJUWNxUVowanQ5SVJ1aW5wdVA0cVNzZUtncmh2Q2FOcHlLTzdiZUdwSDFOR21aYVEifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "telemetry.record", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMzciLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJzT3ZtdkhKb2NwdElRaGY5cVlzTHA1bmI0dXB6UUVmQW1pLXBXNzdTS2lZIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjksInNoYTI1NiI6Ijc1MTNmODM5YTAyZGZhOGNiYWIwNGVkZTlhNTk1Yzk5YjU1ODBiMjQ5Njk1YWRhNWI4MGYzNDUxYzM0ODU1NTMiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiI0MjI0ODZjZTQyMzdmNzYyMzUyMTc5NTg0ZmIzMDg5Y2I0ZTVjZGM0Njg5NDA3ZjA4MTZlNDUzNGVmZWRlNDQ2IiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjQ2LCJzaGEyNTYiOiJjNjdjMzdlNGU4M2ZmZTBhZjU4YTA5YjVkODJkNDFmZTFiMDA2MmM1YzI0ZjdiNjRmMGQ2NTk0NWZlYzA1OTBmIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiY2RBbEZfN2VQVENndks5WVM5VW5fVzlHdHhOR1Bwb1VLSy04Q3V5X3ZpNERiNXNEWFMxVGFQWG5iZ2hwMkxWMHMxYzFxWG1BQWJnc04xeTlyakNQdVEifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmYiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkMyIsInRvb2xJZCI6InRlbGVtZXRyeS5yZWNvcmQiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwzIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoic092bXZISm9jcHRJUWhmOXFZc0xwNW5iNHVwelFFZkFtaS1wVzc3U0tpWSIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NjQ2LCJzaGEyNTYiOiJjNjdjMzdlNGU4M2ZmZTBhZjU4YTA5YjVkODJkNDFmZTFiMDA2MmM1YzI0ZjdiNjRmMGQ2NTk0NWZlYzA1OTBmIn19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiaUJwc2M1eGRVYjFqU3pEWVZESjNXR21kYWl2NExRRmJkVTJpYTZwRi1aQk5VdVJiQzdCVUROcnU0QzA3NE9BSHRTQU9jTHBKMi1mUXhBWHYwSVFJMXcifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmYiLCJ0b29sSWQiOiJ0ZWxlbWV0cnkucmVjb3JkIiwiY2FwYWJpbGl0eSI6InRlbGVtZXRyeS5yZWNvcmRAMSIsIm91dGNvbWUiOiJTVUNDRUVERUQiLCJyZWFzb25Db2RlIjoiQ09NUExFVEUiLCJkdXJhdGlvbk1pbGxpcyI6MTAwMCwicHJvdmVuYW5jZSI6eyJyZXBvc2l0b3J5IjoicnVzdGZzL3J1c3RmcyIsInNvdXJjZUNvbW1pdCI6ImFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWEiLCJleGVjdXRhYmxlU2hhMjU2IjoiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYiIsInJ1c3Rmc1ZlcnNpb24iOiIxLjAuMC1yYy42Iiwib3NGYW1pbHkiOiJMSU5VWCIsImFyY2hpdGVjdHVyZSI6ImFhcmNoNjQiLCJidWlsZEZlYXR1cmVzIjpbXX0sImNvdmVyYWdlIjp7InJlcXVlc3RlZFVuaXRzIjoxLCJjb21wbGV0ZWRVbml0cyI6MSwidW5pdCI6IldJTkRPVyJ9LCJkYXRhIjp7InNwYW5zIjpbeyJvcGVyYXRpb24iOiJHRVRfT0JKRUNUIiwiZHVyYXRpb25NaWNyb3MiOjUwMCwic3RhdHVzIjoiT0sifV0sImRyb3BwZWRTcGFuQ291bnQiOjB9fQ==" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNmYiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkMyIsInRvb2xJZCI6InRlbGVtZXRyeS5yZWNvcmQiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwzIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoic092bXZISm9jcHRJUWhmOXFZc0xwNW5iNHVwelFFZkFtaS1wVzc3U0tpWSIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NjQ2LCJzaGEyNTYiOiJjNjdjMzdlNGU4M2ZmZTBhZjU4YTA5YjVkODJkNDFmZTFiMDA2MmM1YzI0ZjdiNjRmMGQ2NTk0NWZlYzA1OTBmIn19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiVXIzR3ZqOHh3T0Roc3I3S1BYUUQ1aHdqa2Y0Z0kxRXhUaWh0cy03RGhaNWswWFpxOFZRQ3BDSGxLUzVpck14UVJNUFdTTTdRdUxCd3pqaGZXcndaRlEifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "telemetry.otlp", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMzgiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJJRVEtZnFRUXY2cWo0TTBHVW5rbWxNREgycHFLMXV0ellBaEw5ZlhLTjlNIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjcsInNoYTI1NiI6IjBkZGU3MWRhYjc0ZmE0NmMyZTg1NTM2NjE1ODRhNjU1ZmU5YjcwOWM3N2U5MTViNGViZThkYjc4OGUxMjdhMzciLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiJiMDU1ZjZlODIzZTgwMTQzZmJlODBkZDFhYTcwYzYzNmExZTU3NjNmMjYyMDE0NjMwMWM4YTA0ZjQ1OTQ3ZDYyIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjEzLCJzaGEyNTYiOiJmMTQ2YmZhMzJhYTI2MjdlZWExZDhkNzI1NTYzNTVlOWQ2MTUwZjYxODNhMmM2MDFjNmJlMjY1Nzg2NjM3YzNjIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiRnIxYXlMd0w4S0JRMHFPNUJWeThHRktkWWRJcXpDUjJfcmFTT1J3ZjMzRS1pd2dHcFAtNGF4bTVlbDR3eVJfaFBnQ055TXdUVUtiRUZGWnhPMk9TNlEifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzAiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkNCIsInRvb2xJZCI6InRlbGVtZXRyeS5vdGxwIiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMyIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6IklFUS1mcVFRdjZxajRNMEdVbmttbE1ESDJwcUsxdXR6WUFoTDlmWEtOOU0iLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjYxMywic2hhMjU2IjoiZjE0NmJmYTMyYWEyNjI3ZWVhMWQ4ZDcyNTU2MzU1ZTlkNjE1MGY2MTgzYTJjNjAxYzZiZTI2NTc4NjYzN2MzYyJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiVF9DU1J4Q2pmelkxNWQzR3loVXZxSVY4UmNudE5WaXExNFhwQWtiYUJTeHpERmRFdTMwQndzb0V6WUlFdGYtaEJTUGRGQVJfSmlzemJyODFXUzEyQmcifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzAiLCJ0b29sSWQiOiJ0ZWxlbWV0cnkub3RscCIsImNhcGFiaWxpdHkiOiJ0ZWxlbWV0cnkub3RscEAxIiwib3V0Y29tZSI6IlNVQ0NFRURFRCIsInJlYXNvbkNvZGUiOiJDT01QTEVURSIsImR1cmF0aW9uTWlsbGlzIjoxMDAwLCJwcm92ZW5hbmNlIjp7InJlcG9zaXRvcnkiOiJydXN0ZnMvcnVzdGZzIiwic291cmNlQ29tbWl0IjoiYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYSIsImV4ZWN1dGFibGVTaGEyNTYiOiJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiIiwicnVzdGZzVmVyc2lvbiI6IjEuMC4wLXJjLjYiLCJvc0ZhbWlseSI6IkxJTlVYIiwiYXJjaGl0ZWN0dXJlIjoiYWFyY2g2NCIsImJ1aWxkRmVhdHVyZXMiOltdfSwiY292ZXJhZ2UiOnsicmVxdWVzdGVkVW5pdHMiOjEsImNvbXBsZXRlZFVuaXRzIjoxLCJ1bml0IjoiV0lORE9XIn0sImRhdGEiOnsiYWNjZXB0ZWRTcGFuQ291bnQiOjEsInJlamVjdGVkU3BhbkNvdW50IjowLCJleHBvcnRlZEJ5dGVzIjoxMjh9fQ==" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzAiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkNCIsInRvb2xJZCI6InRlbGVtZXRyeS5vdGxwIiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMyIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6IklFUS1mcVFRdjZxajRNMEdVbmttbE1ESDJwcUsxdXR6WUFoTDlmWEtOOU0iLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjYxMywic2hhMjU2IjoiZjE0NmJmYTMyYWEyNjI3ZWVhMWQ4ZDcyNTU2MzU1ZTlkNjE1MGY2MTgzYTJjNjAxYzZiZTI2NTc4NjYzN2MzYyJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoibms4eFlfVG0xdjlUbTJvdDB3NTJlYzRlZl9oZHh6TnQzVGZTQTNyUEJfa29OeVQySzdidnBWLUhkUWNIVUl0d3p4R21PczcyMVJTb1NkVDFiTFZPb0EifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "telemetry.replay", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxMzkiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJWN1pyNkp3SGdrY2s1RXllNXVWNV9mVFdJd1R4Rkp2SjVPWGQtdjBidXJNIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjksInNoYTI1NiI6IjU3ZTY3Y2MxZTA0NjZhYTQwOWNkZTVmMWQ1MWNkYmE1ZGI5MTJmZWIwMTRlMGQzNGY5MzAyODczMWU1NWNjNTEiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiJmZTVmMzUxNGEyNzUxNmRlM2JlNGZlMzEwOWRiYWU4NTFjMWNiNzZmM2M5NjI2ZDM4YmUwMjYyZDgyZmE1ZGExIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NzE0LCJzaGEyNTYiOiI1YjlmZjU4YTFkNjNkYzhhM2I5OTg0OTZhYjUxMmJiNGFlOGMxYzY4OWU3ZTVkYWUxZDhhODgwMThkOWU3MTRlIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiZHNDdUZSa2hfQ0UyTVF2alBTellFVXdyQVdQZ2RUQ0xzVjVzdlF1N2U4WkdjbmNudEtMc2hRZ2ZVU3VNTVpaem5ZdHZwM2hpQU93NHJjZ19QclN4c0EifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzEiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkNSIsInRvb2xJZCI6InRlbGVtZXRyeS5yZXBsYXkiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwzIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoiVjdacjZKd0hna2NrNUV5ZTV1VjVfZlRXSXdUeEZKdko1T1hkLXYwYnVyTSIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NzE0LCJzaGEyNTYiOiI1YjlmZjU4YTFkNjNkYzhhM2I5OTg0OTZhYjUxMmJiNGFlOGMxYzY4OWU3ZTVkYWUxZDhhODgwMThkOWU3MTRlIn19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiWlREM1I0R3c1MFVDRWVON04xLVprRlh2Mm02YWdob2xCWUJuX2lJdkJHWnRSaXhlUTdxQVBfbHZUSFlrNWt3OUpudEcxZ2FJSWxVa1MyMkJ5T19DdkEifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzEiLCJ0b29sSWQiOiJ0ZWxlbWV0cnkucmVwbGF5IiwiY2FwYWJpbGl0eSI6InRlbGVtZXRyeS5yZXBsYXlAMSIsIm91dGNvbWUiOiJTVUNDRUVERUQiLCJyZWFzb25Db2RlIjoiQ09NUExFVEUiLCJkdXJhdGlvbk1pbGxpcyI6MTAwMCwicHJvdmVuYW5jZSI6eyJyZXBvc2l0b3J5IjoicnVzdGZzL3J1c3RmcyIsInNvdXJjZUNvbW1pdCI6ImFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWEiLCJleGVjdXRhYmxlU2hhMjU2IjoiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYiIsInJ1c3Rmc1ZlcnNpb24iOiIxLjAuMC1yYy42Iiwib3NGYW1pbHkiOiJMSU5VWCIsImFyY2hpdGVjdHVyZSI6ImFhcmNoNjQiLCJidWlsZEZlYXR1cmVzIjpbXX0sImNvdmVyYWdlIjp7InJlcXVlc3RlZFVuaXRzIjoxLCJjb21wbGV0ZWRVbml0cyI6MSwidW5pdCI6IldJTkRPVyJ9LCJkYXRhIjp7ImlucHV0QXJ0aWZhY3RTaGEyNTYiOiJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiIiwic3BhbnMiOlt7Im9wZXJhdGlvbiI6IkdFVF9PQkpFQ1QiLCJkdXJhdGlvbk1pY3JvcyI6NTAwLCJzdGF0dXMiOiJPSyJ9XX19" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzEiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkNSIsInRvb2xJZCI6InRlbGVtZXRyeS5yZXBsYXkiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwzIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoiVjdacjZKd0hna2NrNUV5ZTV1VjVfZlRXSXdUeEZKdko1T1hkLXYwYnVyTSIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NzE0LCJzaGEyNTYiOiI1YjlmZjU4YTFkNjNkYzhhM2I5OTg0OTZhYjUxMmJiNGFlOGMxYzY4OWU3ZTVkYWUxZDhhODgwMThkOWU3MTRlIn19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiV1VNRW9lWERwZkVBdk5vOTFZcFdkU3Jkb3RYeHRBQk4xSy1QODZZUkF1UmUwbEw4N1hqckNNVWJpNWtha3dENDFtR0xkSVZFank3MVdLR1ZmVl81bUEifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "top.api", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxM2EiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJybVdkVFZOQ1pyeGptRzFPZEZTcG85ZV9hMm5kSHFkeXV0b2RHQXJlU1hJIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjAsInNoYTI1NiI6IjRhMGZhYTlkZDFkNmI2M2Y2ZDBjMmQyYjY4YjEzMTcwNjFkZGM4MTk1NTc1YmI3NjAyNjc2ZjU4NDgyN2VjNjQiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiI5MzQyMmQyYjc4YWRlMmIyZDU3OTE2NGRmMzFiYTcyNTkzOGI1ZWJjODEwMjY1OWExZWM4Njc5ODI0YTU2OTdhIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjM4LCJzaGEyNTYiOiJhNWFkOWQ4OTc4ZDllZjc4OTA3N2I2NDMzOWYzMmZhODA1ZjZiYTAzZWU0NWY4NGVlZjczN2Y5YmM5OTg5MDg3IiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiWFoxT2lrdzg4LU8yOEJwREVWQmtJdlNqNkQ4bDdhbFlPWEZOV2NuX0tNbEJMcVdvRzhGcVkyM0pYWWV4SUZ5VG9OLVpja0ZQeTRsR1pMbVMtSjFXUUEifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzIiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkNiIsInRvb2xJZCI6InRvcC5hcGkiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwzIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoicm1XZFRWTkNacnhqbUcxT2RGU3BvOWVfYTJuZEhxZHl1dG9kR0FyZVNYSSIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NjM4LCJzaGEyNTYiOiJhNWFkOWQ4OTc4ZDllZjc4OTA3N2I2NDMzOWYzMmZhODA1ZjZiYTAzZWU0NWY4NGVlZjczN2Y5YmM5OTg5MDg3In19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiMUcxYlZkT01wd0trWFNUd29aVDlDX3FmYkdQVDgwVnIyYW9OTDBOVEhzTm11aFJxSDJWZ3ZKUmRMeG04MmlvQlZCZ1hTM2kwb0h6QUpQcENBT29qbGcifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzIiLCJ0b29sSWQiOiJ0b3AuYXBpIiwiY2FwYWJpbGl0eSI6InRvcC5hcGlAMSIsIm91dGNvbWUiOiJTVUNDRUVERUQiLCJyZWFzb25Db2RlIjoiQ09NUExFVEUiLCJkdXJhdGlvbk1pbGxpcyI6MTAwMCwicHJvdmVuYW5jZSI6eyJyZXBvc2l0b3J5IjoicnVzdGZzL3J1c3RmcyIsInNvdXJjZUNvbW1pdCI6ImFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWEiLCJleGVjdXRhYmxlU2hhMjU2IjoiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYiIsInJ1c3Rmc1ZlcnNpb24iOiIxLjAuMC1yYy42Iiwib3NGYW1pbHkiOiJMSU5VWCIsImFyY2hpdGVjdHVyZSI6ImFhcmNoNjQiLCJidWlsZEZlYXR1cmVzIjpbXX0sImNvdmVyYWdlIjp7InJlcXVlc3RlZFVuaXRzIjoxLCJjb21wbGV0ZWRVbml0cyI6MSwidW5pdCI6IldJTkRPVyJ9LCJkYXRhIjp7Im9wZXJhdGlvbiI6IkdFVF9PQkpFQ1QiLCJyZXF1ZXN0Q291bnQiOjIsImVycm9yQ291bnQiOjAsIndpbmRvd01pbGxpcyI6MTAwMCwidG90YWxEdXJhdGlvbk1pY3JvcyI6NzAwfX0=" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzIiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkNiIsInRvb2xJZCI6InRvcC5hcGkiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwzIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoicm1XZFRWTkNacnhqbUcxT2RGU3BvOWVfYTJuZEhxZHl1dG9kR0FyZVNYSSIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NjM4LCJzaGEyNTYiOiJhNWFkOWQ4OTc4ZDllZjc4OTA3N2I2NDMzOWYzMmZhODA1ZjZiYTAzZWU0NWY4NGVlZjczN2Y5YmM5OTg5MDg3In19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoibTZ0YUlxSVZaMDk0MW4yNFdzalZacUdRTEpFSlo1anlyLXJDMVhuVmJieGE4WUY3THpfOWZRRjFVNmNRU0lERHhxaEhpcVgzRlE4THg5ZE8xdFY2SXcifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "top.disk", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxM2IiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJNamQxdzhpVWp0SnlKdUtZTEdjRHF2eU1icmlFU1I5UDQwMXE1cEMxU2c0IiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjEsInNoYTI1NiI6IjI4NzA1ODY0MTY0OGM2NzRkNzJmYjJhZmI3MGMxZDk3ZjQ1MTAwMDg4ZWQzNTU2ODAzYjc2ZmRhNGFjMTRhOTIiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiJlMGEwYThiOGQ3MzExMmI5ZDkwZWY2OWE1YjZiZGE3YzQ3MWEwMDBmMzZhZTk0ZjI5OTUyMTdhYTk4NTFiZjYwIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjMwLCJzaGEyNTYiOiI2NmVlZTY3MTQ5ODkwZTE1NGQ5NTliNzAxYWVhYjlhMjNhZDE3MzA2OTFlOTczZGU1ZGM1MmM4ZDU1NzQ4ZWMwIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiTVd4cm9GWmJUVWJlU2xoemZxUXFMUkpFZkU4N3c1eHpyVFdYeDQzejJLdER1T21mdXR1dmdHeGFyaGh2YV9JVi02VHg5OGpBZV9KWXNjSWJpYi1QcEEifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzMiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkNyIsInRvb2xJZCI6InRvcC5kaXNrIiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMyIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6Ik1qZDF3OGlVanRKeUp1S1lMR2NEcXZ5TWJyaUVTUjlQNDAxcTVwQzFTZzQiLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjYzMCwic2hhMjU2IjoiNjZlZWU2NzE0OTg5MGUxNTRkOTU5YjcwMWFlYWI5YTIzYWQxNzMwNjkxZTk3M2RlNWRjNTJjOGQ1NTc0OGVjMCJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiNGxkUDYtX0lkQTJiQVJxeThRRllZMWFSb1gwc3NYTEFNMzRXR1ZQQjl5OW15eHZGS1JLalRjbFdPMkhoOEI4NFozd2lFaHN2TmhHc1Bzcm9RZTBFaFEifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzMiLCJ0b29sSWQiOiJ0b3AuZGlzayIsImNhcGFiaWxpdHkiOiJ0b3AuZGlza0AxIiwib3V0Y29tZSI6IlNVQ0NFRURFRCIsInJlYXNvbkNvZGUiOiJDT01QTEVURSIsImR1cmF0aW9uTWlsbGlzIjoxMDAwLCJwcm92ZW5hbmNlIjp7InJlcG9zaXRvcnkiOiJydXN0ZnMvcnVzdGZzIiwic291cmNlQ29tbWl0IjoiYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYSIsImV4ZWN1dGFibGVTaGEyNTYiOiJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiIiwicnVzdGZzVmVyc2lvbiI6IjEuMC4wLXJjLjYiLCJvc0ZhbWlseSI6IkxJTlVYIiwiYXJjaGl0ZWN0dXJlIjoiYWFyY2g2NCIsImJ1aWxkRmVhdHVyZXMiOltdfSwiY292ZXJhZ2UiOnsicmVxdWVzdGVkVW5pdHMiOjEsImNvbXBsZXRlZFVuaXRzIjoxLCJ1bml0IjoiV0lORE9XIn0sImRhdGEiOnsicmVzb3VyY2VBbGlhcyI6InJlc291cmNlLTEiLCJyZWFkQnl0ZXMiOjQwOTYsIndyaXRlQnl0ZXMiOjAsImlvQ291bnQiOjEsIndpbmRvd01pbGxpcyI6MTAwMH19" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzMiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkNyIsInRvb2xJZCI6InRvcC5kaXNrIiwic2NoZW1hVmVyc2lvbiI6MSwiY2xhc3NpZmljYXRpb24iOiJMMyIsImNvbnNlbnRVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDUiLCJwb2xpY3lSZXZpc2lvbiI6MiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwiZXhwaXJlc0F0IjoiMjAyNi0wOS0xM1QxMjowOTowMFoiLCJub25jZSI6Ik1qZDF3OGlVanRKeUp1S1lMR2NEcXZ5TWJyaUVTUjlQNDAxcTVwQzFTZzQiLCJkZXZpY2VLZXlJZCI6IjMxOTI5MzJjYWU4YmJjOGE1NzM0YTM2MDVmYTBlNDQzMGQ4MjkxZGUwZmVhMzg5ZDI4ZTgzMWZhZjNhZThhYTIiLCJwYXlsb2FkIjp7InBhdGgiOiJyZXN1bHQuanNvbiIsIm1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL2pzb24iLCJzaXplQnl0ZXMiOjYzMCwic2hhMjU2IjoiNjZlZWU2NzE0OTg5MGUxNTRkOTU5YjcwMWFlYWI5YTIzYWQxNzMwNjkxZTk3M2RlNWRjNTJjOGQ1NTc0OGVjMCJ9fQ==", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiQlNSbWxrMmp5bXp5ZHdlMUdzRFktdFVvejI1US01VnBNZkJlVjBQVl9yaFI3ZUtqNkZqcUMwUWVuckdjQ2U1U0paVkRVMDIzTUkxbFVLUndWZTJobHcifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "top.locks", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxM2MiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJQSVN3VmpDU3ZVRzFuQWpUUDYzY052VGNTbmdtX3hraHo2VXVMSWVKeXFJIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjIsInNoYTI1NiI6ImIyOTQwYjFlNGU1ZjNiOTFiYTk3Y2MxNTg4NTM0OTlkZWEzMDNkMDQ0NDkxZTMzMDJlYWU1YTA3ZDFkZTA2NWQiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiJkNzU4NzAzODA5ZGNjYTQzNmJmY2VmNDY2MTlmZjcyOGNhMzNkNTY0NjNkYmVjODcxZThlMTZlNjZiNGZjNmEzIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NTg4LCJzaGEyNTYiOiIwMTI0MmM4OTE2N2Q0NmQ5ZjQyY2Q5OWU1M2JjNmU0MmEzYTRlNTdlMWE5ZDEwZGQ3YjM5MTEwYjk2NjM2ZGRkIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiZGpLSlppSVFlV3J5UkRhblVGcktGMXJ2b1Z6STFuOEtrXzVTazAya09HUldNMjFmNGIzWjhLRVdTTWFLZWhraWV0WUVLZDBoTi14ck1tYTNHd18tN3cifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzQiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkOCIsInRvb2xJZCI6InRvcC5sb2NrcyIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDMiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiJQSVN3VmpDU3ZVRzFuQWpUUDYzY052VGNTbmdtX3hraHo2VXVMSWVKeXFJIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo1ODgsInNoYTI1NiI6IjAxMjQyYzg5MTY3ZDQ2ZDlmNDJjZDk5ZTUzYmM2ZTQyYTNhNGU1N2UxYTlkMTBkZDdiMzkxMTBiOTY2MzZkZGQifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiLXByTHdyT1dKc1pnbTFidFE0MGVJT3FpbVBtNGk3eHZ2NUJONEt5Nmx5NGhNX3M0dGVPMXJETDdHMkVOZW04QzgwWVlrRnNvQ19Fa1BPME5zcTlvNVEifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzQiLCJ0b29sSWQiOiJ0b3AubG9ja3MiLCJjYXBhYmlsaXR5IjoidG9wLmxvY2tzQDEiLCJvdXRjb21lIjoiU1VDQ0VFREVEIiwicmVhc29uQ29kZSI6IkNPTVBMRVRFIiwiZHVyYXRpb25NaWxsaXMiOjEwMDAsInByb3ZlbmFuY2UiOnsicmVwb3NpdG9yeSI6InJ1c3Rmcy9ydXN0ZnMiLCJzb3VyY2VDb21taXQiOiJhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhIiwiZXhlY3V0YWJsZVNoYTI1NiI6ImJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmIiLCJydXN0ZnNWZXJzaW9uIjoiMS4wLjAtcmMuNiIsIm9zRmFtaWx5IjoiTElOVVgiLCJhcmNoaXRlY3R1cmUiOiJhYXJjaDY0IiwiYnVpbGRGZWF0dXJlcyI6W119LCJjb3ZlcmFnZSI6eyJyZXF1ZXN0ZWRVbml0cyI6MSwiY29tcGxldGVkVW5pdHMiOjEsInVuaXQiOiJXSU5ET1cifSwiZGF0YSI6eyJoZWxkQ291bnQiOjIsIndhaXRpbmdDb3VudCI6MSwidHJ1bmNhdGVkIjpmYWxzZX19" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzQiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkOCIsInRvb2xJZCI6InRvcC5sb2NrcyIsInNjaGVtYVZlcnNpb24iOjEsImNsYXNzaWZpY2F0aW9uIjoiTDMiLCJjb25zZW50VWlkIjoiMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDA1IiwicG9saWN5UmV2aXNpb24iOjIsInByb2R1Y2VkQXQiOiIyMDI2LTA5LTEzVDExOjU5OjAwWiIsImV4cGlyZXNBdCI6IjIwMjYtMDktMTNUMTI6MDk6MDBaIiwibm9uY2UiOiJQSVN3VmpDU3ZVRzFuQWpUUDYzY052VGNTbmdtX3hraHo2VXVMSWVKeXFJIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwicGF5bG9hZCI6eyJwYXRoIjoicmVzdWx0Lmpzb24iLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi9qc29uIiwic2l6ZUJ5dGVzIjo1ODgsInNoYTI1NiI6IjAxMjQyYzg5MTY3ZDQ2ZDlmNDJjZDk5ZTUzYmM2ZTQyYTNhNGU1N2UxYTlkMTBkZDdiMzkxMTBiOTY2MzZkZGQifX0=", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoidVpvTzJHTGlvN0V6TVp3blNDcU9TSmxMeFptYjB2LTY3Q1dPRkRHcVVCQmg1WllDN0E1NlZMQ080aEZ2ejRJYmdwRjlCZWk4QUJvWFRLaHVlX3pEQ1EifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "top.net", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxM2QiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJlckJPRVVMQm5mblZmRVJwTS0zUjdVUWxQWFI2RDBsVWVPRG1YSnI2eThjIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjAsInNoYTI1NiI6IjYyMDc5ZTE0NTc0ZWQwNzI3YzJlZjZkYWJkMWVhZGMzNTVjNjgyZjg2YWM0ZTZmZWFlODU1YzdkZTIxZmY4NTYiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiI5OGFkY2I5YWYwZTcyYTczNjRhZmNjZjBmNmVhYjNjNDcxNmVmZDA4NmJkMjY2NGEwNmU1NWE4YWMzYTE1NjAzIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NTkyLCJzaGEyNTYiOiIyN2NiNGY4NDM4ZTBlNjg0NGNhZDFmMGU4ZDU1YmU1NTk0ZjgxMTIyM2Y5ODM1OGZiNGZmY2MxYTk4ZGU2ZDVkIiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiVDJ6a0FkeDMxeTMycnZRbEVDM1J4N0RoNE00WnN4NmhhQnF5QU9sQXdqTWJ0bVduNjRRQzlCcm5wRmttRk5WVjl3QmRueUNqSnZTbW9CX1ItVkE2encifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzUiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkOSIsInRvb2xJZCI6InRvcC5uZXQiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwzIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoiZXJCT0VVTEJuZm5WZkVScE0tM1I3VVFsUFhSNkQwbFVlT0RtWEpyNnk4YyIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NTkyLCJzaGEyNTYiOiIyN2NiNGY4NDM4ZTBlNjg0NGNhZDFmMGU4ZDU1YmU1NTk0ZjgxMTIyM2Y5ODM1OGZiNGZmY2MxYTk4ZGU2ZDVkIn19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiZHVJZUVfejFpZkpOdTBJdGFwMU5pX1I5Ykt1b0hyR25XMGZKUW1Lazh0a0N0QmM3Q1J4MWFDSVlJYlpoNUhmRXE0U2hCSWVEN185bzFkU09DUDdqMWcifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzUiLCJ0b29sSWQiOiJ0b3AubmV0IiwiY2FwYWJpbGl0eSI6InRvcC5uZXRAMSIsIm91dGNvbWUiOiJTVUNDRUVERUQiLCJyZWFzb25Db2RlIjoiQ09NUExFVEUiLCJkdXJhdGlvbk1pbGxpcyI6MTAwMCwicHJvdmVuYW5jZSI6eyJyZXBvc2l0b3J5IjoicnVzdGZzL3J1c3RmcyIsInNvdXJjZUNvbW1pdCI6ImFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWEiLCJleGVjdXRhYmxlU2hhMjU2IjoiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYiIsInJ1c3Rmc1ZlcnNpb24iOiIxLjAuMC1yYy42Iiwib3NGYW1pbHkiOiJMSU5VWCIsImFyY2hpdGVjdHVyZSI6ImFhcmNoNjQiLCJidWlsZEZlYXR1cmVzIjpbXX0sImNvdmVyYWdlIjp7InJlcXVlc3RlZFVuaXRzIjoxLCJjb21wbGV0ZWRVbml0cyI6MSwidW5pdCI6IldJTkRPVyJ9LCJkYXRhIjp7InJlY2VpdmVkQnl0ZXMiOjQwOTYsInNlbnRCeXRlcyI6MTI4LCJ3aW5kb3dNaWxsaXMiOjEwMDB9fQ==" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzUiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkOSIsInRvb2xJZCI6InRvcC5uZXQiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwzIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoiZXJCT0VVTEJuZm5WZkVScE0tM1I3VVFsUFhSNkQwbFVlT0RtWEpyNnk4YyIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NTkyLCJzaGEyNTYiOiIyN2NiNGY4NDM4ZTBlNjg0NGNhZDFmMGU4ZDU1YmU1NTk0ZjgxMTIyM2Y5ODM1OGZiNGZmY2MxYTk4ZGU2ZDVkIn19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiLWZsaUpIZXRiOVY3aFpFVkJxTVdMSXNzS2V0SDB5T1pBb1NyOEFiSVBwZFQ0WjZSTkpqUjVRMDBRVkd2cDFrU2NyMmxHUjRJNlAxcS1vc0xwT2JCNmcifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + }, + { + "toolId": "top.rpc", + "classification": "L3", + "expectedDisposition": "PUBLISH_TYPED_REPORT", + "archive": { + "manifestBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3Quc3VwcG9ydC5idW5kbGVNYW5pZmVzdC8xIiwicHJvdG9jb2xWZXJzaW9uIjoidjEiLCJidW5kbGVVaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAxM2UiLCJvcmdhbml6YXRpb25OYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEiLCJjbHVzdGVyTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMiIsImRldmljZU5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMS9jbHVzdGVycy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDIvY2x1c3RlckRldmljZXMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAzIiwiZGV2aWNlS2V5SWQiOiIzMTkyOTMyY2FlOGJiYzhhNTczNGEzNjA1ZmEwZTQ0MzBkODI5MWRlMGZlYTM4OWQyOGU4MzFmYWYzYWU4YWEyIiwibm9uY2UiOiJLUnhnMF9zb2pDOTBwZ3BCUFh3STVBOXIyenI3R2JqQWZFNUZhRVoyQ19RIiwicHJvZHVjZWRBdCI6IjIwMjYtMDktMTNUMTE6NTk6MDBaIiwicmVkYWN0aW9uVmVyc2lvbiI6InJ1c3Rmcy5jb25uZWN0LnJlZGFjdGlvbi52MSIsInJ1bGVzZXRIYXNoIjoiYjM3NDM2ZDhlNzI1MTUzOTRhMTIyZDYzMzg2NWIxZGMwMjhkNGVjZTM0OTM1MmEwYTNhMjNmNTJjYTQyODVmMyIsImNsYXNzaWZpY2F0aW9uUmVnaXN0cnlWZXJzaW9uIjoxLCJlbnRyaWVzIjpbeyJwYXRoIjoiZW52ZWxvcGUuanNvbiIsInR5cGUiOiJvZmZsaW5lLWRpYWdub3N0aWMiLCJzaXplQnl0ZXMiOjEwMjAsInNoYTI1NiI6ImVkYzg0MjIwMmZlMDdjN2MzZTZjZDZmOTk5Y2U3MjVlZWVjMGIzMTYzZWJjODVkNWFkM2IwYjY3OWRhMzE1YTYiLCJjbGFzc2lmaWNhdGlvbiI6IkwzIn0seyJwYXRoIjoiZW52ZWxvcGUuc2lnIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6MTkzLCJzaGEyNTYiOiI5OTgxMTYxNDU3MWY3MzUxNjViZDBhMjAzMjRmM2MwOTE4NjRkYmQ4NWRkNjUzZDg2OWNkYjE2MWIzMGViM2U0IiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9LHsicGF0aCI6InJlc3VsdC5qc29uIiwidHlwZSI6Im9mZmxpbmUtZGlhZ25vc3RpYyIsInNpemVCeXRlcyI6NjEzLCJzaGEyNTYiOiIwYTFhMGM4ZmQzMWQ1YWE4MDBiYzY3MGY5NTdkNWQ2ZjBlYWIxNmJjOTA0NGJmYTI3Y2Q4ZTliNWQ1MWFkMjI2IiwiY2xhc3NpZmljYXRpb24iOiJMMyJ9XX0=", + "manifestSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiRFpCeDJ2REs3U05zMlMyWWFkZmpzU19EbGZqSkxrSFpYSUdFR2p3Vk1BWVF5akNDU3REdkMza3hHMzhFRWZnWWd1Z0Q5cmEySGlKbXVCMmMyeXBGOEEifQ==", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDMiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzYiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkYSIsInRvb2xJZCI6InRvcC5ycGMiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwzIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoiS1J4ZzBfc29qQzkwcGdwQlBYd0k1QTlyMnpyN0diakFmRTVGYUVaMkNfUSIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NjEzLCJzaGEyNTYiOiIwYTFhMGM4ZmQzMWQ1YWE4MDBiYzY3MGY5NTdkNWQ2ZjBlYWIxNmJjOTA0NGJmYTI3Y2Q4ZTliNWQ1MWFkMjI2In19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiYjhNek1YbE5jSU9wVUNBLXh6ODNhZ0xOS2JORXVHZkJoOURzVkVHNG1hTXd4eVVjVWtSR1NQcXNvRkZ5SHVaWkNXaXJrc1lOQjJTN3lOUjVqQVVsT0EifQ==", + "resultBytes": "eyJzY2hlbWFWZXJzaW9uIjoxLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzYiLCJ0b29sSWQiOiJ0b3AucnBjIiwiY2FwYWJpbGl0eSI6InRvcC5ycGNAMSIsIm91dGNvbWUiOiJTVUNDRUVERUQiLCJyZWFzb25Db2RlIjoiQ09NUExFVEUiLCJkdXJhdGlvbk1pbGxpcyI6MTAwMCwicHJvdmVuYW5jZSI6eyJyZXBvc2l0b3J5IjoicnVzdGZzL3J1c3RmcyIsInNvdXJjZUNvbW1pdCI6ImFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWEiLCJleGVjdXRhYmxlU2hhMjU2IjoiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYiIsInJ1c3Rmc1ZlcnNpb24iOiIxLjAuMC1yYy42Iiwib3NGYW1pbHkiOiJMSU5VWCIsImFyY2hpdGVjdHVyZSI6ImFhcmNoNjQiLCJidWlsZEZlYXR1cmVzIjpbXX0sImNvdmVyYWdlIjp7InJlcXVlc3RlZFVuaXRzIjoxLCJjb21wbGV0ZWRVbml0cyI6MSwidW5pdCI6IldJTkRPVyJ9LCJkYXRhIjp7InJlcXVlc3RDb3VudCI6MiwiZXJyb3JDb3VudCI6MCwid2luZG93TWlsbGlzIjoxMDAwLCJ0b3RhbER1cmF0aW9uTWljcm9zIjo5MDB9fQ==" + }, + "negativeVectors": { + "tampered": { + "mutation": "APPEND_NEWLINE_TO_ENVELOPE", + "expectedReason": "SIGNATURE_INVALID" + }, + "replayed": { + "nonceAlreadyAccepted": true, + "expectedReason": "BUNDLE_REPLAYED" + }, + "foreignDevice": { + "deviceName": "organizations/019f0000-0000-7000-8000-000000000001/clusters/019f0000-0000-7000-8000-000000000002/clusterDevices/019f0000-0000-7000-8000-000000000004", + "envelopeBytes": "eyJmb3JtYXRWZXJzaW9uIjoicnVzdGZzLmNvbm5lY3QuZGlhZ25vc3RpY0VudmVsb3BlLzEiLCJwcm90b2NvbFZlcnNpb24iOiJ2MSIsIm9yZ2FuaXphdGlvbk5hbWUiOiJvcmdhbml6YXRpb25zLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMSIsImNsdXN0ZXJOYW1lIjoib3JnYW5pemF0aW9ucy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDEvY2x1c3RlcnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAyIiwiZGV2aWNlTmFtZSI6Im9yZ2FuaXphdGlvbnMvMDE5ZjAwMDAtMDAwMC03MDAwLTgwMDAtMDAwMDAwMDAwMDAxL2NsdXN0ZXJzLzAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwMi9jbHVzdGVyRGV2aWNlcy8wMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwMDQiLCJydW5VaWQiOiIwMTlmMDAwMC0wMDAwLTcwMDAtODAwMC0wMDAwMDAwMDAwNzYiLCJhcnRpZmFjdFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDBkYSIsInRvb2xJZCI6InRvcC5ycGMiLCJzY2hlbWFWZXJzaW9uIjoxLCJjbGFzc2lmaWNhdGlvbiI6IkwzIiwiY29uc2VudFVpZCI6IjAxOWYwMDAwLTAwMDAtNzAwMC04MDAwLTAwMDAwMDAwMDAwNSIsInBvbGljeVJldmlzaW9uIjoyLCJwcm9kdWNlZEF0IjoiMjAyNi0wOS0xM1QxMTo1OTowMFoiLCJleHBpcmVzQXQiOiIyMDI2LTA5LTEzVDEyOjA5OjAwWiIsIm5vbmNlIjoiS1J4ZzBfc29qQzkwcGdwQlBYd0k1QTlyMnpyN0diakFmRTVGYUVaMkNfUSIsImRldmljZUtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInBheWxvYWQiOnsicGF0aCI6InJlc3VsdC5qc29uIiwibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInNpemVCeXRlcyI6NjEzLCJzaGEyNTYiOiIwYTFhMGM4ZmQzMWQ1YWE4MDBiYzY3MGY5NTdkNWQ2ZjBlYWIxNmJjOTA0NGJmYTI3Y2Q4ZTliNWQ1MWFkMjI2In19", + "envelopeSignatureBytes": "eyJhbGdvcml0aG0iOiJFUzI1NiIsImtleUlkIjoiMzE5MjkzMmNhZThiYmM4YTU3MzRhMzYwNWZhMGU0NDMwZDgyOTFkZTBmZWEzODlkMjhlODMxZmFmM2FlOGFhMiIsInZhbHVlIjoiU04xUXZrcEhOUkhaWVF0NHZRNmtpVGJOa2JnVUlOdkhjRlVGMV8zS2Qzc0JJbDBYa3BzRDZuSG15a1JweW5zSm5Xb3RDdWplOGNsWmdXMk53dU91VVEifQ==", + "expectedReason": "DEVICE_MISMATCH" + } + } + } + ] +} diff --git a/protocol/agent/v1/fixtures/diagnostic-scheduler/MANIFEST.sha256 b/protocol/agent/v1/fixtures/diagnostic-scheduler/MANIFEST.sha256 new file mode 100644 index 000000000..c74e0ed28 --- /dev/null +++ b/protocol/agent/v1/fixtures/diagnostic-scheduler/MANIFEST.sha256 @@ -0,0 +1,3 @@ +29c93b4c8c27164896c5197e1df4ba03175425ccf5b12dc8e3f3d8d6bf19dc51 accept-vectors.json +fef527fdd89aa87fa9f81394eab6484fc2ac05a31559e1589149716193b336fb receipt-vectors.json +219749ea7bdf73461108320e13ba14b604a1967e32139c5317a017ad95dab000 reject-vectors.json diff --git a/protocol/agent/v1/fixtures/diagnostic-scheduler/accept-vectors.json b/protocol/agent/v1/fixtures/diagnostic-scheduler/accept-vectors.json new file mode 100644 index 000000000..069b75b93 --- /dev/null +++ b/protocol/agent/v1/fixtures/diagnostic-scheduler/accept-vectors.json @@ -0,0 +1,169 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "diagnostic-scheduler", + "capability": "diagnostics.policy.v1", + "toolCapability": "inventory.environment@1", + "classification": "L1", + "description": "Frozen vectors for the consent-bound RustFS diagnostic scheduler. Receipts describe local producer execution and do not claim Connect delivery.", + "fixture": "accept-vectors", + "producerEvidence": { + "repository": "rustfs/rustfs", + "sourceCommit": "37250f649a92fc816a7b45e1cae096bef6943a77", + "executableSha256": "236066133e15d5f3fb9ffd36702069959527c97be184b1b757c1375ea0ebfb9a", + "testCommand": "cargo test -p rustfs --test connect_diagnostic_schedule -- --nocapture", + "testsPassed": 4, + "testsFailed": 0 + }, + "limits": { + "minimumIntervalSeconds": 300, + "maximumIntervalSeconds": 86400, + "maximumRetentionDays": 14, + "maximumAttempts": 3, + "maximumResultBytes": 65536 + }, + "vectors": [ + { + "id": "diagnostic-scheduler.periodic-success", + "policy": { + "policyName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/diagnosticCollectionPreference", + "revision": 7, + "supportState": "SUPPORTED", + "desiredState": "RUNNING", + "reasonCode": null, + "toolIds": ["inventory.environment"], + "intervalSeconds": 300, + "scope": "CLUSTER", + "retentionDays": 7, + "consentReference": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/diagnosticCollectionConsents/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92", + "consentExpiresAt": "2099-01-01T00:00:00Z" + }, + "receipt": { + "receiptId": "123e4567-e89b-42d3-a456-426614174000", + "policyRevision": 7, + "toolId": "inventory.environment", + "intervalStartedAt": "2030-01-01T00:00:00Z", + "completedAt": "2030-01-01T00:00:01Z", + "outcome": "SUCCEEDED", + "attemptCount": 1, + "reason": null, + "resultSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "resultBytes": 256 + }, + "expected": { + "scheduled": true, + "nextIntervalStartedAt": "2030-01-01T00:05:00Z", + "receiptPersistence": "LOCAL_DURABLE_STATE", + "connectVisibility": false + } + }, + { + "id": "diagnostic-scheduler.failure-after-bounded-retries", + "policy": { + "policyName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/diagnosticCollectionPreference", + "revision": 8, + "supportState": "SUPPORTED", + "desiredState": "RUNNING", + "reasonCode": null, + "toolIds": ["inventory.environment"], + "intervalSeconds": 300, + "scope": "CLUSTER", + "retentionDays": 7, + "consentReference": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/diagnosticCollectionConsents/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92", + "consentExpiresAt": "2099-01-01T00:00:00Z" + }, + "receipt": { + "receiptId": "123e4567-e89b-42d3-a456-426614174001", + "policyRevision": 8, + "toolId": "inventory.environment", + "intervalStartedAt": "2030-01-01T00:00:00Z", + "completedAt": "2030-01-01T00:00:07Z", + "outcome": "FAILED", + "attemptCount": 3, + "reason": "connect_diagnostic_inventory_unavailable", + "resultSha256": null, + "resultBytes": null + }, + "expected": { + "scheduled": true, + "retryDelaysSeconds": [1, 2], + "receiptPersistence": "LOCAL_DURABLE_STATE", + "connectVisibility": false + } + }, + { + "id": "diagnostic-scheduler.disable-cancels-active-run", + "policy": { + "policyName": null, + "revision": 9, + "supportState": "SUPPORTED", + "desiredState": "STOPPED", + "reasonCode": "DISABLED", + "toolIds": [], + "intervalSeconds": null, + "scope": "CLUSTER", + "retentionDays": null, + "consentReference": null, + "consentExpiresAt": null + }, + "receipt": { + "receiptId": "123e4567-e89b-42d3-a456-426614174002", + "policyRevision": 8, + "toolId": "inventory.environment", + "intervalStartedAt": "2030-01-01T00:00:00Z", + "completedAt": "2030-01-01T00:00:01Z", + "outcome": "CANCELLED", + "attemptCount": 1, + "reason": "connect_diagnostic_cancelled", + "resultSha256": null, + "resultBytes": null + }, + "expected": { + "scheduled": false, + "activeRunCancelled": true, + "nextDueAt": null, + "receiptPersistence": "LOCAL_DURABLE_STATE", + "connectVisibility": false + } + }, + { + "id": "diagnostic-scheduler.restart-does-not-repeat-interval", + "policy": { + "policyName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/diagnosticCollectionPreference", + "revision": 10, + "supportState": "SUPPORTED", + "desiredState": "RUNNING", + "reasonCode": null, + "toolIds": ["inventory.environment"], + "intervalSeconds": 300, + "scope": "CLUSTER", + "retentionDays": 7, + "consentReference": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/diagnosticCollectionConsents/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92", + "consentExpiresAt": "2099-01-01T00:00:00Z" + }, + "persistedState": { + "policyRevision": 10, + "policyFingerprint": "81f75b115ea87f4d2eb90ae5e5c94ae8b0aead668a6a5a324af8ccfb265bbee1", + "nextDueAt": "2030-01-01T00:05:00Z", + "activeIntervalStartedAt": null, + "lastReceipt": { + "receiptId": "123e4567-e89b-42d3-a456-426614174003", + "policyRevision": 10, + "toolId": "inventory.environment", + "intervalStartedAt": "2030-01-01T00:00:00Z", + "completedAt": "2030-01-01T00:00:01Z", + "outcome": "FAILED", + "attemptCount": 3, + "reason": "connect_diagnostic_inventory_unavailable", + "resultSha256": null, + "resultBytes": null + } + }, + "expected": { + "scheduled": true, + "rerunLastInterval": false, + "nextIntervalStartedAt": "2030-01-01T00:05:00Z", + "connectVisibility": false + } + } + ] +} diff --git a/protocol/agent/v1/fixtures/diagnostic-scheduler/receipt-vectors.json b/protocol/agent/v1/fixtures/diagnostic-scheduler/receipt-vectors.json new file mode 100644 index 000000000..6abfcb3c3 --- /dev/null +++ b/protocol/agent/v1/fixtures/diagnostic-scheduler/receipt-vectors.json @@ -0,0 +1,69 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "diagnostic-scheduler", + "classification": "L1", + "description": "Outbound-only diagnostic execution receipt ingestion. Connect binds the route to the authenticated device, retains exact replays, and rejects changed or expired claims.", + "fixture": "receipt-vectors", + "accepted": [ + { + "id": "diagnostic-receipt.success", + "request": { + "protocolVersion": "v1", + "requestId": "123e4567-e89b-42d3-a456-426614174000", + "receiptId": "123e4567-e89b-42d3-a456-426614174000", + "policyRevision": 7, + "toolId": "inventory.environment", + "intervalStartedAt": "2030-01-01T00:00:00Z", + "completedAt": "2030-01-01T00:00:01Z", + "outcome": "SUCCEEDED", + "attemptCount": 1, + "reason": null, + "resultSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "resultBytes": 256 + }, + "expected": { "httpStatus": 200, "connectVisible": true } + }, + { + "id": "diagnostic-receipt.failure-after-three-attempts", + "request": { + "protocolVersion": "v1", + "requestId": "123e4567-e89b-42d3-a456-426614174001", + "receiptId": "123e4567-e89b-42d3-a456-426614174001", + "policyRevision": 8, + "toolId": "inventory.environment", + "intervalStartedAt": "2030-01-01T00:00:00Z", + "completedAt": "2030-01-01T00:00:07Z", + "outcome": "FAILED", + "attemptCount": 3, + "reason": "connect_diagnostic_inventory_unavailable", + "resultSha256": null, + "resultBytes": null + }, + "expected": { "httpStatus": 200, "connectVisible": true } + }, + { + "id": "diagnostic-receipt.disable-or-revoke-cancelled", + "request": { + "protocolVersion": "v1", + "requestId": "123e4567-e89b-42d3-a456-426614174002", + "receiptId": "123e4567-e89b-42d3-a456-426614174002", + "policyRevision": 8, + "toolId": "inventory.environment", + "intervalStartedAt": "2030-01-01T00:00:00Z", + "completedAt": "2030-01-01T00:00:01Z", + "outcome": "CANCELLED", + "attemptCount": 1, + "reason": "connect_diagnostic_cancelled", + "resultSha256": null, + "resultBytes": null + }, + "expected": { "httpStatus": 200, "connectVisible": true } + } + ], + "rejected": [ + { "id": "diagnostic-receipt.changed-replay", "expected": { "httpStatus": 409, "reason": "REQUEST_ID_REUSED" } }, + { "id": "diagnostic-receipt.expired", "expected": { "httpStatus": 400, "reason": "DIAGNOSTIC_RECEIPT_INVALID" } }, + { "id": "diagnostic-receipt.wrong-cluster", "expected": { "httpStatus": 403, "reason": "CLUSTER_MISMATCH" } }, + { "id": "diagnostic-receipt.expired-device", "expected": { "httpStatus": 401, "reason": "CREDENTIAL_EXPIRED" } } + ] +} diff --git a/protocol/agent/v1/fixtures/diagnostic-scheduler/reject-vectors.json b/protocol/agent/v1/fixtures/diagnostic-scheduler/reject-vectors.json new file mode 100644 index 000000000..8083f62a5 --- /dev/null +++ b/protocol/agent/v1/fixtures/diagnostic-scheduler/reject-vectors.json @@ -0,0 +1,112 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "diagnostic-scheduler", + "capability": "diagnostics.policy.v1", + "toolCapability": "inventory.environment@1", + "classification": "L1", + "description": "Rejected consent-bound scheduler inputs and state. Rejection stops collection and never turns missing evidence into a successful receipt.", + "fixture": "reject-vectors", + "vectors": [ + { + "id": "diagnostic-scheduler.interval-below-cap", + "policy": { + "policyName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/diagnosticCollectionPreference", + "revision": 11, + "supportState": "SUPPORTED", + "desiredState": "RUNNING", + "reasonCode": null, + "toolIds": ["inventory.environment"], + "intervalSeconds": 299, + "scope": "CLUSTER", + "retentionDays": 7, + "consentReference": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/diagnosticCollectionConsents/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92", + "consentExpiresAt": "2099-01-01T00:00:00Z" + }, + "expected": { "accepted": false, "reason": "connect_diagnostic_policy_invalid" } + }, + { + "id": "diagnostic-scheduler.retention-above-cap", + "policy": { + "policyName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/diagnosticCollectionPreference", + "revision": 12, + "supportState": "SUPPORTED", + "desiredState": "RUNNING", + "reasonCode": null, + "toolIds": ["inventory.environment"], + "intervalSeconds": 300, + "scope": "CLUSTER", + "retentionDays": 15, + "consentReference": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/diagnosticCollectionConsents/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92", + "consentExpiresAt": "2099-01-01T00:00:00Z" + }, + "expected": { "accepted": false, "reason": "connect_diagnostic_policy_invalid" } + }, + { + "id": "diagnostic-scheduler.unsupported-tool", + "policy": { + "policyName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/diagnosticCollectionPreference", + "revision": 13, + "supportState": "SUPPORTED", + "desiredState": "RUNNING", + "reasonCode": null, + "toolIds": ["logs.capture"], + "intervalSeconds": 300, + "scope": "CLUSTER", + "retentionDays": 7, + "consentReference": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/diagnosticCollectionConsents/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92", + "consentExpiresAt": "2099-01-01T00:00:00Z" + }, + "expected": { "accepted": false, "reason": "connect_diagnostic_policy_invalid" } + }, + { + "id": "diagnostic-scheduler.expired-consent", + "policy": { + "policyName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/diagnosticCollectionPreference", + "revision": 14, + "supportState": "SUPPORTED", + "desiredState": "RUNNING", + "reasonCode": null, + "toolIds": ["inventory.environment"], + "intervalSeconds": 300, + "scope": "CLUSTER", + "retentionDays": 7, + "consentReference": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/diagnosticCollectionConsents/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92", + "consentExpiresAt": "2020-01-01T00:00:00Z" + }, + "expected": { "accepted": false, "reason": "connect_diagnostic_policy_invalid" } + }, + { + "id": "diagnostic-scheduler.changed-policy-with-same-revision", + "policy": { + "policyName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/diagnosticCollectionPreference", + "revision": 15, + "supportState": "SUPPORTED", + "desiredState": "RUNNING", + "reasonCode": null, + "toolIds": ["inventory.environment"], + "intervalSeconds": 600, + "scope": "CLUSTER", + "retentionDays": 7, + "consentReference": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/diagnosticCollectionConsents/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92", + "consentExpiresAt": "2099-01-01T00:00:00Z" + }, + "persistedState": { + "policyRevision": 15, + "policyFingerprint": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "nextDueAt": "2030-01-01T00:05:00Z", + "activeIntervalStartedAt": null, + "lastReceipt": null + }, + "expected": { "accepted": false, "reason": "connect_diagnostic_state_corrupt" } + }, + { + "id": "diagnostic-scheduler.result-over-size-cap", + "resultBytes": 65537, + "expected": { + "accepted": false, + "receiptOutcome": "FAILED", + "reason": "connect_diagnostic_result_too_large" + } + } + ] +} diff --git a/protocol/agent/v1/fixtures/fixture-sets.json b/protocol/agent/v1/fixtures/fixture-sets.json index 72c4c1919..151c95840 100644 --- a/protocol/agent/v1/fixtures/fixture-sets.json +++ b/protocol/agent/v1/fixtures/fixture-sets.json @@ -33,6 +33,31 @@ "status": "populated", "purpose": "Inventory snapshot payloads and their allow-listed collection fields." }, + { + "name": "object-performance", + "status": "populated", + "purpose": "Bounded performance.object@1 result acceptance and rejection vectors." + }, + { + "name": "network-performance", + "status": "populated", + "purpose": "Bounded performance.network@1 aggregate results, peer attribution, unit semantics, and budget rejection vectors." + }, + { + "name": "site-replication-performance", + "status": "populated", + "purpose": "Bounded performance.siteReplication@1 aggregate results, destination-confirmation semantics, and rejection of operational secrets and unresolved target controls." + }, + { + "name": "telemetry", + "status": "populated", + "purpose": "Consent-bound telemetry.record@1, telemetry.otlp@1, and telemetry.replay@1 producer results, resource limits, redaction boundaries, cancellation, and signed local exports." + }, + { + "name": "diagnostic-scheduler", + "status": "populated", + "purpose": "Consent-bound diagnostic policy scheduling, bounded retry, cancellation, restart deduplication, and outbound Connect-visible execution receipts." + }, { "name": "offline-enrollment", "status": "populated", @@ -41,7 +66,7 @@ { "name": "bundle", "status": "populated", - "purpose": "Support bundle manifests, upload authorization, and archive validation." + "purpose": "Support bundle manifests, upload authorization, archive validation, and signed typed offline diagnostic imports." }, { "name": "redaction", diff --git a/protocol/agent/v1/fixtures/heartbeat/MANIFEST.sha256 b/protocol/agent/v1/fixtures/heartbeat/MANIFEST.sha256 index 7aa8e3191..f5271d17f 100644 --- a/protocol/agent/v1/fixtures/heartbeat/MANIFEST.sha256 +++ b/protocol/agent/v1/fixtures/heartbeat/MANIFEST.sha256 @@ -1,5 +1,6 @@ 975c1ca53eefeef6766a6fc0b3d3281f7408255342b0686e5e2aee5ad055414c duplicate.json 963529a38a02849c6c2acc6d72668dca9f63218b49c89fae41a451b584850411 overflow.json +6afec3806c98fb55d477455b43935798d8fdb7f9ae39bd9f6637465539e5c02b producer-capabilities.json e3adeee1c8a19aa17e70894896fb79c072e3785bea3611b93c11e79f039ed5af stale.json -35b9cebd8525389a701e8fe69fbe96407bcb31aa28392fe95babf4a4886985ad unknown.json -37941735dbd6ad3d238258a7b2cae6f0b3aa0ecaae1d8817817c3d718d11d633 valid.json +22cc7337f545271ebf11ca6b76e7e479d62c51278ac001a8d870b07d0fa9884e unknown.json +b06a72ff1d964efe65e996797705ec2ef72574a813aec5d8e9aff26c899f1352 valid.json diff --git a/protocol/agent/v1/fixtures/heartbeat/producer-capabilities.json b/protocol/agent/v1/fixtures/heartbeat/producer-capabilities.json new file mode 100644 index 000000000..716d651f2 --- /dev/null +++ b/protocol/agent/v1/fixtures/heartbeat/producer-capabilities.json @@ -0,0 +1,54 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "heartbeat", + "fixture": "producer-capabilities", + "description": "The exact base and diagnostic capability tokens published by a RustFS agent heartbeat.", + "baseCapabilities": [ + "heartbeat", + "diagnostics.policy.v1", + "inventory.environment@1" + ], + "producerCapabilities": [ + "performance.client@1", + "performance.drive@1", + "performance.network@1", + "performance.object@1", + "performance.siteReplication@1", + "logs.capture@1", + "profile.cpu@1", + "profile.memory@1", + "profile.threads@1", + "telemetry.record@1", + "telemetry.otlp@1", + "telemetry.replay@1", + "top.api@1", + "top.disk@1", + "top.locks@1", + "top.net@1", + "top.rpc@1", + "inspect.object@1" + ], + "heartbeatCapabilities": [ + "heartbeat", + "diagnostics.policy.v1", + "inventory.environment@1", + "performance.client@1", + "performance.drive@1", + "performance.network@1", + "performance.object@1", + "performance.siteReplication@1", + "logs.capture@1", + "profile.cpu@1", + "profile.memory@1", + "profile.threads@1", + "telemetry.record@1", + "telemetry.otlp@1", + "telemetry.replay@1", + "top.api@1", + "top.disk@1", + "top.locks@1", + "top.net@1", + "top.rpc@1", + "inspect.object@1" + ] +} diff --git a/protocol/agent/v1/fixtures/heartbeat/unknown.json b/protocol/agent/v1/fixtures/heartbeat/unknown.json index 6639cc0ff..cb9ace735 100644 --- a/protocol/agent/v1/fixtures/heartbeat/unknown.json +++ b/protocol/agent/v1/fixtures/heartbeat/unknown.json @@ -6,13 +6,13 @@ "requestAdditions": { "telemetryProfile": "extended", "authorization": "Bearer non-functional-example", - "capabilities": ["heartbeat", "future.capability"], + "capabilities": ["heartbeat", "inventory.environment@1", "inventory.environment@2", "future.capability"], "coarseNodeSummary": {"rackNames": ["customer-rack"]} }, "expected": { "decision": "ACCEPT", - "storedCapabilities": ["heartbeat"], - "discarded": ["authorization", "future.capability", "telemetryProfile", "coarseNodeSummary.rackNames"], + "storedCapabilities": ["heartbeat", "inventory.environment@1"], + "discarded": ["authorization", "future.capability", "inventory.environment@2", "telemetryProfile", "coarseNodeSummary.rackNames"], "echoed": [] } } diff --git a/protocol/agent/v1/fixtures/heartbeat/valid.json b/protocol/agent/v1/fixtures/heartbeat/valid.json index ed3561b80..850a22f4f 100644 --- a/protocol/agent/v1/fixtures/heartbeat/valid.json +++ b/protocol/agent/v1/fixtures/heartbeat/valid.json @@ -15,7 +15,7 @@ "expected": { "decision": "ACCEPT", "acceptedVersion": "v1", - "responseFields": ["serverTime", "acceptedVersion", "capabilityHints"], + "responseFields": ["serverTime", "acceptedVersion", "capabilityHints", "diagnosticCollectionPolicy"], "onlineAuthority": "serverTime" } } diff --git a/protocol/agent/v1/fixtures/inventory/MANIFEST.sha256 b/protocol/agent/v1/fixtures/inventory/MANIFEST.sha256 index cc0d06f4c..f1aef5b5c 100644 --- a/protocol/agent/v1/fixtures/inventory/MANIFEST.sha256 +++ b/protocol/agent/v1/fixtures/inventory/MANIFEST.sha256 @@ -1,3 +1,4 @@ +58d61bb10b9443eca3c0e9dc340cbe91144eed32a705617cc3406bd825439824 accept-vectors.json d1a73b0a348845bf3ed9fb68301babc5abbc6e72243a610db1cd4794b1328070 canonical-hash.json f1107d3e6accbaee468f1a1fdb79d7103fb2aadafd85c39020fbbca5173b03b4 field-registry.json b3b2e7f761198d4823c94440637a48153437183f4cacec5118570e1920f73b29 old-agent-vectors.json diff --git a/protocol/agent/v1/fixtures/inventory/accept-vectors.json b/protocol/agent/v1/fixtures/inventory/accept-vectors.json new file mode 100644 index 000000000..772d85a91 --- /dev/null +++ b/protocol/agent/v1/fixtures/inventory/accept-vectors.json @@ -0,0 +1,58 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "inventory", + "fixture": "accept-vectors", + "description": "Producer vectors for the negotiated inventory.environment@1 summary. The input names source observations used by the RustFS collector; output is the complete identifier-free payload allowed to leave the deployment.", + "schemaVersion": 1, + "capability": "inventory.environment@1", + "vectors": [ + { + "name": "known two-node deployment", + "input": { + "persistedInventory": { "nodeCount": 2, "driveCount": 8 }, + "sourceObservation": { "osFamily": "LINUX", "filesystemTypes": ["xfs"] } + }, + "expected": { + "decision": "ACCEPT", + "output": { "nodeCount": 2, "driveCount": 8, "osFamily": "LINUX", "filesystemTypes": ["xfs"] } + } + }, + { + "name": "secret-like filesystem observations are not exported", + "input": { + "persistedInventory": { "nodeCount": 2, "driveCount": 8 }, + "sourceObservation": { + "osFamily": "LINUX", + "filesystemTypes": ["xfs"], + "mountPath": "/srv/synthetic-customer-a", + "mountOptions": "rw,password=SYNTHETIC_NOT_A_REAL_SECRET", + "deviceLabel": "synthetic-customer-volume", + "credential": "SYNTHETIC_NOT_A_REAL_CREDENTIAL" + } + }, + "expected": { + "decision": "ACCEPT", + "discarded": ["credential", "deviceLabel", "mountOptions", "mountPath"], + "output": { "nodeCount": 2, "driveCount": 8, "osFamily": "LINUX", "filesystemTypes": ["xfs"] } + } + }, + { + "name": "old schema is rejected before collection", + "input": { "schemaVersion": 0, "capability": "inventory.environment@1" }, + "expected": { "decision": "REJECT", "producerError": "inventory_environment_unsupported_version" } + }, + { + "name": "unknown capability is rejected before collection", + "input": { "schemaVersion": 1, "capability": "inventory.environment@2" }, + "expected": { "decision": "REJECT", "producerError": "inventory_environment_unsupported_capability" } + }, + { + "name": "missing filesystem source is not replaced with an empty success", + "input": { + "persistedInventory": { "nodeCount": 2, "driveCount": 8 }, + "sourceObservation": { "osFamily": "LINUX", "filesystemTypes": [] } + }, + "expected": { "decision": "UNSUPPORTED", "producerError": "inventory_environment_source_unavailable", "output": null } + } + ] +} diff --git a/protocol/agent/v1/fixtures/manifest.sha256 b/protocol/agent/v1/fixtures/manifest.sha256 new file mode 100644 index 000000000..4ffe6e680 --- /dev/null +++ b/protocol/agent/v1/fixtures/manifest.sha256 @@ -0,0 +1,13 @@ +a499742c06046e9cf2f781fc17d8b4cf95e018ab45fa41ee5944ba15b0e2902c auth +9b54e39584a442270ae486e8378df1d6f168c45d0c59baba9459b46bb212d2df version +159d116e965b51771f1687fc6ec9bdfbf2a4a0e6d6aa26d3f1046b3dac2697dd registration +fe1287eac16832bed88320254245ba07a985e6ae7e66e1d1570e324ffa972873 heartbeat +cc6c3f3bf5e6a938f2e00b8ce42d8fa7bfb974f4eb2927101d4ede4ebdc10dd5 inventory +0207458ec9b97b368e2f347da112511a3aabd27d752890cacda4fa2408e7f19e object-performance +a464d4c695d76dd985c0f2abf68c2135c080221098ad2427022f85243a7bf6ae network-performance +813cbbf209e8fc9ddaafaeb4eb4745024b1c0edcb26634073a85023844141477 site-replication-performance +fad21eec58d4d4d547893af9050cdc5eaf9d4f84b15fb423214b16be50e7ff7a telemetry +484da2e7599148d36e1ac19d0ae5d305c4a96da7a00495a8ea202d3bae240a88 diagnostic-scheduler +a2557ab1f7f70fb86a8affc8451596c272478fd2d5500c367b8f9e8385d72578 offline-enrollment +8b6fd07759d1264489a8bb54d2dda4836ad0f59b5ee5d52966365918b3de4c9e bundle +42cc936dc8fe87335ebff5abe01e3ebf85bdada7f37b43bfd8add46435c9d6f9 redaction diff --git a/protocol/agent/v1/fixtures/network-performance/MANIFEST.sha256 b/protocol/agent/v1/fixtures/network-performance/MANIFEST.sha256 new file mode 100644 index 000000000..f02b8dfa6 --- /dev/null +++ b/protocol/agent/v1/fixtures/network-performance/MANIFEST.sha256 @@ -0,0 +1,2 @@ +e886a13ecc53a2f67d490bed9f0af06e0cf6a998d6eafa6e58291aec94ee96cd accept-vectors.json +57a3c119a13a098e3a5fcee9775f7c0eac8d11f98652c64abf2e04bd0b69f0f1 reject-vectors.json diff --git a/protocol/agent/v1/fixtures/network-performance/accept-vectors.json b/protocol/agent/v1/fixtures/network-performance/accept-vectors.json new file mode 100644 index 000000000..c9e713847 --- /dev/null +++ b/protocol/agent/v1/fixtures/network-performance/accept-vectors.json @@ -0,0 +1,294 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "network-performance", + "toolId": "performance.network", + "schemaVersion": 1, + "capability": "performance.network@1", + "classification": "L1", + "description": "Frozen performance.network@1 aggregate results and producer-side peer attribution. Probe bytes are synthetic and stay inside the RustFS deployment. Peer aliases are opaque; addresses, endpoints, payload bodies, credentials, and free text are forbidden.", + "fixture": "accept-vectors", + "vectors": [ + { + "id": "performance.network.succeeded", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 1048576, + "durationMillis": 1000, + "errorCount": 0, + "peerCount": 2 + } + }, + "peerResults": [ + { + "peerAlias": "peer-1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "transferredBytes": 524288, + "durationMillis": 900, + "latencyMicros": 1500 + }, + { + "peerAlias": "peer-2", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "transferredBytes": 524288, + "durationMillis": 1000, + "latencyMicros": 2200 + } + ], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.network.partial-disconnected-peer", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "PARTIAL", + "reasonCode": "COLLECTION_FAILED", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 524288, + "durationMillis": 1000, + "errorCount": 1, + "peerCount": 2 + } + }, + "peerResults": [ + { + "peerAlias": "peer-1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "transferredBytes": 524288, + "durationMillis": 900, + "latencyMicros": 1500 + }, + { + "peerAlias": "peer-2", + "outcome": "FAILED", + "reasonCode": "UNREACHABLE", + "transferredBytes": 0, + "durationMillis": 12, + "latencyMicros": null + } + ], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.network.failed-slow-peer", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "FAILED", + "reasonCode": "COLLECTION_FAILED", + "durationMillis": 30000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "OPERATION" + }, + "data": null + }, + "peerResults": [ + { + "peerAlias": "peer-1", + "outcome": "FAILED", + "reasonCode": "TIMED_OUT", + "transferredBytes": 0, + "durationMillis": 30000, + "latencyMicros": null + } + ], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.network.unsupported-without-topology", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "UNSUPPORTED", + "reasonCode": "SOURCE_UNAVAILABLE", + "durationMillis": 0, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 0, + "unit": "OPERATION" + }, + "data": null + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.network.cancelled-after-one-peer", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "CANCELLED", + "reasonCode": "CANCELLED", + "durationMillis": 200, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 1, + "unit": "OPERATION" + }, + "data": null + }, + "peerResults": [ + { + "peerAlias": "peer-1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "transferredBytes": 524288, + "durationMillis": 180, + "latencyMicros": 1500 + } + ], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": true, + "reason": null + } + } + ] +} diff --git a/protocol/agent/v1/fixtures/network-performance/reject-vectors.json b/protocol/agent/v1/fixtures/network-performance/reject-vectors.json new file mode 100644 index 000000000..b304bbb14 --- /dev/null +++ b/protocol/agent/v1/fixtures/network-performance/reject-vectors.json @@ -0,0 +1,505 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "network-performance", + "toolId": "performance.network", + "schemaVersion": 1, + "capability": "performance.network@1", + "classification": "L1", + "description": "Frozen performance.network@1 aggregate results and producer-side peer attribution. Probe bytes are synthetic and stay inside the RustFS deployment. Peer aliases are opaque; addresses, endpoints, payload bodies, credentials, and free text are forbidden.", + "fixture": "reject-vectors", + "vectors": [ + { + "id": "performance.network.over-concurrency-budget-precedes-document-validation", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 1048576, + "durationMillis": 1000, + "errorCount": 0, + "peerCount": 2 + }, + "unexpected": "must-not-be-parsed" + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 2, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": false, + "reason": "INVALID_LIMITS" + } + }, + { + "id": "performance.network.over-traffic-budget-precedes-document-validation", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 1048576, + "durationMillis": 1000, + "errorCount": 0, + "peerCount": 2 + }, + "unexpected": "must-not-be-parsed" + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048577, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": false, + "reason": "INVALID_LIMITS" + } + }, + { + "id": "performance.network.transferred-bytes-over-budget", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 1048577, + "durationMillis": 1000, + "errorCount": 0, + "peerCount": 2 + } + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": false, + "reason": "RESULT_LIMIT_EXCEEDED" + } + }, + { + "id": "performance.network.bandwidth-over-budget", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 999, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 1048576, + "durationMillis": 999, + "errorCount": 0, + "peerCount": 2 + } + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": false, + "reason": "RESULT_LIMIT_EXCEEDED" + } + }, + { + "id": "performance.network.elapsed-time-mismatch", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 999, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 1048576, + "durationMillis": 1000, + "errorCount": 0, + "peerCount": 2 + } + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": false, + "reason": "INVALID_SCHEMA" + } + }, + { + "id": "performance.network.partial-without-peer-error", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "PARTIAL", + "reasonCode": "COLLECTION_FAILED", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 524288, + "durationMillis": 1000, + "errorCount": 0, + "peerCount": 2 + } + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": false, + "reason": "INVALID_SCHEMA" + } + }, + { + "id": "performance.network.partial-with-all-peers-failed", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "PARTIAL", + "reasonCode": "COLLECTION_FAILED", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 0, + "durationMillis": 1000, + "errorCount": 2, + "peerCount": 2 + } + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": false, + "reason": "INVALID_SCHEMA" + } + }, + { + "id": "performance.network.peer-count-disagrees-with-coverage", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 1048576, + "durationMillis": 1000, + "errorCount": 0, + "peerCount": 1 + } + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": false, + "reason": "INVALID_SCHEMA" + } + }, + { + "id": "performance.network.customer-address-field", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 2, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 1048576, + "durationMillis": 1000, + "errorCount": 0, + "peerCount": 2, + "peerAddress": "10.0.0.8" + } + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": false, + "reason": "INVALID_SCHEMA" + } + }, + { + "id": "performance.network.unsupported-with-measurement-data", + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.network", + "capability": "performance.network@1", + "outcome": "UNSUPPORTED", + "reasonCode": "SOURCE_UNAVAILABLE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 0, + "unit": "OPERATION" + }, + "data": { + "transferredBytes": 1048576, + "durationMillis": 1000, + "errorCount": 0, + "peerCount": 2 + } + }, + "peerResults": [], + "declaredLimits": { + "maxDurationMillis": 30000, + "maxResultBytes": 262144, + "maxWorkingMemoryBytes": 67108864, + "maxRecords": 1024, + "maxConcurrency": 1, + "maxCpuMillis": 5000, + "maxTemporaryBytes": 2097152, + "maxTrafficBytes": 1048576, + "maxBandwidthBytesPerSecond": 1048576, + "maxOperations": 1024 + }, + "expected": { + "accepted": false, + "reason": "INVALID_SCHEMA" + } + } + ] +} diff --git a/protocol/agent/v1/fixtures/object-performance/MANIFEST.sha256 b/protocol/agent/v1/fixtures/object-performance/MANIFEST.sha256 new file mode 100644 index 000000000..d1768292b --- /dev/null +++ b/protocol/agent/v1/fixtures/object-performance/MANIFEST.sha256 @@ -0,0 +1,2 @@ +24cd96339f27034a0e2bb656ee0c7da9bec9326ded4583a3c9afef06595626b2 accept-vectors.json +a9f4376ea5bbf636851e683cdfaca0cb7ebbc9dfce8ffa4dc8b4a12f82307a53 reject-vectors.json diff --git a/protocol/agent/v1/fixtures/object-performance/accept-vectors.json b/protocol/agent/v1/fixtures/object-performance/accept-vectors.json new file mode 100644 index 000000000..20cc02ccc --- /dev/null +++ b/protocol/agent/v1/fixtures/object-performance/accept-vectors.json @@ -0,0 +1,222 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "object-performance", + "toolId": "performance.object", + "schemaVersion": 1, + "capability": "performance.object@1", + "classification": "L1", + "description": "Frozen performance.object@1 result vectors. Payload bytes are synthetic and remain on the customer deployment; bucket, object, endpoint, credential, and free-text fields are forbidden.", + "fixture": "accept-vectors", + "vectors": [ + { + "id": "performance.object.succeeded", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "GET_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 1000, + "errorCount": 0 + } + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.object.put-succeeded", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "PUT_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 1000, + "errorCount": 0 + } + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.object.partial", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "PARTIAL", + "reasonCode": "LIMIT_EXCEEDED", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "GET_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 1000, + "errorCount": 0 + } + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.object.failed", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "FAILED", + "reasonCode": "COLLECTION_FAILED", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 0, + "unit": "WINDOW" + }, + "data": null + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.object.unsupported", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "UNSUPPORTED", + "reasonCode": "UNSUPPORTED_TOOL", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 0, + "unit": "WINDOW" + }, + "data": null + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.object.cancelled", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "CANCELLED", + "reasonCode": "CANCELLED", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 0, + "unit": "WINDOW" + }, + "data": null + }, + "expected": { + "accepted": true, + "reason": null + } + } + ] +} diff --git a/protocol/agent/v1/fixtures/object-performance/reject-vectors.json b/protocol/agent/v1/fixtures/object-performance/reject-vectors.json new file mode 100644 index 000000000..da77f847b --- /dev/null +++ b/protocol/agent/v1/fixtures/object-performance/reject-vectors.json @@ -0,0 +1,319 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "object-performance", + "toolId": "performance.object", + "schemaVersion": 1, + "capability": "performance.object@1", + "classification": "L1", + "description": "Frozen performance.object@1 result vectors. Payload bytes are synthetic and remain on the customer deployment; bucket, object, endpoint, credential, and free-text fields are forbidden.", + "fixture": "reject-vectors", + "vectors": [ + { + "id": "performance.object.unknown-data-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "GET_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 1000, + "errorCount": 0, + "rawData": "SYNTHETIC_FORBIDDEN_BYTES" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + }, + { + "id": "performance.object.unknown-version", + "schemaValid": false, + "value": { + "schemaVersion": 2, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "GET_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 1000, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "UNSUPPORTED_SCHEMA" + } + }, + { + "id": "performance.object.unsupported-with-data", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "UNSUPPORTED", + "reasonCode": "UNSUPPORTED_TOOL", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "GET_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 1000, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "INVALID_OUTCOME_DATA" + } + }, + { + "id": "performance.object.error-count-exceeds-operations", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "GET_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 1000, + "errorCount": 2 + } + }, + "expected": { + "accepted": false, + "reason": "INVALID_MEASUREMENT" + } + }, + { + "id": "performance.object.zero-data-duration", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "GET_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 0, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "INVALID_MEASUREMENT" + } + }, + { + "id": "performance.object.unknown-operation", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "DELETE_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 1000, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_OPERATION" + } + }, + { + "id": "performance.object.customer-namespace-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "GET_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 1000, + "errorCount": 0, + "bucket": "customer-bucket" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + }, + { + "id": "performance.object.credential-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.object", + "capability": "performance.object@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "operation": "GET_OBJECT", + "transferredBytes": 1048576, + "completedOperations": 1, + "durationMillis": 1000, + "errorCount": 0, + "accessKey": "SYNTHETIC_FORBIDDEN_CREDENTIAL" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + } + ] +} diff --git a/protocol/agent/v1/fixtures/site-replication-performance/MANIFEST.sha256 b/protocol/agent/v1/fixtures/site-replication-performance/MANIFEST.sha256 new file mode 100644 index 000000000..f7df42341 --- /dev/null +++ b/protocol/agent/v1/fixtures/site-replication-performance/MANIFEST.sha256 @@ -0,0 +1,2 @@ +392877f31c52cb762963e9fae32300eb84e44ed695b8915f66241fc319304bcc accept-vectors.json +bf58aa86a7a075935c491cb88fc92e4f9611020a1a9ae27af87e3c55d82df86c reject-vectors.json diff --git a/protocol/agent/v1/fixtures/site-replication-performance/accept-vectors.json b/protocol/agent/v1/fixtures/site-replication-performance/accept-vectors.json new file mode 100644 index 000000000..2b2792766 --- /dev/null +++ b/protocol/agent/v1/fixtures/site-replication-performance/accept-vectors.json @@ -0,0 +1,184 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "site-replication-performance", + "toolId": "performance.siteReplication", + "schemaVersion": 1, + "capability": "performance.siteReplication@1", + "classification": "L2", + "description": "Performance.siteReplication@1 result vectors for the bounded native producer. Results contain only aggregate destination-confirmed measurements; endpoint URLs, credentials, CA material or paths, scratch bucket names, object/version identifiers, and cleanup controls are forbidden.", + "fixture": "accept-vectors", + "vectors": [ + { + "id": "performance.siteReplication.succeeded", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0 + } + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.siteReplication.partial", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "PARTIAL", + "reasonCode": "LIMIT_EXCEEDED", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 2, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0 + } + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.siteReplication.failed", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "FAILED", + "reasonCode": "COLLECTION_FAILED", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 0, + "unit": "WINDOW" + }, + "data": null + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.siteReplication.unsupported", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "UNSUPPORTED", + "reasonCode": "UNSUPPORTED_TOOL", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 0, + "unit": "WINDOW" + }, + "data": null + }, + "expected": { + "accepted": true, + "reason": null + } + }, + { + "id": "performance.siteReplication.cancelled", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "CANCELLED", + "reasonCode": "CANCELLED", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 0, + "unit": "WINDOW" + }, + "data": null + }, + "expected": { + "accepted": true, + "reason": null + } + } + ] +} diff --git a/protocol/agent/v1/fixtures/site-replication-performance/reject-vectors.json b/protocol/agent/v1/fixtures/site-replication-performance/reject-vectors.json new file mode 100644 index 000000000..0d814e89e --- /dev/null +++ b/protocol/agent/v1/fixtures/site-replication-performance/reject-vectors.json @@ -0,0 +1,590 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "site-replication-performance", + "toolId": "performance.siteReplication", + "schemaVersion": 1, + "capability": "performance.siteReplication@1", + "classification": "L2", + "description": "Performance.siteReplication@1 result vectors for the bounded native producer. Results contain only aggregate destination-confirmed measurements; endpoint URLs, credentials, CA material or paths, scratch bucket names, object/version identifiers, and cleanup controls are forbidden.", + "fixture": "reject-vectors", + "vectors": [ + { + "id": "performance.siteReplication.unknown-data-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0, + "rawData": "SYNTHETIC_FORBIDDEN_BYTES" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + }, + { + "id": "performance.siteReplication.unknown-version", + "schemaValid": false, + "value": { + "schemaVersion": 2, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "UNSUPPORTED_SCHEMA" + } + }, + { + "id": "performance.siteReplication.unsupported-with-data", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "UNSUPPORTED", + "reasonCode": "UNSUPPORTED_TOOL", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "INVALID_OUTCOME_DATA" + } + }, + { + "id": "performance.siteReplication.source-endpoint-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0, + "sourceEndpoint": "https://source.example.invalid" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + }, + { + "id": "performance.siteReplication.destination-endpoint-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0, + "destinationEndpoint": "https://destination.example.invalid" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + }, + { + "id": "performance.siteReplication.credential-file-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0, + "credentialFile": "/private/operator/credentials.json" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + }, + { + "id": "performance.siteReplication.ca-file-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0, + "caFile": "/private/operator/ca.pem" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + }, + { + "id": "performance.siteReplication.scratch-bucket-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0, + "scratchBucket": "connect-replication-scratch" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + }, + { + "id": "performance.siteReplication.cleanup-version-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0, + "cleanupVersionId": "synthetic-version-id" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + }, + { + "id": "performance.siteReplication.late-arrival-field", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0, + "lateArrivalCleanup": "DELETE_ALL_VERSIONS" + } + }, + "expected": { + "accepted": false, + "reason": "UNKNOWN_FIELD" + } + }, + { + "id": "performance.siteReplication.zero-data-duration", + "schemaValid": false, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 0, + "maxObservedLagMillis": 250, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "INVALID_MEASUREMENT" + } + }, + { + "id": "performance.siteReplication.traffic-limit-exceeded", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048577, + "confirmedObjects": 1, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "RESULT_LIMIT_EXCEEDED" + } + }, + { + "id": "performance.siteReplication.duration-limit-exceeded", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 30000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1, + "durationMillis": 30001, + "maxObservedLagMillis": 250, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "RESULT_LIMIT_EXCEEDED" + } + }, + { + "id": "performance.siteReplication.operation-limit-exceeded", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 1025, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "RESULT_LIMIT_EXCEEDED" + } + }, + { + "id": "performance.siteReplication.succeeded-without-destination-confirmation", + "schemaValid": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "performance.siteReplication", + "capability": "performance.siteReplication@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "aarch64", + "buildFeatures": [] + }, + "coverage": { + "requestedUnits": 1, + "completedUnits": 1, + "unit": "WINDOW" + }, + "data": { + "replicatedBytes": 1048576, + "confirmedObjects": 0, + "durationMillis": 1000, + "maxObservedLagMillis": 250, + "errorCount": 0 + } + }, + "expected": { + "accepted": false, + "reason": "INVALID_MEASUREMENT" + } + } + ] +} diff --git a/protocol/agent/v1/fixtures/telemetry/MANIFEST.sha256 b/protocol/agent/v1/fixtures/telemetry/MANIFEST.sha256 new file mode 100644 index 000000000..65d742e89 --- /dev/null +++ b/protocol/agent/v1/fixtures/telemetry/MANIFEST.sha256 @@ -0,0 +1,2 @@ +1fd1bf2b23a9db2e83ec53f4d9b90449bc7d58f9a586a0bff2dbbe057426b4b3 accept-vectors.json +18f33760811bb599b0914cb36b9a636f8f41a2558bb7e9aaec34eaaff38e0678 reject-vectors.json diff --git a/protocol/agent/v1/fixtures/telemetry/accept-vectors.json b/protocol/agent/v1/fixtures/telemetry/accept-vectors.json new file mode 100644 index 000000000..f4d02f3ef --- /dev/null +++ b/protocol/agent/v1/fixtures/telemetry/accept-vectors.json @@ -0,0 +1,138 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "telemetry", + "schemaVersion": 1, + "classification": "L3", + "fixture": "accept-vectors", + "producer": { + "repository": "rustfs/rustfs", + "headCommit": "85b8dc44d7d06d17ad5f5324b0a2766889d873e5", + "mergeCommit": "f2cdca42eddd44681cd220d7b9dfdbbe91c719a1", + "pullRequest": 7720, + "evidenceUrl": "https://github.com/rustfs/rustfs/pull/7720", + "verifiedRuns": { + "record": "UNSUPPORTED_SOURCE_UNAVAILABLE", + "otlp": "REAL_BINARY_LOOPBACK_VERIFIED", + "replay": "REAL_BINARY_ARCHIVE_VERIFIED" + }, + "sourceFiles": [ + "rustfs/src/connect/diagnostics/trace_record.rs", + "rustfs/src/connect/diagnostics/trace_otlp.rs", + "rustfs/src/connect/diagnostics/trace_replay.rs", + "rustfs/src/connect/diagnostics/trace_analysis.rs" + ], + "testCommand": "cargo test -p rustfs --test connect_trace_analysis --test connect_trace_otlp --test connect_trace_record --test connect_trace_replay" + }, + "limits": { + "maxDurationMillis": 30000, + "maxSpans": 1024, + "maxResultBytes": 262144, + "maxOtlpBodyBytes": 1048576, + "maxArchiveBytes": 524288, + "maxDecompressedBytes": 278528, + "maxConcurrency": 1 + }, + "signedExport": { + "members": ["envelope.json", "envelope.sig", "result.json"], + "algorithm": "ES256", + "signatureDomain": "rustfs-diagnostic-envelope-v1\u0000", + "fileMode": "0600", + "noClobber": true, + "classification": "L3", + "requiresConfirmedConsent": true + }, + "vectors": [ + { + "id": "telemetry.record.source-unavailable", + "evidenceType": "SOURCE_VERIFIED_CONTRACT_VECTOR", + "accepted": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "telemetry.record", + "capability": "telemetry.record@1", + "outcome": "UNSUPPORTED", + "reasonCode": "UNSUPPORTED_TOOL", + "durationMillis": 1, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "85b8dc44d7d06d17ad5f5324b0a2766889d873e5", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "x86_64", + "buildFeatures": [] + }, + "coverage": {"requestedUnits": 1, "completedUnits": 0, "unit": "WINDOW"}, + "data": null + }, + "expected": { + "artifactCreated": false, + "reason": "The current TraceBus has no approved typed S3 or RPC source." + } + }, + { + "id": "telemetry.otlp.succeeded", + "evidenceType": "SYNTHETIC_CONTRACT_VECTOR_WITH_REAL_BINARY_RUN", + "accepted": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "telemetry.otlp", + "capability": "telemetry.otlp@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 10, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "85b8dc44d7d06d17ad5f5324b0a2766889d873e5", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "x86_64", + "buildFeatures": [] + }, + "coverage": {"requestedUnits": 1, "completedUnits": 1, "unit": "WINDOW"}, + "data": {"acceptedSpanCount": 2, "rejectedSpanCount": 0, "exportedBytes": 128} + }, + "expected": { + "artifactCreated": true, + "credentialsRemainLocal": true, + "receiptContainsCredentials": false + } + }, + { + "id": "telemetry.replay.succeeded", + "evidenceType": "SYNTHETIC_CONTRACT_VECTOR_WITH_REAL_BINARY_RUN", + "accepted": true, + "value": { + "schemaVersion": 1, + "runUid": "019e3ae0-0000-7000-8000-000000000001", + "toolId": "telemetry.replay", + "capability": "telemetry.replay@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "85b8dc44d7d06d17ad5f5324b0a2766889d873e5", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0-rc.6", + "osFamily": "LINUX", + "architecture": "x86_64", + "buildFeatures": [] + }, + "coverage": {"requestedUnits": 1, "completedUnits": 1, "unit": "WINDOW"}, + "data": { + "inputArtifactSha256": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "spans": [{"operation": "GET_OBJECT", "durationMicros": 500, "status": "OK"}] + } + }, + "expected": { + "artifactCreated": true, + "inputWasLocallyReviewed": true, + "analysisUsesObservedValuesOnly": true + } + } + ] +} diff --git a/protocol/agent/v1/fixtures/telemetry/reject-vectors.json b/protocol/agent/v1/fixtures/telemetry/reject-vectors.json new file mode 100644 index 000000000..718011117 --- /dev/null +++ b/protocol/agent/v1/fixtures/telemetry/reject-vectors.json @@ -0,0 +1,28 @@ +{ + "protocolVersion": "v1", + "fixtureSet": "telemetry", + "schemaVersion": 1, + "classification": "L3", + "fixture": "reject-vectors", + "vectors": [ + {"id": "telemetry.consent.missing", "stage": "REQUEST", "input": {"acknowledgeL3": false}, "expected": {"accepted": false, "reason": "CONSENT_REQUIRED", "artifactCreated": false}}, + {"id": "telemetry.consent.expired", "stage": "REQUEST", "input": {"consentExpiresAt": "2026-09-11T23:59:59Z", "producedAt": "2026-09-12T00:00:00Z"}, "expected": {"accepted": false, "reason": "CONSENT_EXPIRED", "artifactCreated": false}}, + {"id": "telemetry.record.duration-zero", "stage": "REQUEST", "input": {"durationMillis": 0, "maxSpans": 1}, "expected": {"accepted": false, "reason": "INVALID_DURATION", "artifactCreated": false}}, + {"id": "telemetry.record.duration-over-limit", "stage": "REQUEST", "input": {"durationMillis": 30001, "maxSpans": 1}, "expected": {"accepted": false, "reason": "INVALID_DURATION", "artifactCreated": false}}, + {"id": "telemetry.record.spans-zero", "stage": "REQUEST", "input": {"durationMillis": 1, "maxSpans": 0}, "expected": {"accepted": false, "reason": "INVALID_SPAN_LIMIT", "artifactCreated": false}}, + {"id": "telemetry.record.spans-over-limit", "stage": "REQUEST", "input": {"durationMillis": 1, "maxSpans": 1025}, "expected": {"accepted": false, "reason": "INVALID_SPAN_LIMIT", "artifactCreated": false}}, + {"id": "telemetry.record.stopped", "stage": "CAPTURE", "input": {"cancelled": true}, "expected": {"accepted": false, "reason": "CANCELLED", "artifactCreated": false}}, + {"id": "telemetry.record.source-unavailable-with-data", "stage": "RESULT", "input": {"outcome": "UNSUPPORTED", "reasonCode": "UNSUPPORTED_TOOL", "data": {"spans": [], "droppedSpanCount": 0}}, "expected": {"accepted": false, "reason": "INVALID_OUTCOME_DATA", "artifactCreated": false}}, + {"id": "telemetry.record.header-field", "stage": "RESULT", "input": {"data": {"spans": [{"operation": "GET_OBJECT", "durationMicros": 1, "status": "OK", "authorization": "SYNTHETIC_FORBIDDEN_SECRET"}], "droppedSpanCount": 0}}, "expected": {"accepted": false, "reason": "UNKNOWN_FIELD", "artifactCreated": false}}, + {"id": "telemetry.record.customer-name-field", "stage": "RESULT", "input": {"data": {"spans": [{"operation": "GET_OBJECT", "durationMicros": 1, "status": "OK", "objectName": "SYNTHETIC_FORBIDDEN_OBJECT"}], "droppedSpanCount": 0}}, "expected": {"accepted": false, "reason": "UNKNOWN_FIELD", "artifactCreated": false}}, + {"id": "telemetry.otlp.remote-plaintext", "stage": "REQUEST", "input": {"endpoint": "http://collector.example/v1/traces"}, "expected": {"accepted": false, "reason": "INVALID_ENDPOINT", "artifactCreated": false}}, + {"id": "telemetry.otlp.body-over-limit", "stage": "REQUEST", "input": {"bodySizeBytes": 1048577}, "expected": {"accepted": false, "reason": "INVALID_BATCH", "artifactCreated": false}}, + {"id": "telemetry.otlp.credential-in-receipt", "stage": "RESULT", "input": {"data": {"acceptedSpanCount": 1, "rejectedSpanCount": 0, "exportedBytes": 128, "authorization": "SYNTHETIC_FORBIDDEN_SECRET"}}, "expected": {"accepted": false, "reason": "UNKNOWN_FIELD", "artifactCreated": false}}, + {"id": "telemetry.replay.unreviewed", "stage": "REQUEST", "input": {"locallyReviewed": false}, "expected": {"accepted": false, "reason": "INVALID_INPUT", "artifactCreated": false}}, + {"id": "telemetry.replay.input-over-limit", "stage": "REQUEST", "input": {"inputSizeBytes": 262145}, "expected": {"accepted": false, "reason": "INVALID_INPUT", "artifactCreated": false}}, + {"id": "telemetry.export.unsigned", "stage": "EXPORT", "input": {"members": ["envelope.json", "result.json"]}, "expected": {"accepted": false, "reason": "SIGNATURE_REQUIRED", "artifactCreated": false}}, + {"id": "telemetry.export.tampered", "stage": "EXPORT", "input": {"payloadDigestMatches": false}, "expected": {"accepted": false, "reason": "DIGEST_MISMATCH", "artifactCreated": false}}, + {"id": "telemetry.export.archive-over-limit", "stage": "EXPORT", "input": {"archiveSizeBytes": 524289}, "expected": {"accepted": false, "reason": "LIMIT_EXCEEDED", "artifactCreated": false}}, + {"id": "telemetry.export.existing-output", "stage": "EXPORT", "input": {"outputExists": true}, "expected": {"accepted": false, "reason": "ALREADY_EXISTS", "artifactCreated": false}} + ] +} diff --git a/rustfs/Cargo.toml b/rustfs/Cargo.toml index 3230d47bd..11bd1c8e0 100644 --- a/rustfs/Cargo.toml +++ b/rustfs/Cargo.toml @@ -74,7 +74,7 @@ connect-e2e-short-credentials = [] # Builds the dedicated rustfs-cli-e2e target with a build-time public enrollment root. offline-enrollment-e2e-root = [] rio-v2 = ["rustfs-ecstore/rio-v2"] -pyroscope = ["rustfs-obs/pyroscope"] +pyroscope = ["rustfs-obs/pyroscope", "dep:pyroscope"] # Tokio runtime telemetry. Requires `--cfg tokio_unstable`; use `make build-profiling`. dial9 = ["rustfs-obs/dial9"] hotpath = [ @@ -307,6 +307,7 @@ bytes = { workspace = true, features = ["serde"] } chrono = { workspace = true, features = ["serde"] } flate2 = { workspace = true } flatbuffers.workspace = true +prost.workspace = true rmp-serde.workspace = true quick-xml.workspace = true rustfs-signer.workspace = true @@ -317,15 +318,17 @@ snap.workspace = true zstd.workspace = true # Cryptography and Security +ed25519-dalek.workspace = true rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] } rustls-pki-types = { workspace = true } +rustix.workspace = true x509-parser = { workspace = true } subtle = { workspace = true } jiff = { workspace = true, features = ["serde"] } time = { workspace = true, features = ["parsing", "formatting", "serde", "macros"] } # Utilities and Tools -astral-tokio-tar = { workspace = true } +rustfs-tokio-tar = { workspace = true } atoi = { workspace = true } atomic_enum = { workspace = true } async_zip = { workspace = true, default-features = false, features = ["tokio", "deflate"] } @@ -364,6 +367,7 @@ chacha20poly1305 = { workspace = true } # Observability and Metrics metrics = { workspace = true } opentelemetry = { workspace = true } +opentelemetry-proto = { workspace = true, features = ["trace"] } tracing-opentelemetry = { workspace = true } # Data structures hashbrown = { workspace = true, features = ["serde", "rayon"] } @@ -372,6 +376,9 @@ rustfs-mimalloc = { workspace = true } [target.'cfg(target_os = "linux")'.dependencies] libsystemd.workspace = true +[target.'cfg(any(all(target_os = "macos", any(target_arch = "x86_64", target_arch = "aarch64")), all(target_os = "linux", target_env = "gnu", any(target_arch = "x86_64", target_arch = "aarch64"))))'.dependencies] +pyroscope = { workspace = true, optional = true, features = ["backend-pprof-rs"] } + [dev-dependencies] uuid = { workspace = true, features = ["v4", "v5", "fast-rng", "macro-diagnostics"] } serial_test = { workspace = true } diff --git a/rustfs/src/admin/console.rs b/rustfs/src/admin/console.rs index d50fedc48..a46a3e795 100644 --- a/rustfs/src/admin/console.rs +++ b/rustfs/src/admin/console.rs @@ -22,9 +22,9 @@ use crate::server::rate_limit::{ apply_throttle_headers, client_ip, }; use crate::server::{ - APPLE_TOUCH_ICON_PATH, APPLE_TOUCH_ICON_PRECOMPOSED_PATH, CONSOLE_PREFIX, FAVICON_PATH, HEALTH_PREFIX, HEALTH_READY_PATH, - HeaderMapCarrier, HealthProbe, LICENSE, RUSTFS_ADMIN_PREFIX, RequestContextLayer, VERSION, build_health_response_parts, - collect_probe_readiness, + APPLE_TOUCH_ICON_PATH, APPLE_TOUCH_ICON_PRECOMPOSED_PATH, FAVICON_PATH, HEALTH_PREFIX, HEALTH_READY_PATH, HeaderMapCarrier, + HealthProbe, LICENSE, RUSTFS_ADMIN_PREFIX, RequestContextLayer, VERSION, build_health_response_parts, + collect_probe_readiness, console_prefix, }; use crate::version::{self, build}; use axum::{ @@ -83,7 +83,7 @@ async fn static_handler(uri: Uri) -> impl IntoResponse { return Response::builder() .status(StatusCode::OK) .header("Content-Type", mime_type.to_string()) - .body(Body::from(file.data)) + .body(Body::from(rewrite_console_asset(path, file.data, crate::server::console_prefix()))) .unwrap(); } @@ -95,7 +95,7 @@ async fn static_handler(uri: Uri) -> impl IntoResponse { return Response::builder() .status(StatusCode::OK) .header("Content-Type", mime_type.to_string()) - .body(Body::from(file.data)) + .body(Body::from(rewrite_console_asset(&index_path, file.data, crate::server::console_prefix()))) .unwrap(); } } @@ -106,7 +106,11 @@ async fn static_handler(uri: Uri) -> impl IntoResponse { Response::builder() .status(StatusCode::OK) .header("Content-Type", mime_type.to_string()) - .body(Body::from(file.data)) + .body(Body::from(rewrite_console_asset( + "index.html", + file.data, + crate::server::console_prefix(), + ))) .unwrap() } else { Response::builder() @@ -116,6 +120,57 @@ async fn static_handler(uri: Uri) -> impl IntoResponse { } } +// Next exports bake the base path into HTML, chunk loaders, and RSC text payloads. +// Rewrite only path references, preserving external URLs such as the source repository. +fn rewrite_console_asset<'a>(path: &str, data: std::borrow::Cow<'a, [u8]>, prefix: &str) -> std::borrow::Cow<'a, [u8]> { + use std::borrow::Cow; + + if prefix == crate::server::CONSOLE_PREFIX + || !matches!( + path.rsplit('.').next(), + Some("html" | "js" | "css" | "json" | "txt" | "webmanifest" | "svg") + ) + { + return data; + } + let Ok(text) = std::str::from_utf8(&data) else { + return data; + }; + let mut rewritten = Cow::Borrowed(text); + let escaped_default = crate::server::CONSOLE_PREFIX.replace('/', "\\/"); + let escaped_prefix = prefix.replace('/', "\\/"); + for (source, target) in [ + (crate::server::CONSOLE_PREFIX, prefix), + (escaped_default.as_str(), escaped_prefix.as_str()), + ] { + let mut output = String::new(); + let mut copied = 0; + for (offset, _) in rewritten.match_indices(source) { + let end = offset + source.len(); + let before = rewritten.as_bytes().get(offset.wrapping_sub(1)).copied(); + let after = rewritten.as_bytes().get(end).copied(); + let starts_path = before + .is_none_or(|byte| byte.is_ascii_whitespace() || matches!(byte, b'"' | b'\'' | b'`' | b'(' | b'=' | b'}' | b'>')); + let ends_prefix = after.is_none_or(|byte| { + byte.is_ascii_whitespace() || matches!(byte, b'/' | b'\\' | b'"' | b'\'' | b'`' | b'?' | b'#' | b')' | b'<') + }); + if starts_path && ends_prefix { + output.push_str(&rewritten[copied..offset]); + output.push_str(target); + copied = end; + } + } + if copied != 0 { + output.push_str(&rewritten[copied..]); + rewritten = Cow::Owned(output); + } + } + match rewritten { + Cow::Borrowed(_) => data, + Cow::Owned(text) => Cow::Owned(text.into_bytes()), + } +} + #[derive(Debug, Serialize, Clone)] pub(crate) struct Config { #[serde(skip)] @@ -468,7 +523,7 @@ fn get_console_config_from_env() -> (bool, u32, u64, String) { /// - `true` if the path is for console access, `false` otherwise. pub fn is_console_path(path: &str) -> bool { matches!(path, FAVICON_PATH | APPLE_TOUCH_ICON_PATH | APPLE_TOUCH_ICON_PRECOMPOSED_PATH) - || has_path_prefix(path, CONSOLE_PREFIX) + || has_path_prefix(path, console_prefix()) } /// Setup comprehensive middleware stack with tower-http features @@ -487,34 +542,35 @@ fn setup_console_middleware_stack( rate_limit_rpm: u32, auth_timeout: u64, ) -> Router { + let console_prefix = console_prefix(); let mut app = Router::new() .route(FAVICON_PATH, get(static_handler)) - .route(&format!("{CONSOLE_PREFIX}{LICENSE}"), get(license_handler)) - .route(&format!("{CONSOLE_PREFIX}{VERSION}"), get(version_handler)) - .nest(CONSOLE_PREFIX, Router::new().fallback_service(get(static_handler))) + .route(&format!("{console_prefix}{LICENSE}"), get(license_handler)) + .route(&format!("{console_prefix}{VERSION}"), get(version_handler)) + .nest(console_prefix, Router::new().fallback_service(get(static_handler))) .fallback_service(get(static_handler)); if rustfs_utils::get_env_bool(rustfs_config::ENV_HEALTH_ENDPOINT_ENABLE, rustfs_config::DEFAULT_HEALTH_ENDPOINT_ENABLE) { app = app - .route(&format!("{CONSOLE_PREFIX}{HEALTH_PREFIX}"), get(health_check).head(health_check)) + .route(&format!("{console_prefix}{HEALTH_PREFIX}"), get(health_check).head(health_check)) .route( - &format!("{CONSOLE_PREFIX}{}", crate::server::HEALTH_COMPAT_LIVE_PATH), + &format!("{console_prefix}{}", crate::server::HEALTH_COMPAT_LIVE_PATH), get(health_check).head(health_check), ) - .route(&format!("{CONSOLE_PREFIX}{HEALTH_READY_PATH}"), get(health_check).head(health_check)); + .route(&format!("{console_prefix}{HEALTH_READY_PATH}"), get(health_check).head(health_check)); } else { // Keep disabled health probes from falling through to the SPA fallback. app = app .route( - &format!("{CONSOLE_PREFIX}{HEALTH_PREFIX}"), + &format!("{console_prefix}{HEALTH_PREFIX}"), get(health_route_disabled).head(health_route_disabled), ) .route( - &format!("{CONSOLE_PREFIX}{}", crate::server::HEALTH_COMPAT_LIVE_PATH), + &format!("{console_prefix}{}", crate::server::HEALTH_COMPAT_LIVE_PATH), get(health_route_disabled).head(health_route_disabled), ) .route( - &format!("{CONSOLE_PREFIX}{HEALTH_READY_PATH}"), + &format!("{console_prefix}{HEALTH_READY_PATH}"), get(health_route_disabled).head(health_route_disabled), ); } @@ -624,7 +680,7 @@ async fn health_check( uri: Uri, server_ctx: Option>>, ) -> Response { - let probe = if uri.path().strip_prefix(CONSOLE_PREFIX) == Some(HEALTH_READY_PATH) { + let probe = if uri.path().strip_prefix(console_prefix()) == Some(HEALTH_READY_PATH) { HealthProbe::Readiness } else { HealthProbe::Liveness @@ -786,6 +842,7 @@ pub(crate) fn make_console_server() -> Router { #[cfg(test)] mod tests { use super::*; + use crate::server::CONSOLE_PREFIX; use axum::body::Body; use axum::routing::get; use http::{Request, StatusCode}; @@ -867,9 +924,14 @@ mod tests { async fn console_config_handler_serializes_admin_discovery_paths() { init_console_cfg(IpAddr::V4(Ipv4Addr::LOCALHOST), 9001); - let response = config_handler(Uri::from_static("http://127.0.0.1:9001/rustfs/console/api/v1/config"), HeaderMap::new()) - .await - .into_response(); + let response = config_handler( + format!("http://127.0.0.1:9001{CONSOLE_PREFIX}/api/v1/config") + .parse() + .expect("console URI"), + HeaderMap::new(), + ) + .await + .into_response(); assert_eq!(response.status(), StatusCode::OK); let body = response.into_body(); @@ -889,7 +951,8 @@ mod tests { #[test] fn external_admin_paths_are_not_console_paths() { - assert!(is_console_path("/rustfs/console/")); + assert!(is_console_path(&format!("{CONSOLE_PREFIX}/"))); + assert!(!is_console_path(&format!("{CONSOLE_PREFIX}-other/index.html"))); assert!(is_console_path("/apple-touch-icon.png")); assert!(is_console_path("/apple-touch-icon-precomposed.png")); assert!(!is_console_path("/minio/admin/v3/info")); @@ -1273,3 +1336,87 @@ mod tests { assert!(value.get("expired").is_none()); } } + +#[cfg(test)] +mod console_asset_prefix_tests { + use super::rewrite_console_asset; + use crate::server::CONSOLE_PREFIX; + use std::borrow::Cow; + + #[test] + fn rewrites_exported_assets_and_client_routes() { + let fixtures = [ + ( + "index.html", + r#""#, + r#""#, + ), + ( + "app.js", + r#"let base="/rustfs/console";fetch(`${host}/rustfs/console/version`)"#, + r#"let base="/console";fetch(`${host}/console/version`)"#, + ), + ("app.css", "url(/rustfs/console/logo.svg)", "url(/console/logo.svg)"), + ( + "route.txt", + r#"2:I[1,["/rustfs/console/_next/app.js"],"default"]"#, + r#"2:I[1,["/console/_next/app.js"],"default"]"#, + ), + ("config.json", r#"{"url":"\/rustfs\/console\/login"}"#, r#"{"url":"\/console\/login"}"#), + ("site.webmanifest", r#"{"start_url":"/rustfs/console/"}"#, r#"{"start_url":"/console/"}"#), + ( + "logo.svg", + r#""#, + r#""#, + ), + ]; + for (path, input, expected) in fixtures { + let input = input + .replace("/rustfs/console", CONSOLE_PREFIX) + .replace("\\/rustfs\\/console", &CONSOLE_PREFIX.replace('/', "\\/")); + assert_eq!( + rewrite_console_asset(path, Cow::Borrowed(input.as_bytes()), "/console").as_ref(), + expected.as_bytes(), + "{path}" + ); + } + } + + #[test] + fn restores_the_standard_path_from_an_oem_build() { + let input = format!(r#""#); + let output = rewrite_console_asset("index.html", Cow::Borrowed(input.as_bytes()), rustfs_config::DEFAULT_CONSOLE_PREFIX); + assert_eq!(output.as_ref(), br#""#); + } + + #[test] + fn preserves_unrelated_urls_paths_and_default_bytes() { + let input = format!( + r#"["https://github.com/rustfs/console","/other{CONSOLE_PREFIX}","{CONSOLE_PREFIX}-extra","{CONSOLE_PREFIX}/index.html"]"# + ); + let expected = format!( + r#"["https://github.com/rustfs/console","/other{CONSOLE_PREFIX}","{CONSOLE_PREFIX}-extra","/console/index.html"]"# + ); + assert_eq!( + rewrite_console_asset("app.js", Cow::Borrowed(input.as_bytes()), "/console").as_ref(), + expected.as_bytes() + ); + let unchanged = rewrite_console_asset("app.js", Cow::Borrowed(input.as_bytes()), CONSOLE_PREFIX); + assert!(matches!(unchanged, Cow::Borrowed(_))); + assert_eq!(unchanged.as_ref(), input.as_bytes()); + } + + #[test] + fn preserves_binary_invalid_utf8_and_text_without_paths() { + let invalid_utf8 = [b"\xff".as_slice(), CONSOLE_PREFIX.as_bytes()].concat(); + for (path, bytes) in [ + ("image.png", CONSOLE_PREFIX.as_bytes()), + ("app.js", invalid_utf8.as_slice()), + ("app.js", b"https://github.com/rustfs/console".as_slice()), + ] { + let output = rewrite_console_asset(path, Cow::Borrowed(bytes), "/console"); + assert!(matches!(output, Cow::Borrowed(_)), "{path}"); + assert_eq!(output.as_ref(), bytes); + } + } +} diff --git a/rustfs/src/admin/handlers/diagnostics.rs b/rustfs/src/admin/handlers/diagnostics.rs index 18758b2fc..bba323b08 100644 --- a/rustfs/src/admin/handlers/diagnostics.rs +++ b/rustfs/src/admin/handlers/diagnostics.rs @@ -47,13 +47,20 @@ use std::task::{Context, Poll}; use std::time::{Duration, SystemTime}; use tokio::sync::{Semaphore, SemaphorePermit, mpsc}; use tokio_stream::wrappers::ReceiverStream; +use tokio_util::sync::CancellationToken; use tracing::warn; const CONTENT_TYPE_NDJSON: &str = "application/x-ndjson"; pub(crate) const CLIENT_DEVNULL_MAX_BYTES: u64 = 1024 * 1024 * 1024; pub(crate) const CLIENT_DEVNULL_MAX_DURATION: Duration = Duration::from_secs(30); pub(crate) const CLIENT_DEVNULL_MAX_CONCURRENCY: usize = 4; +pub(crate) const CLIENT_DEVNULL_SOURCE_MAX_BYTES: u64 = 1024 * 1024; +pub(crate) const NETWORK_PROBE_MAX_CONCURRENCY: usize = 1; +pub(crate) use crate::storage::storage_api::ecstore_rpc::{ + MAX_NETWORK_PROBE_BYTES as NETWORK_PROBE_MAX_BYTES, MAX_NETWORK_PROBE_DURATION as NETWORK_PROBE_MAX_DURATION, +}; static CLIENT_DEVNULL_ADMISSION: Semaphore = Semaphore::const_new(CLIENT_DEVNULL_MAX_CONCURRENCY); +static NETWORK_PROBE_ADMISSION: Semaphore = Semaphore::const_new(NETWORK_PROBE_MAX_CONCURRENCY); /// Cap on how many locks a single `top/locks` response enumerates, matching the /// MinIO default page size and bounding response size on busy clusters. @@ -124,6 +131,11 @@ pub fn register_diagnostics_route(r: &mut S3Router) -> std::io:: format!("{ADMIN_PREFIX}/v3/speedtest/client/devnull").as_str(), AdminOperation(&SpeedtestClientDevnullHandler {}), )?; + r.insert( + Method::GET, + format!("{ADMIN_PREFIX}/v3/speedtest/client/devnull").as_str(), + AdminOperation(&SpeedtestClientSourceHandler {}), + )?; Ok(()) } @@ -805,6 +817,17 @@ struct DriveSpeedtestEntry { write_latency_secs: f64, } +#[derive(Debug, Clone, Serialize)] +struct NetworkSpeedtestEntry { + peer_alias: String, + outcome: &'static str, + reason: &'static str, + transferred_bytes: u64, + duration_secs: f64, + #[serde(skip_serializing_if = "Option::is_none")] + latency_micros: Option, +} + #[derive(Debug, Clone, Serialize)] struct SpeedtestResponse { kind: &'static str, @@ -821,6 +844,8 @@ struct SpeedtestResponse { aggregate_write_throughput_bytes_per_sec: Option, #[serde(skip_serializing_if = "Vec::is_empty")] drives: Vec, + #[serde(skip_serializing_if = "Vec::is_empty")] + peers: Vec, /// Bytes drained by the net/devnull probe and how long it took. #[serde(skip_serializing_if = "Option::is_none")] rx_bytes: Option, @@ -861,6 +886,7 @@ async fn run_drive_speedtest() -> S3Result { aggregate_read_throughput_bytes_per_sec: Some(agg_read), aggregate_write_throughput_bytes_per_sec: Some(agg_write), drives, + peers: Vec::new(), rx_bytes: None, duration_secs: None, }) @@ -879,28 +905,112 @@ fn object_speedtest_unsupported() -> SpeedtestResponse { aggregate_read_throughput_bytes_per_sec: None, aggregate_write_throughput_bytes_per_sec: None, drives: Vec::new(), + peers: Vec::new(), rx_bytes: None, duration_secs: None, } } -fn net_speedtest_single_node() -> SpeedtestResponse { +async fn run_network_speedtest() -> SpeedtestResponse { + use crate::storage::storage_api::ecstore_rpc::{NetworkPeerProbeClient, NetworkPeerProbeError}; + + let Some(endpoint_pools) = crate::runtime_sources::current_endpoints_handle() else { + return unavailable_network_speedtest("cluster topology is not initialized"); + }; + let client = NetworkPeerProbeClient::from_endpoint_pools(&endpoint_pools); + let targets = client.targets(); + if targets.is_empty() { + return unavailable_network_speedtest("the current topology has no remote peers"); + } + let peer_count = u64::try_from(targets.len()).unwrap_or(u64::MAX); + let traffic_bytes = NETWORK_PROBE_MAX_BYTES.checked_div(peer_count).unwrap_or(0); + if traffic_bytes == 0 { + return unavailable_network_speedtest("the current topology exceeds the probe traffic budget"); + } + let Ok(_permit) = NETWORK_PROBE_ADMISSION.try_acquire() else { + return unavailable_network_speedtest("another inter-node network probe is already running"); + }; + + let started = std::time::Instant::now(); + let cancel = CancellationToken::new(); + let mut peers = Vec::with_capacity(targets.len()); + let mut transferred_bytes = 0_u64; + for target in targets { + let remaining = NETWORK_PROBE_MAX_DURATION.saturating_sub(started.elapsed()); + if remaining.is_zero() { + peers.push(network_speedtest_failure(target.alias, NetworkPeerProbeError::TimedOut, Duration::ZERO)); + continue; + } + let peer_started = std::time::Instant::now(); + match client.probe(&target.alias, traffic_bytes, remaining, &cancel).await { + Ok(measurement) => { + transferred_bytes = transferred_bytes.saturating_add(measurement.transferred_bytes); + peers.push(NetworkSpeedtestEntry { + peer_alias: target.alias, + outcome: "SUCCEEDED", + reason: "COMPLETE", + transferred_bytes: measurement.transferred_bytes, + duration_secs: measurement.duration.as_secs_f64(), + latency_micros: u64::try_from(measurement.latency.as_micros()).ok(), + }); + } + Err(error) => peers.push(network_speedtest_failure(target.alias, error, peer_started.elapsed())), + } + } + let elapsed = started.elapsed().min(NETWORK_PROBE_MAX_DURATION); + let successful = peers.iter().filter(|peer| peer.outcome == "SUCCEEDED").count(); + let throughput = (successful > 0 && !elapsed.is_zero()).then(|| transferred_bytes as f64 / elapsed.as_secs_f64()); + SpeedtestResponse { + kind: "net", + measured: successful > 0, + capability_note: (successful != peers.len()).then(|| "one or more inter-node probes failed".to_string()), + aggregate_read_throughput_bytes_per_sec: None, + aggregate_write_throughput_bytes_per_sec: throughput, + drives: Vec::new(), + peers, + rx_bytes: None, + duration_secs: Some(elapsed.as_secs_f64()), + } +} + +fn unavailable_network_speedtest(reason: &str) -> SpeedtestResponse { SpeedtestResponse { kind: "net", measured: false, - capability_note: Some( - "network speedtest measures inter-node bandwidth; a distributed peer-perf harness is not yet wired. See \ - /v3/site-replication/netperf for the site-to-site variant" - .to_string(), - ), + capability_note: Some(reason.to_owned()), aggregate_read_throughput_bytes_per_sec: None, aggregate_write_throughput_bytes_per_sec: None, drives: Vec::new(), + peers: Vec::new(), rx_bytes: None, duration_secs: None, } } +fn network_speedtest_failure( + peer_alias: String, + error: crate::storage::storage_api::ecstore_rpc::NetworkPeerProbeError, + duration: Duration, +) -> NetworkSpeedtestEntry { + use crate::storage::storage_api::ecstore_rpc::NetworkPeerProbeError; + let reason = match error { + NetworkPeerProbeError::UnknownPeer => "UNKNOWN_PEER", + NetworkPeerProbeError::LimitExceeded => "LIMIT_EXCEEDED", + NetworkPeerProbeError::Cancelled => "CANCELLED", + NetworkPeerProbeError::Unreachable => "UNREACHABLE", + NetworkPeerProbeError::TimedOut => "TIMED_OUT", + NetworkPeerProbeError::ProtocolFailure => "PROTOCOL_FAILURE", + }; + NetworkSpeedtestEntry { + peer_alias, + outcome: "FAILED", + reason, + transferred_bytes: 0, + duration_secs: duration.as_secs_f64(), + latency_micros: None, + } +} + pub struct SpeedtestHandler {} #[async_trait::async_trait] @@ -915,7 +1025,7 @@ impl Operation for SpeedtestHandler { let response = match kind { SpeedtestKind::Drive => run_drive_speedtest().await?, SpeedtestKind::Object => object_speedtest_unsupported(), - SpeedtestKind::Net => net_speedtest_single_node(), + SpeedtestKind::Net => run_network_speedtest().await, SpeedtestKind::Site => SpeedtestResponse { kind: "site", measured: false, @@ -925,6 +1035,7 @@ impl Operation for SpeedtestHandler { aggregate_read_throughput_bytes_per_sec: None, aggregate_write_throughput_bytes_per_sec: None, drives: Vec::new(), + peers: Vec::new(), rx_bytes: None, duration_secs: None, }, @@ -940,6 +1051,23 @@ impl Operation for SpeedtestHandler { /// number (mirrors MinIO's `ClientDevNull`). pub struct SpeedtestClientDevnullHandler {} +/// `GET /v3/speedtest/client/devnull?bytes=N` — bounded generated download. +pub struct SpeedtestClientSourceHandler {} + +fn client_source_bytes(uri: &Uri) -> S3Result { + let bytes = query_value(uri, "bytes") + .and_then(|value| value.parse::().ok()) + .ok_or_else(|| s3_error!(InvalidRequest, "client speedtest requires a positive bytes parameter"))?; + if bytes == 0 || bytes > CLIENT_DEVNULL_SOURCE_MAX_BYTES { + return Err(s3_error!( + EntityTooLarge, + "client speedtest download exceeds the {}-byte limit", + CLIENT_DEVNULL_SOURCE_MAX_BYTES + )); + } + usize::try_from(bytes).map_err(|_| s3_error!(EntityTooLarge, "client speedtest download exceeds platform limits")) +} + fn validate_client_devnull_content_length(headers: &HeaderMap) -> S3Result<()> { let Some(content_length) = headers.get(CONTENT_LENGTH) else { return Ok(()); @@ -1022,6 +1150,7 @@ impl Operation for SpeedtestClientDevnullHandler { 0.0 }), drives: Vec::new(), + peers: Vec::new(), rx_bytes: Some(total), duration_secs: Some(elapsed.as_secs_f64()), }; @@ -1029,6 +1158,17 @@ impl Operation for SpeedtestClientDevnullHandler { } } +#[async_trait::async_trait] +impl Operation for SpeedtestClientSourceHandler { + async fn call(&self, req: S3Request, _params: Params<'_, '_>) -> S3Result> { + authorize(&req, AdminAction::HealthInfoAdminAction).await?; + let bytes = client_source_bytes(&req.uri)?; + let _permit = acquire_client_devnull_permit()?; + + Ok(S3Response::new((StatusCode::OK, Body::from(vec![0_u8; bytes])))) + } +} + // --------------------------------------------------------------------------- // Query helpers // --------------------------------------------------------------------------- @@ -1179,6 +1319,32 @@ mod tests { assert_eq!(total, 4); } + #[test] + fn client_source_requires_a_bounded_positive_size() { + let valid = format!("/rustfs/admin/v3/speedtest/client/devnull?bytes={CLIENT_DEVNULL_SOURCE_MAX_BYTES}") + .parse::() + .expect("valid URI"); + assert_eq!( + client_source_bytes(&valid).expect("size at the limit should succeed"), + usize::try_from(CLIENT_DEVNULL_SOURCE_MAX_BYTES).expect("source limit fits usize") + ); + + for invalid in [ + "/rustfs/admin/v3/speedtest/client/devnull", + "/rustfs/admin/v3/speedtest/client/devnull?bytes=0", + "/rustfs/admin/v3/speedtest/client/devnull?bytes=invalid", + ] { + let uri = invalid.parse::().expect("valid URI"); + assert!(client_source_bytes(&uri).is_err(), "{invalid} must fail"); + } + + let oversized = format!("/rustfs/admin/v3/speedtest/client/devnull?bytes={}", CLIENT_DEVNULL_SOURCE_MAX_BYTES + 1) + .parse::() + .expect("valid URI"); + let err = client_source_bytes(&oversized).expect_err("oversized source request must fail"); + assert_eq!(err.code(), &S3ErrorCode::EntityTooLarge); + } + #[test] fn client_devnull_rejects_invalid_content_length() { let mut headers = HeaderMap::new(); @@ -1214,6 +1380,28 @@ mod tests { let _permit = acquire_client_devnull_permit().expect("released admission slots must be reusable"); } + #[test] + fn network_probe_admission_fails_fast_and_recovers() { + let permit = NETWORK_PROBE_ADMISSION.try_acquire().expect("network probe admission slot"); + assert!(NETWORK_PROBE_ADMISSION.try_acquire().is_err()); + drop(permit); + let _permit = NETWORK_PROBE_ADMISSION.try_acquire().expect("released network probe slot"); + } + + #[test] + fn network_probe_failure_keeps_peer_attribution_and_elapsed_time() { + let failure = network_speedtest_failure( + "peer-2".to_owned(), + crate::storage::storage_api::ecstore_rpc::NetworkPeerProbeError::TimedOut, + Duration::from_millis(25), + ); + assert_eq!(failure.peer_alias, "peer-2"); + assert_eq!(failure.outcome, "FAILED"); + assert_eq!(failure.reason, "TIMED_OUT"); + assert_eq!(failure.transferred_bytes, 0); + assert_eq!(failure.duration_secs, 0.025); + } + #[test] fn speedtest_kind_routing() { assert_eq!(speedtest_kind_from_path("/rustfs/admin/v3/speedtest"), SpeedtestKind::Object); diff --git a/rustfs/src/admin/handlers/gateway_key_inventory.rs b/rustfs/src/admin/handlers/gateway_key_inventory.rs new file mode 100644 index 000000000..5a12218ea --- /dev/null +++ b/rustfs/src/admin/handlers/gateway_key_inventory.rs @@ -0,0 +1,421 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Pre-migration key inventory for the switch to the RustFS S3 gateway (rustfs/gateway#754). +//! +//! The gateway refuses some object keys on every operation that names one, reads and deletes +//! included, so an object stored under such a key today would be unreachable through the gateway +//! after the switch. This read-only endpoint lists every stored key the gateway would refuse, so +//! an operator can copy those objects to a safe key through the current stack before switching. +//! +//! The rules mirror the gateway's single key normalisation (`rustfs/gateway`, +//! `crates/types/src/scalar/naming.rs`: the residual-encoding check, then `floor_check_key`), for a +//! client that percent-encodes the stored key exactly once. A `.` or `..` segment and a `//` run are +//! absent from the report because this store refuses them on every write +//! (`is_valid_object_prefix`), so no stored key can carry one. + +use crate::admin::auth::authorize_admin_request; +use crate::admin::handlers::admin_json_response; +use crate::admin::router::{AdminOperation, Operation, S3Router}; +use crate::admin::runtime_sources::current_object_store_handle; +use crate::admin::storage_api::contract::bucket::{BucketOperations as _, BucketOptions}; +use crate::admin::storage_api::contract::list::ListOperations as _; +use crate::admin::storage_api::s3::{self, Body, S3ErrorCode, S3Request, S3Response, S3Result}; +use crate::error::ApiError; +use crate::server::ADMIN_PREFIX; +use http::StatusCode; +use hyper::Method; +use matchit::Params; +use rustfs_policy::policy::action::{Action, AdminAction}; +use serde::Serialize; +use std::collections::BTreeMap; + +pub const GATEWAY_KEY_INVENTORY_ROUTE_SUFFIX: &str = "/v3/gateway-key-inventory"; + +/// The gateway's key length limit, in UTF-8 bytes. +const GATEWAY_MAX_KEY_BYTES: usize = 1024; +const LIST_PAGE_KEYS: i32 = 1000; +const DEFAULT_MAX_FINDINGS: usize = 1000; +const MAX_FINDINGS_LIMIT: usize = 10_000; + +/// Why the gateway would refuse a stored key. One variant per gateway rule. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum GatewayKeyRefusal { + /// A literal `%2F`, `%5C` or `%2E%2E`, which the gateway refuses after its single decode. + EncodedSeparator, + /// An empty key. + Empty, + /// Longer than 1024 UTF-8 bytes. + TooLong, + /// A NUL byte. + Nul, + /// A C0 control, DEL or C1 control. + ControlCharacter, + /// A leading `//` or `\`, or a drive root such as `C:/` or `c:\`. + AbsoluteOrUnc, + /// A `..` segment delimited by `/` or `\`. + TraversalSegment, +} + +impl GatewayKeyRefusal { + /// Stable report label. + pub const fn slug(self) -> &'static str { + match self { + Self::EncodedSeparator => "encoded_separator", + Self::Empty => "empty", + Self::TooLong => "too_long", + Self::Nul => "nul", + Self::ControlCharacter => "control_character", + Self::AbsoluteOrUnc => "absolute_or_unc", + Self::TraversalSegment => "traversal_segment", + } + } +} + +/// The rule under which the gateway would refuse `key`, first rule first, or `None` when the +/// gateway reaches it. +pub fn gateway_key_refusal(key: &str) -> Option { + let lower = key.to_ascii_lowercase(); + if lower.contains("%2f") || lower.contains("%5c") || lower.contains("%2e%2e") { + return Some(GatewayKeyRefusal::EncodedSeparator); + } + if key.is_empty() { + return Some(GatewayKeyRefusal::Empty); + } + if key.len() > GATEWAY_MAX_KEY_BYTES { + return Some(GatewayKeyRefusal::TooLong); + } + if key.contains('\0') { + return Some(GatewayKeyRefusal::Nul); + } + if key.chars().any(char::is_control) { + return Some(GatewayKeyRefusal::ControlCharacter); + } + if key.starts_with("//") || key.starts_with('\\') || is_drive_rooted(key) { + return Some(GatewayKeyRefusal::AbsoluteOrUnc); + } + if key.split(['/', '\\']).any(|segment| segment == "..") { + return Some(GatewayKeyRefusal::TraversalSegment); + } + None +} + +fn is_drive_rooted(key: &str) -> bool { + let bytes = key.as_bytes(); + bytes.len() >= 3 && bytes[0].is_ascii_alphabetic() && bytes[1] == b':' && (bytes[2] == b'/' || bytes[2] == b'\\') +} + +#[derive(Debug, PartialEq, Eq, Serialize)] +pub struct GatewayKeyFinding { + pub bucket: String, + pub key: String, + pub rule: &'static str, +} + +#[derive(Debug, Serialize)] +pub struct GatewayKeyInventoryReport { + pub buckets_scanned: usize, + /// Distinct keys seen; the versions of one key count once. + pub keys_scanned: u64, + pub refused_keys: u64, + pub by_rule: BTreeMap<&'static str, u64>, + pub findings: Vec, + /// True when more keys are refused than `findings` lists; the counts stay complete. + pub findings_truncated: bool, + #[serde(skip)] + max_findings: usize, + #[serde(skip)] + last: Option<(String, String)>, +} + +impl GatewayKeyInventoryReport { + pub fn new(max_findings: usize) -> Self { + Self { + buckets_scanned: 0, + keys_scanned: 0, + refused_keys: 0, + by_rule: BTreeMap::new(), + findings: Vec::new(), + findings_truncated: false, + max_findings, + last: None, + } + } + + pub fn begin_bucket(&mut self) { + self.buckets_scanned += 1; + self.last = None; + } + + /// Records one listed entry. A version listing returns every version of a key back to back, + /// so an entry repeating the previous key is the same key and is not counted again. + pub fn record(&mut self, bucket: &str, key: &str) { + if self.last.as_ref().is_some_and(|(b, k)| b == bucket && k == key) { + return; + } + self.last = Some((bucket.to_owned(), key.to_owned())); + self.keys_scanned += 1; + let Some(rule) = gateway_key_refusal(key) else { + return; + }; + self.refused_keys += 1; + *self.by_rule.entry(rule.slug()).or_default() += 1; + if self.findings.len() < self.max_findings { + self.findings.push(GatewayKeyFinding { + bucket: bucket.to_owned(), + key: key.to_owned(), + rule: rule.slug(), + }); + } else { + self.findings_truncated = true; + } + } +} + +#[derive(Debug, PartialEq, Eq)] +struct InventoryQuery { + bucket: Option, + max_findings: usize, +} + +fn parse_inventory_query(query: Option<&str>) -> S3Result { + let mut bucket = None; + let mut max_findings = None; + for (key, value) in url::form_urlencoded::parse(query.unwrap_or_default().as_bytes()) { + match key.as_ref() { + "bucket" => { + crate::storage::ecstore_bucket::utils::check_valid_bucket_name_strict(&value) + .map_err(|_| s3::error(S3ErrorCode::InvalidArgument, "invalid bucket name"))?; + bucket = Some(value.into_owned()); + } + "max-findings" => { + max_findings = Some( + value + .parse::() + .map_err(|_| s3::error(S3ErrorCode::InvalidArgument, "max-findings must be a positive integer"))?, + ); + } + other => return Err(s3::error(S3ErrorCode::InvalidArgument, format!("unknown query parameter: {other}"))), + } + } + Ok(InventoryQuery { + bucket, + max_findings: max_findings.unwrap_or(DEFAULT_MAX_FINDINGS).clamp(1, MAX_FINDINGS_LIMIT), + }) +} + +pub fn register_gateway_key_inventory_route(r: &mut S3Router) -> std::io::Result<()> { + r.insert( + Method::GET, + format!("{ADMIN_PREFIX}{GATEWAY_KEY_INVENTORY_ROUTE_SUFFIX}").as_str(), + AdminOperation(&GatewayKeyInventoryHandler {}), + )?; + Ok(()) +} + +pub struct GatewayKeyInventoryHandler {} + +#[async_trait::async_trait] +impl Operation for GatewayKeyInventoryHandler { + async fn call(&self, req: S3Request, _params: Params<'_, '_>) -> S3Result> { + let cred = authorize_admin_request(&req, vec![Action::AdminAction(AdminAction::InspectDataAction)]).await?; + let query = parse_inventory_query(req.uri.query())?; + let store = + current_object_store_handle().ok_or_else(|| s3::error(S3ErrorCode::InternalError, "object store not initialized"))?; + + let buckets = match query.bucket { + Some(bucket) => vec![bucket], + None => store + .list_bucket(&BucketOptions::default()) + .await + .map_err(ApiError::from)? + .into_iter() + .map(|bucket| bucket.name) + .collect(), + }; + + let mut report = GatewayKeyInventoryReport::new(query.max_findings); + for bucket in buckets { + report.begin_bucket(); + let (mut marker, mut version_marker) = (None, None); + loop { + let page = store + .clone() + .list_object_versions(&bucket, "", marker, version_marker, None, LIST_PAGE_KEYS) + .await + .map_err(ApiError::from)?; + for object in &page.objects { + report.record(&bucket, &object.name); + } + if !page.is_truncated || (page.next_marker.is_none() && page.next_version_idmarker.is_none()) { + break; + } + marker = page.next_marker; + version_marker = page.next_version_idmarker; + } + } + + admin_json_response(req.uri.path(), &cred.secret_key, StatusCode::OK, &report) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use http::HeaderMap; + + #[test] + fn keys_the_gateway_reaches_are_not_reported() { + let long = "k".repeat(GATEWAY_MAX_KEY_BYTES); + for key in [ + "photos/2026/a.jpg", + "a/./b", + "a//b", + "/leading", + "dir/", + "100%done", + "%25", + "a\\b", + "C:x", + "...", + "a..b", + "é/ü", + long.as_str(), + ] { + assert_eq!(gateway_key_refusal(key), None, "{key:?}"); + } + } + + #[test] + fn each_gateway_rule_is_reported_by_name() { + let too_long = "k".repeat(GATEWAY_MAX_KEY_BYTES + 1); + let cases = [ + ("a%2Fb", GatewayKeyRefusal::EncodedSeparator), + ("a%5cb", GatewayKeyRefusal::EncodedSeparator), + ("%2E%2e/x", GatewayKeyRefusal::EncodedSeparator), + ("", GatewayKeyRefusal::Empty), + (too_long.as_str(), GatewayKeyRefusal::TooLong), + ("a\0b", GatewayKeyRefusal::Nul), + ("a\tb", GatewayKeyRefusal::ControlCharacter), + ("a\u{7f}", GatewayKeyRefusal::ControlCharacter), + ("a\u{85}b", GatewayKeyRefusal::ControlCharacter), + ("//server/share", GatewayKeyRefusal::AbsoluteOrUnc), + ("\\lead", GatewayKeyRefusal::AbsoluteOrUnc), + ("C:/x", GatewayKeyRefusal::AbsoluteOrUnc), + ("c:\\x", GatewayKeyRefusal::AbsoluteOrUnc), + ("..", GatewayKeyRefusal::TraversalSegment), + ("a/../b", GatewayKeyRefusal::TraversalSegment), + ("a\\..\\b", GatewayKeyRefusal::TraversalSegment), + ("a/..", GatewayKeyRefusal::TraversalSegment), + ]; + for (key, rule) in cases { + assert_eq!(gateway_key_refusal(key), Some(rule), "{key:?}"); + } + } + + #[test] + fn versions_of_one_key_count_once_and_counts_outlive_the_findings_cap() { + let mut report = GatewayKeyInventoryReport::new(1); + report.begin_bucket(); + for key in ["ok", "a\tb", "a\tb", "C:/x", "fine"] { + report.record("b1", key); + } + report.begin_bucket(); + report.record("b2", "a\tb"); + + assert_eq!(report.buckets_scanned, 2); + assert_eq!(report.keys_scanned, 5); + assert_eq!(report.refused_keys, 3); + assert_eq!(report.by_rule.get("control_character"), Some(&2)); + assert_eq!(report.by_rule.get("absolute_or_unc"), Some(&1)); + assert_eq!( + report.findings, + [GatewayKeyFinding { + bucket: "b1".to_owned(), + key: "a\tb".to_owned(), + rule: "control_character", + }] + ); + assert!(report.findings_truncated); + + let encoded = serde_json::to_value(&report).expect("report serializes"); + assert_eq!(encoded["by_rule"]["control_character"], 2); + assert_eq!(encoded["findings"][0]["key"], "a\tb"); + assert!(encoded.get("max_findings").is_none()); + } + + #[test] + fn query_defaults_clamps_and_refuses_unknown_parameters() { + assert_eq!( + parse_inventory_query(None).expect("empty query"), + InventoryQuery { + bucket: None, + max_findings: DEFAULT_MAX_FINDINGS, + } + ); + let parsed = parse_inventory_query(Some("bucket=photos&max-findings=0")).expect("valid query"); + assert_eq!(parsed.bucket.as_deref(), Some("photos")); + assert_eq!(parsed.max_findings, 1, "zero must clamp up, not mean unlimited"); + assert_eq!( + parse_inventory_query(Some("max-findings=99999999")) + .expect("valid") + .max_findings, + MAX_FINDINGS_LIMIT + ); + assert!(parse_inventory_query(Some("max-findings=-1")).is_err()); + assert!(parse_inventory_query(Some("bucket=Bad_Name")).is_err()); + assert!(parse_inventory_query(Some("prefix=x")).is_err()); + } + + #[test] + fn query_errors_keep_their_s3_code_status_and_message() { + let cases = [ + ("bucket=Bad_Name", "invalid bucket name"), + ("max-findings=-1", "max-findings must be a positive integer"), + ("prefix=x", "unknown query parameter: prefix"), + ]; + for (query, message) in cases { + let err = parse_inventory_query(Some(query)).expect_err(query); + assert_eq!(err.code(), &S3ErrorCode::InvalidArgument, "{query}"); + assert_eq!(err.status_code(), Some(StatusCode::BAD_REQUEST), "{query}"); + assert_eq!(err.message(), Some(message), "{query}"); + } + + let err = s3::error(S3ErrorCode::InternalError, "object store not initialized"); + assert_eq!(err.code(), &S3ErrorCode::InternalError); + assert_eq!(err.status_code(), Some(StatusCode::INTERNAL_SERVER_ERROR)); + assert_eq!(err.message(), Some("object store not initialized")); + } + + #[tokio::test] + async fn a_request_without_credentials_is_refused_before_any_listing() { + let req = S3Request { + input: Body::from(String::new()), + method: Method::GET, + uri: http::Uri::from_static("/rustfs/admin/v3/gateway-key-inventory"), + headers: HeaderMap::new(), + extensions: http::Extensions::new(), + credentials: None, + region: None, + service: None, + trailing_headers: None, + }; + + let err = GatewayKeyInventoryHandler {} + .call(req, Params::new()) + .await + .expect_err("a request without credentials must be refused"); + assert_eq!(err.code(), &S3ErrorCode::InvalidRequest); + } +} diff --git a/rustfs/src/admin/handlers/mod.rs b/rustfs/src/admin/handlers/mod.rs index 288f1849f..45b2f2d67 100644 --- a/rustfs/src/admin/handlers/mod.rs +++ b/rustfs/src/admin/handlers/mod.rs @@ -25,6 +25,7 @@ pub mod diagnostics; pub mod durability; pub mod event; pub mod extensions; +pub mod gateway_key_inventory; pub mod group; pub mod heal; pub mod health; diff --git a/rustfs/src/admin/handlers/oidc.rs b/rustfs/src/admin/handlers/oidc.rs index 9ed5afdc4..b148af73e 100644 --- a/rustfs/src/admin/handlers/oidc.rs +++ b/rustfs/src/admin/handlers/oidc.rs @@ -24,7 +24,7 @@ use crate::admin::storage_api::config::{ read_admin_config_without_migrate, read_admin_server_config_snapshot, save_admin_server_config_snapshot, }; use crate::admin::utils::json_response; -use crate::server::{ADMIN_PREFIX, CONSOLE_PREFIX, MINIO_ADMIN_PREFIX}; +use crate::server::{ADMIN_PREFIX, MINIO_ADMIN_PREFIX, console_prefix}; use http::StatusCode; use hyper::Method; use matchit::Params; @@ -824,7 +824,8 @@ fn build_console_redirect( let fragment = build_console_callback_fragment(access_key, secret_key, session_token, expiration, redirect_after, logout_token); - let callback_path = format!("{CONSOLE_PREFIX}{CONSOLE_OIDC_CALLBACK_SUFFIX}"); + let console_prefix = console_prefix(); + let callback_path = format!("{console_prefix}{CONSOLE_OIDC_CALLBACK_SUFFIX}"); if let Some(base_url) = browser_redirect_url(&callback_path)? { return Ok(format!("{base_url}#{fragment}")); } @@ -836,7 +837,8 @@ fn build_console_redirect( } fn build_console_login_redirect(req: &S3Request) -> S3Result { - let login_path = format!("{CONSOLE_PREFIX}{CONSOLE_LOGIN_SUFFIX}"); + let console_prefix = console_prefix(); + let login_path = format!("{console_prefix}{CONSOLE_LOGIN_SUFFIX}"); if let Some(url) = browser_redirect_url(&login_path)? { return Ok(url); } @@ -1672,6 +1674,26 @@ mod tests { assert_eq!(callback, "https://internal:9000/rustfs/admin/v3/oidc/callback/default"); } + #[test] + fn console_prefix_process_case_oidc() { + if std::env::var_os("RUSTFS_TEST_CONSOLE_PREFIX_PROCESS").is_none() { + return; + } + crate::server::init_console_prefix().expect("initialize console prefix"); + let prefix = console_prefix(); + let req = build_oidc_request("http://internal/rustfs/admin/v3/oidc/callback/default", Some("internal:9000"), None); + assert_eq!( + build_console_login_redirect(&req).expect("login URL"), + format!("https://console.example.com{prefix}/auth/login") + ); + let redirect = build_console_redirect(&req, "access", "secret", "token", None, None, None).expect("console callback URL"); + assert!(redirect.starts_with(&format!("https://console.example.com{prefix}/auth/oidc-callback/#"))); + assert_eq!( + derive_callback_uri(&req, "default").expect("admin callback URL"), + "https://console.example.com/rustfs/admin/v3/oidc/callback/default" + ); + } + #[test] fn test_build_console_redirect_uses_browser_redirect_url() { let req = build_oidc_request("http://internal/rustfs/admin/v3/oidc/callback/default", Some("internal:9000"), None); @@ -1681,7 +1703,7 @@ mod tests { .expect("console redirect should use browser redirect URL") }); - assert!(redirect.starts_with("https://console.example.com/rustfs/console/auth/oidc-callback/#")); + assert!(redirect.starts_with(&format!("https://console.example.com{}/auth/oidc-callback/#", console_prefix()))); assert!(redirect.contains("redirect=%2Fbuckets")); assert!(redirect.contains("logoutToken=logout-token")); } @@ -1694,7 +1716,7 @@ mod tests { build_console_login_redirect(&req).expect("login redirect should use browser redirect URL") }); - assert_eq!(redirect, "https://console.example.com/rustfs/console/auth/login"); + assert_eq!(redirect, format!("https://console.example.com{}/auth/login", console_prefix())); } #[test] diff --git a/rustfs/src/admin/handlers/system.rs b/rustfs/src/admin/handlers/system.rs index 165baf70c..8b8ea09bd 100644 --- a/rustfs/src/admin/handlers/system.rs +++ b/rustfs/src/admin/handlers/system.rs @@ -904,10 +904,15 @@ impl DiagnosticProbeCapabilities { "object PUT/GET benchmark harness is not implemented", Some("/rustfs/admin/v3/speedtest/object"), ), - inter_node_netperf: unsupported( - "inter-node traffic benchmark harness is not implemented", - Some("/rustfs/admin/v3/speedtest/net"), - ), + inter_node_netperf: DiagnosticProbeCapability { + status: CapabilityStatus::supported() + .with_reason("actively measures bounded authenticated traffic to every remote cluster peer"), + mode: "active_inter_node_probe", + route: Some("/rustfs/admin/v3/speedtest/net"), + max_bytes: Some(super::diagnostics::NETWORK_PROBE_MAX_BYTES), + max_duration_secs: Some(super::diagnostics::NETWORK_PROBE_MAX_DURATION.as_secs()), + max_concurrency: Some(super::diagnostics::NETWORK_PROBE_MAX_CONCURRENCY), + }, site_speedtest: unsupported( "site traffic benchmark harness is not implemented", Some("/rustfs/admin/v3/speedtest/site"), @@ -1421,10 +1426,23 @@ mod tests { response.diagnostic_probes.client_devnull.max_concurrency, Some(super::super::diagnostics::CLIENT_DEVNULL_MAX_CONCURRENCY) ); + assert_eq!(response.diagnostic_probes.inter_node_netperf.status.state, CapabilityState::Supported); + assert_eq!(response.diagnostic_probes.inter_node_netperf.mode, "active_inter_node_probe"); + assert_eq!( + response.diagnostic_probes.inter_node_netperf.max_bytes, + Some(super::super::diagnostics::NETWORK_PROBE_MAX_BYTES) + ); + assert_eq!( + response.diagnostic_probes.inter_node_netperf.max_duration_secs, + Some(super::super::diagnostics::NETWORK_PROBE_MAX_DURATION.as_secs()) + ); + assert_eq!( + response.diagnostic_probes.inter_node_netperf.max_concurrency, + Some(super::super::diagnostics::NETWORK_PROBE_MAX_CONCURRENCY) + ); for probe in [ &response.diagnostic_probes.inspect_archive, &response.diagnostic_probes.object_speedtest, - &response.diagnostic_probes.inter_node_netperf, &response.diagnostic_probes.site_speedtest, &response.diagnostic_probes.site_replication_netperf, ] { diff --git a/rustfs/src/admin/mod.rs b/rustfs/src/admin/mod.rs index e287525a2..b6d2c424e 100644 --- a/rustfs/src/admin/mod.rs +++ b/rustfs/src/admin/mod.rs @@ -35,8 +35,8 @@ mod route_registration_test; use handlers::{ account, audit, batch_job, bucket_meta, cluster_snapshot, config_admin, diagnostics, durability as durability_handler, - extensions, heal, health, idp_compat, ilm_transition, inspect_archive, kms, mfa, module_switch, object_data_cache, - object_zip_download, oidc, on_demand_migration, plugins_catalog, plugins_instances, pools, profile_admin, + extensions, gateway_key_inventory, heal, health, idp_compat, ilm_transition, inspect_archive, kms, mfa, module_switch, + object_data_cache, object_zip_download, oidc, on_demand_migration, plugins_catalog, plugins_instances, pools, profile_admin, quota as quota_handler, rebalance, replication as replication_handler, scanner, site_replication, sts, system, table_catalog, tier, tls_debug, usage_prefix, user, }; @@ -98,6 +98,7 @@ fn register_admin_routes(r: &mut S3Router) -> std::io::Result<() profile_admin::register_profiling_route(r)?; diagnostics::register_diagnostics_route(r)?; inspect_archive::register_inspect_archive_route(r)?; + gateway_key_inventory::register_gateway_key_inventory_route(r)?; tls_debug::register_tls_debug_route(r)?; kms::register_kms_route(r)?; oidc::register_oidc_route(r)?; diff --git a/rustfs/src/admin/route_policy.rs b/rustfs/src/admin/route_policy.rs index 18b70d757..9593c80c6 100644 --- a/rustfs/src/admin/route_policy.rs +++ b/rustfs/src/admin/route_policy.rs @@ -812,7 +812,19 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[ HEALTH_INFO, RouteRiskLevel::High, ), + admin( + HttpMethod::Get, + "/rustfs/admin/v3/speedtest/client/devnull", + HEALTH_INFO, + RouteRiskLevel::High, + ), admin(HttpMethod::Post, "/rustfs/admin/v4/inspect/archive", INSPECT_DATA, RouteRiskLevel::High), + admin( + HttpMethod::Get, + "/rustfs/admin/v3/gateway-key-inventory", + INSPECT_DATA, + RouteRiskLevel::High, + ), // MinIO-compatible profiling / trace endpoints. admin(HttpMethod::Post, "/rustfs/admin/v3/profiling/start", PROFILING, RouteRiskLevel::High), admin( @@ -2340,6 +2352,12 @@ mod tests { assert_action(HttpMethod::Post, "/rustfs/admin/v4/inspect/archive", INSPECT_DATA); } + #[test] + fn route_policy_requires_inspect_action_for_gateway_key_inventory() { + assert_action(HttpMethod::Get, "/rustfs/admin/v3/gateway-key-inventory", INSPECT_DATA); + assert_not_action(HttpMethod::Get, "/rustfs/admin/v3/gateway-key-inventory", SERVER_INFO); + } + #[test] fn route_policy_requires_operation_for_site_replication_diagnostics() { for path in [ diff --git a/rustfs/src/admin/route_registration_test.rs b/rustfs/src/admin/route_registration_test.rs index a0a178a9a..d2edb819f 100644 --- a/rustfs/src/admin/route_registration_test.rs +++ b/rustfs/src/admin/route_registration_test.rs @@ -181,6 +181,7 @@ fn expected_admin_route_matrix() -> Vec { admin_route(Method::GET, "/v3/inspect-data"), admin_route(Method::POST, "/v3/inspect-data"), admin_route(Method::POST, "/v4/inspect/archive"), + admin_route(Method::GET, "/v3/gateway-key-inventory"), admin_route(Method::GET, "/v3/storageinfo"), admin_route(Method::GET, "/v3/datausageinfo"), admin_route_sample(Method::GET, "/v3/usage/{bucket}", "/v3/usage/test-bucket"), @@ -375,6 +376,7 @@ fn expected_admin_route_matrix() -> Vec { admin_route(Method::POST, "/v3/speedtest/net"), admin_route(Method::POST, "/v3/speedtest/site"), admin_route(Method::POST, "/v3/speedtest/client/devnull"), + admin_route(Method::GET, "/v3/speedtest/client/devnull"), admin_route(Method::GET, "/debug/tls/status"), admin_route(Method::POST, "/v3/kms/create-key"), admin_route(Method::POST, "/v3/kms/key/create"), diff --git a/rustfs/src/app/mod.rs b/rustfs/src/app/mod.rs index f5b0f0d60..b75ae6019 100644 --- a/rustfs/src/app/mod.rs +++ b/rustfs/src/app/mod.rs @@ -29,6 +29,7 @@ pub(crate) mod runtime_sources; mod select_object; pub(crate) mod storage_api; pub(crate) mod table_list_isolation; +pub(crate) mod trailer_adapter; #[cfg(test)] mod capacity_dirty_scope_test; diff --git a/rustfs/src/app/multipart_usecase.rs b/rustfs/src/app/multipart_usecase.rs index 81e78e99e..d2809d41f 100644 --- a/rustfs/src/app/multipart_usecase.rs +++ b/rustfs/src/app/multipart_usecase.rs @@ -67,6 +67,7 @@ use super::storage_api::multipart_usecase::sse::{ use super::storage_api::multipart_usecase::{ StorageObjectInfo as ObjectInfo, StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader, }; +use super::trailer_adapter::trailer_source; use crate::app::object::{ ConcurrencyManager, ForegroundWriteAdmission, get_concurrency_manager, guard_put_object_body_read_timeout, put_object_body_read_timeout, reject_oversize_single_upload, @@ -1266,7 +1267,7 @@ impl DefaultMultipartUsecase { let mut hrd = HashReader::from_stream(body, size, actual_size, md5hex.take(), sha256hex.take(), false) .map_err(ApiError::from)?; - if let Err(err) = hrd.add_checksum_from_s3s(&req.headers, req.trailing_headers.clone(), false) { + if let Err(err) = hrd.add_checksum(&req.headers, trailer_source(req.trailing_headers.clone()), false) { return Err(ApiError::from(err).into()); } @@ -1277,7 +1278,7 @@ impl DefaultMultipartUsecase { HashReader::from_stream(body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)? }; - if let Err(err) = reader.add_checksum_from_s3s(&req.headers, req.trailing_headers.clone(), size < 0) { + if let Err(err) = reader.add_checksum(&req.headers, trailer_source(req.trailing_headers.clone()), size < 0) { return Err(ApiError::from(err).into()); } opts.want_checksum = reader.checksum(); @@ -2017,7 +2018,7 @@ mod tests { Some(ChecksumAlgorithm::SHA256) ); let mut reader = HashReader::from_stream(Cursor::new(payload), 3, 3, None, None, false).unwrap(); - reader.add_checksum_from_s3s(&req.headers, None, false).unwrap(); + reader.add_checksum(&req.headers, None, false).unwrap(); assert_eq!(reader.content_crc_type(), Some(rustfs_rio::ChecksumType::SHA256)); let mut bytes = Vec::new(); let result = reader.read_to_end(&mut bytes).await; @@ -2038,7 +2039,7 @@ mod tests { normalize_presigned_part_checksums(&mut req).unwrap(); assert_eq!(req.input.checksum_crc32.as_deref(), Some("y/Q5Jg==")); let mut reader = HashReader::from_stream(Cursor::new(b"123456789"), 9, 9, None, None, false).unwrap(); - reader.add_checksum_from_s3s(&req.headers, None, false).unwrap(); + reader.add_checksum(&req.headers, None, false).unwrap(); let mut bytes = Vec::new(); reader.read_to_end(&mut bytes).await.unwrap(); assert_eq!(bytes, b"123456789"); diff --git a/rustfs/src/app/object/extract.rs b/rustfs/src/app/object/extract.rs index 1e59e2675..204a0c1f3 100644 --- a/rustfs/src/app/object/extract.rs +++ b/rustfs/src/app/object/extract.rs @@ -19,6 +19,7 @@ use crate::app::storage_api::object_usecase::bucket::replication::ReplicateDecis #[cfg(test)] use crate::app::storage_api::object_usecase::concurrency::SNOWBALL_MEMBER_COMMIT_LIMIT; use crate::app::storage_api::object_usecase::concurrency::SNOWBALL_STAGING_BYTES_LIMIT; +use crate::app::trailer_adapter::trailer_source; use futures::stream::FuturesUnordered; use std::collections::HashSet; @@ -2140,7 +2141,7 @@ impl DefaultObjectUsecase { let mut archive_reader = HashReader::from_stream(body, size, actual_size, md5hex, sha256hex, false).map_err(ApiError::from)?; - if let Err(err) = archive_reader.add_checksum_from_s3s(&req.headers, req.trailing_headers.clone(), false) { + if let Err(err) = archive_reader.add_checksum(&req.headers, trailer_source(req.trailing_headers.clone()), false) { return Err(ApiError::from(err).into()); } diff --git a/rustfs/src/app/object/mod.rs b/rustfs/src/app/object/mod.rs index b10143743..a0b4f1aea 100644 --- a/rustfs/src/app/object/mod.rs +++ b/rustfs/src/app/object/mod.rs @@ -171,10 +171,10 @@ use s3s::dto::{ DeleteObjectsOutput, DeletedObject, ETag, GetObjectAttributesInput, GetObjectAttributesOutput, GetObjectAttributesParts, GetObjectInput, GetObjectOutput, HeadObjectInput, HeadObjectOutput, MetadataDirective, ObjectAttributes, ObjectLockLegalHold, ObjectLockLegalHoldStatus, ObjectLockMode, ObjectLockRetention, ObjectLockRetentionMode, ObjectPart, PutObjectInput, - PutObjectOutput, Range, RequestCharged, RestoreObjectInput, RestoreObjectOutput, RestoreRequestType, RestoreStatus, - SSECustomerAlgorithm, SSECustomerKeyMD5, SSEKMSKeyId, SelectObjectContentInput, SelectObjectContentOutput, - ServerSideEncryption, ServerSideEncryptionConfiguration, StorageClass, StreamingBlob, TaggingDirective, TaggingHeader, - Timestamp, TimestampFormat, WebsiteRedirectLocation, + PutObjectOutput, Range, RequestCharged, RestoreObjectInput, RestoreObjectOutput, RestoreRequestType, SSECustomerAlgorithm, + SSECustomerKeyMD5, SSEKMSKeyId, SelectObjectContentInput, SelectObjectContentOutput, ServerSideEncryption, + ServerSideEncryptionConfiguration, StorageClass, StreamingBlob, TaggingDirective, TaggingHeader, Timestamp, TimestampFormat, + WebsiteRedirectLocation, }; use s3s::header::X_AMZ_RESTORE; use s3s::stream::{ByteStream, DynByteStream, RemainingLength}; diff --git a/rustfs/src/app/object/put.rs b/rustfs/src/app/object/put.rs index c279862fb..5584678cc 100644 --- a/rustfs/src/app/object/put.rs +++ b/rustfs/src/app/object/put.rs @@ -16,6 +16,7 @@ use super::*; +use crate::app::trailer_adapter::trailer_source; use crate::auth::{RUSTFS_MAX_CONTENT_LENGTH_QUERY, VerifiedPresignedRequest, parse_presigned_put_max_content_length}; use crate::error::UploadLimitExceeded; static PUT_FAILURE_LOGS: rustfs_utils::LogThrottle = rustfs_utils::LogThrottle::new(5_000); @@ -1749,7 +1750,7 @@ impl DefaultObjectUsecase { let mut hrd = HashReader::from_stream(body, size, size, md5hex.take(), sha256hex.take(), false).map_err(ApiError::from)?; - if let Err(err) = hrd.add_checksum_from_s3s(headers, trailing_headers.clone(), false) { + if let Err(err) = hrd.add_checksum(headers, trailer_source(trailing_headers.clone()), false) { return Err(ApiError::from(err).into()); } @@ -1799,7 +1800,7 @@ impl DefaultObjectUsecase { }; if size >= 0 { - if let Err(err) = reader.add_checksum_from_s3s(headers, trailing_headers.clone(), false) { + if let Err(err) = reader.add_checksum(headers, trailer_source(trailing_headers.clone()), false) { return Err(ApiError::from(err).into()); } diff --git a/rustfs/src/app/object/request_body/tests/protocol.rs b/rustfs/src/app/object/request_body/tests/protocol.rs index f3d9f0552..773411521 100644 --- a/rustfs/src/app/object/request_body/tests/protocol.rs +++ b/rustfs/src/app/object/request_body/tests/protocol.rs @@ -17,13 +17,16 @@ use crate::app::storage_api::s3::{ Body as S3Body, S3, S3Config, S3Error, S3Request, S3Response, S3Result, S3Service, S3ServiceBuilder, SimpleAuth, StaticConfigProvider, UploadPartInput, UploadPartOutput, }; +use crate::app::trailer_adapter::trailer_source; use http_body_util::BodyExt; +use rustfs_rio::{SharedTrailerSource, TrailerValue}; use std::sync::atomic::{AtomicUsize, Ordering}; #[derive(Clone, Default)] struct Consumer { received: Arc, committed: Arc>>>, + trailer: Arc>>, } #[async_trait::async_trait] @@ -41,8 +44,10 @@ impl S3 for Consumer { let mut reader = rustfs_rio::HashReader::from_stream(DemandReader::new(inner, control), expected, expected, None, None, false) .expect("logical body reader"); + let trailer = trailer_source(req.trailing_headers); + *self.trailer.lock() = trailer.clone(); reader - .add_checksum_from_s3s(&req.headers, req.trailing_headers, false) + .add_checksum(&req.headers, trailer, false) .expect("request checksum context"); let mut output = Vec::new(); let mut buffer = [0; 8192]; @@ -278,3 +283,33 @@ async fn unsigned_trailer_with_wrong_checksum_cannot_commit() { assert!(String::from_utf8_lossy(&xml).contains("BadDigest")); assert!(consumer.committed.lock().is_none()); } + +#[tokio::test] +async fn trailer_adapter_is_pending_until_the_decoded_body_ends() { + let payload = b"123456789"; + let SignedRequest { + request, + sender, + prefix, + suffix, + } = signed_request(payload, true); + let consumer = Consumer::default(); + let service = service(consumer.clone()); + sender.send(Ok(Frame::data(prefix))).expect("prefix"); + sender.send(Ok(Frame::data(Bytes::from_static(payload)))).expect("payload"); + sender.send(Ok(Frame::data(suffix.slice(..2)))).expect("chunk terminator"); + let mut call = Box::pin(service.call(request)); + assert!(poll!(call.as_mut()).is_pending()); + assert_eq!(consumer.received.load(Ordering::Relaxed), payload.len()); + let trailer = consumer.trailer.lock().clone().expect("declared trailer is adapted"); + assert_eq!(trailer.lookup("x-amz-checksum-crc32"), TrailerValue::Pending); + assert!(consumer.committed.lock().is_none()); + + sender.send(Ok(Frame::data(suffix.slice(2..)))).expect("trailer section"); + drop(sender); + let response = call.await.expect("S3 response"); + assert_eq!(response.status(), http::StatusCode::OK); + assert_eq!(trailer.lookup("x-amz-checksum-crc32"), TrailerValue::Present("y/Q5Jg==".to_owned())); + assert_eq!(trailer.lookup("x-amz-checksum-sha256"), TrailerValue::Missing); + assert_eq!(*consumer.committed.lock(), Some(payload.to_vec())); +} diff --git a/rustfs/src/app/object/restore.rs b/rustfs/src/app/object/restore.rs index f941e7e3d..59445ec98 100644 --- a/rustfs/src/app/object/restore.rs +++ b/rustfs/src/app/object/restore.rs @@ -15,6 +15,7 @@ //! RestoreObject path. use super::*; +use rustfs_filemeta::RestoreStatus; // RUSTFS_COMPAT_TODO(backlog-1337): legacy restores lack a liveness marker. Remove after the minimum supported release writes v1 on every restore. const LEGACY_RESTORE_ORPHAN_GRACE: time::Duration = time::Duration::hours(24); @@ -359,7 +360,7 @@ impl DefaultObjectUsecase { X_AMZ_RESTORE.as_str().to_string(), RestoreStatus { is_restore_in_progress: Some(false), - restore_expiry_date: Some(Timestamp::from(restore_expiry)), + restore_expiry_date: Some(restore_expiry), } .to_string(), ); @@ -368,7 +369,7 @@ impl DefaultObjectUsecase { X_AMZ_RESTORE.as_str().to_string(), RestoreStatus { is_restore_in_progress: Some(true), - restore_expiry_date: Some(Timestamp::from(OffsetDateTime::now_utc())), + restore_expiry_date: Some(OffsetDateTime::now_utc()), } .to_string(), ); diff --git a/rustfs/src/app/storage_api.rs b/rustfs/src/app/storage_api.rs index 891f2738f..55cc875d0 100644 --- a/rustfs/src/app/storage_api.rs +++ b/rustfs/src/app/storage_api.rs @@ -46,7 +46,7 @@ pub(crate) mod s3 { pub(crate) use s3s::xml::{Serialize as XmlSerialize, Serializer as XmlSerializer}; #[cfg(test)] pub(crate) use s3s::{Body, S3, S3Response}; - pub(crate) use s3s::{S3Error, S3ErrorCode, S3Request, S3Result}; + pub(crate) use s3s::{S3Error, S3ErrorCode, S3Request, S3Result, TrailingHeaders}; } pub(crate) mod admin { diff --git a/rustfs/src/app/trailer_adapter.rs b/rustfs/src/app/trailer_adapter.rs new file mode 100644 index 000000000..3bcafc0e4 --- /dev/null +++ b/rustfs/src/app/trailer_adapter.rs @@ -0,0 +1,44 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! The single bridge from the HTTP framework's trailer handle to the +//! framework-neutral `rustfs_rio::TrailerSource`. + +use crate::app::storage_api::s3::TrailingHeaders; +use rustfs_rio::{SharedTrailerSource, TrailerSource, TrailerValue}; +use std::sync::Arc; + +// RUSTFS_COMPAT_TODO(s3gate-trailer-adapter): s3s still decodes aws-chunked bodies and publishes their trailers through its own handle, so the object write path adapts that handle to rio's TrailerSource here. Remove after the gateway stack replaces s3s as the request body decoder. +#[derive(Clone, Debug)] +struct S3sTrailerSource(TrailingHeaders); + +impl TrailerSource for S3sTrailerSource { + fn lookup(&self, name: &str) -> TrailerValue { + // s3s fills the handle after the final chunk has been decoded and + // before the decoded body ends, so an unfilled handle is `Pending`. + match self + .0 + .read(|headers| headers.get(name).and_then(|value| value.to_str().ok()).map(str::to_owned)) + { + None => TrailerValue::Pending, + Some(None) => TrailerValue::Missing, + Some(Some(value)) => TrailerValue::Present(value), + } + } +} + +/// Adapts the request's trailer handle, if the request declared one. +pub(crate) fn trailer_source(trailing_headers: Option) -> Option { + trailing_headers.map(|headers| Arc::new(S3sTrailerSource(headers)) as SharedTrailerSource) +} diff --git a/rustfs/src/config/cli.rs b/rustfs/src/config/cli.rs index eec9f64b0..2a6f4d3d3 100644 --- a/rustfs/src/config/cli.rs +++ b/rustfs/src/config/cli.rs @@ -110,7 +110,7 @@ pub enum Commands { /// Offline, read-only inspection of on-disk data (no server required) Inspect(InspectOpts), /// Configure outbound RustFS Connect integration - Connect(ConnectOpts), + Connect(Box), } /// RustFS Connect subcommand options @@ -125,6 +125,992 @@ pub struct ConnectOpts { pub enum ConnectCommands { /// Exchange a protected one-time token for a durable device credential (Unix only) Register(ConnectRegisterOpts), + /// Import, verify, or inspect a signed Connect service license + License(ConnectLicenseOpts), + /// Deliver a reviewed signed artifact through the customer relay (Unix only) + Relay(Box), + /// Upload an explicitly selected support report with the registered device identity + Report(ConnectReportOpts), + /// Read the persisted deployment inventory and collect an approved environment summary + Inventory(ConnectInventoryOpts), + /// Run an explicitly approved, bounded local performance measurement + Performance(ConnectPerformanceOpts), + /// Capture a consent-bound local profile and write a signed export + Profile(ConnectProfileOpts), + /// Capture allow-listed local log events and write a signed export + Logs(ConnectLogsOpts), + /// Record, forward, or replay consent-bound telemetry + Telemetry(ConnectTelemetryOpts), + /// Capture a consent-bound local top snapshot + Top(ConnectTopOpts), + /// Inspect one local object and write a signed integrity summary + Inspect(ConnectInspectOpts), +} + +#[derive(Args, Clone)] +pub struct ConnectInspectOpts { + #[command(subcommand)] + pub command: ConnectInspectCommands, +} + +#[derive(Subcommand, Clone)] +pub enum ConnectInspectCommands { + /// Inspect one object's local erasure metadata and shards + Object(ConnectInspectObjectOpts), +} + +#[derive(Args, Clone)] +pub struct ConnectInspectObjectOpts { + /// Directory containing an enrolled Connect device identity + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + /// New local archive path; an existing file is never replaced + #[arg(long)] + pub output: PathBuf, + /// Organization resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub organization: String, + /// Cluster resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub cluster: String, + /// Cluster-device resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub device: String, + /// UUIDv7 diagnostic run identifier issued by Connect + #[arg(long = "run-uid", value_parser = NonEmptyStringValueParser::new())] + pub run_uid: String, + /// UUIDv7 artifact identifier issued by Connect + #[arg(long = "artifact-uid", value_parser = NonEmptyStringValueParser::new())] + pub artifact_uid: String, + /// UUIDv7 consent identifier issued by Connect + #[arg(long = "consent-uid", value_parser = NonEmptyStringValueParser::new())] + pub consent_uid: String, + /// Consent policy revision bound to this inspection + #[arg(long = "policy-revision")] + pub policy_revision: u64, + /// Consent expiry as UTC Unix seconds + #[arg(long = "consent-expires-at")] + pub consent_expires_at_unix: i64, + /// Artifact expiry as UTC Unix seconds + #[arg(long = "expires-at")] + pub expires_at_unix: i64, + /// Drive root containing the object's local erasure state; repeat for every local drive + #[arg(long = "path", required = true)] + pub paths: Vec, + /// Bucket containing the object + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub bucket: String, + /// Object key to inspect + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub object: String, + /// Optional exact object version UUID + #[arg(long = "version-id", value_parser = NonEmptyStringValueParser::new())] + pub version_id: Option, + /// Negotiated producer schema version + #[arg(long = "schema-version", default_value_t = 1)] + pub schema_version: u16, + /// Negotiated producer capability + #[arg(long, default_value = "inspect.object@1", value_parser = NonEmptyStringValueParser::new())] + pub capability: String, + /// Maximum wall-clock duration in milliseconds + #[arg(long = "duration-millis", default_value_t = 30_000)] + pub duration_millis: u64, + /// Maximum bytes read from local metadata and shards + #[arg(long = "max-read-bytes", default_value_t = 268_435_456)] + pub max_read_bytes: u64, + /// Maximum working memory in bytes + #[arg(long = "max-memory-bytes", default_value_t = 67_108_864)] + pub max_memory_bytes: u64, + /// Confirm this explicit local L3 diagnostic operation + #[arg(long = "acknowledge-l3", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_l3: bool, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)] +pub enum ConnectRelayMaterialKind { + OfflineEnrollmentResponse, + DiagnosticBundleManifest, +} + +#[derive(Args, Clone)] +pub struct ConnectRelayOpts { + /// HTTPS Connect control API base ending in /api/ + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub endpoint: String, + /// PEM root CA file used only for this Connect endpoint + #[arg(long = "ca-file")] + pub ca_file: PathBuf, + /// Owner-only file containing the browser Cookie header value + #[arg(long = "session-cookie-file")] + pub session_cookie_file: PathBuf, + /// Owner-only file containing the browser X-XSRF-TOKEN header value + #[arg(long = "csrf-token-file")] + pub csrf_token_file: PathBuf, + /// Organization UUIDv7 used by the approved Connect tenant + #[arg(long = "organization-uid", value_parser = NonEmptyStringValueParser::new())] + pub organization_uid: String, + /// Approval resource UID returned after customer review in Connect + #[arg(long = "approval-reference", value_parser = NonEmptyStringValueParser::new())] + pub approval_reference: String, + /// UUIDv7 identifying this exact relay attempt + #[arg(long = "transfer-uid", value_parser = NonEmptyStringValueParser::new())] + pub transfer_uid: String, + /// Allow-listed signed artifact type accepted by the Connect receiver + #[arg(long = "material-kind", value_enum)] + pub material_kind: ConnectRelayMaterialKind, + /// Owner-only signed artifact wrapper to transfer unchanged + #[arg(long)] + pub artifact: PathBuf, + /// Device or candidate-device resource name shown during review + #[arg(long = "producer-name", value_parser = NonEmptyStringValueParser::new())] + pub producer_name: String, + /// SHA-256 key ID of the device that signed the artifact + #[arg(long = "producer-key-id", value_parser = NonEmptyStringValueParser::new())] + pub producer_key_id: String, + /// Owner-only file containing the pinned receipt public key + #[arg(long = "receipt-public-key-file")] + pub receipt_public_key_file: PathBuf, + /// SHA-256 key ID of the pinned Connect receipt key + #[arg(long = "receipt-key-id", value_parser = NonEmptyStringValueParser::new())] + pub receipt_key_id: String, + /// Bounded HTTPS request timeout + #[arg(long = "timeout-seconds", default_value_t = 30)] + pub timeout_seconds: u64, + /// Confirm the artifact, producer, destination, digest and classification were reviewed + #[arg(long = "acknowledge-reviewed", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_reviewed: bool, +} + +/// Support report operations. +#[derive(Args, Clone)] +pub struct ConnectReportOpts { + #[command(subcommand)] + pub command: ConnectReportCommands, +} + +/// Device-authenticated support report operations. +#[derive(Subcommand, Clone)] +pub enum ConnectReportCommands { + /// Upload one bounded archive through a short-lived object-store authorization + Upload(ConnectReportUploadOpts), +} + +/// `connect report upload` options. +#[derive(Args, Clone)] +pub struct ConnectReportUploadOpts { + /// Connect agent API HTTPS base URL + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub endpoint: String, + + /// PEM root CA file used for Connect and its authorized object store + #[arg(long = "ca-file")] + pub ca_file: PathBuf, + + /// Directory containing the registered Connect device identity + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + + /// Explicitly selected support report archive + #[arg(long)] + pub archive: PathBuf, + + /// Bounded timeout for each object upload request + #[arg(long = "upload-timeout-seconds", default_value_t = 600, value_parser = clap::value_parser!(u64).range(1..=900))] + pub upload_timeout_seconds: u64, +} + +#[derive(Args, Clone)] +pub struct ConnectInventoryOpts { + #[command(subcommand)] + pub command: ConnectInventoryCommands, +} + +#[derive(Subcommand, Clone)] +pub enum ConnectInventoryCommands { + /// Collect the bounded inventory.environment@1 summary + Environment(ConnectEnvironmentInventoryOpts), +} + +#[derive(Args, Clone)] +pub struct ConnectEnvironmentInventoryOpts { + /// Directory containing the persisted Connect inventory + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + /// Negotiated environment schema version + #[arg(long = "schema-version", default_value_t = 1)] + pub schema_version: u16, + /// Negotiated environment capability + #[arg(long, default_value = "inventory.environment@1", value_parser = NonEmptyStringValueParser::new())] + pub capability: String, + /// Maximum collection time in seconds + #[arg(long = "timeout-seconds", default_value_t = 30)] + pub timeout_seconds: u64, + /// Confirm this explicit local L1 inventory operation + #[arg(long = "acknowledge-l1", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_l1: bool, + /// New local signed archive path; omit to print the four-field JSON inventory + #[arg(long)] + pub output: Option, + #[arg(long)] + pub organization: Option, + #[arg(long)] + pub cluster: Option, + #[arg(long)] + pub device: Option, + #[arg(long = "run-uid")] + pub run_uid: Option, + #[arg(long = "artifact-uid")] + pub artifact_uid: Option, + #[arg(long = "consent-uid")] + pub consent_uid: Option, + #[arg(long = "policy-revision")] + pub policy_revision: Option, + #[arg(long = "expires-at")] + pub expires_at_unix: Option, +} + +#[derive(Args, Clone)] +pub struct ConnectTopOpts { + #[command(subcommand)] + pub command: ConnectTopCommands, +} + +#[derive(Subcommand, Clone)] +pub enum ConnectTopCommands { + /// Capture API activity when an approved typed source is available + Api(ConnectTopCaptureOpts), + /// Capture process disk I/O on supported platforms + Disk(ConnectTopCaptureOpts), + /// Capture lock activity when an approved typed source is available + Locks(ConnectTopCaptureOpts), + /// Capture internode network traffic + Net(ConnectTopCaptureOpts), + /// Capture RPC activity when an approved typed source is available + Rpc(ConnectTopCaptureOpts), +} + +#[derive(Args, Clone)] +pub struct ConnectTopCaptureOpts { + /// Directory containing an enrolled Connect device identity + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + /// New local archive path; an existing file is never replaced + #[arg(long)] + pub output: PathBuf, + /// Organization resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub organization: String, + /// Cluster resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub cluster: String, + /// Cluster-device resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub device: String, + /// UUIDv7 diagnostic run identifier issued by Connect + #[arg(long = "run-uid", value_parser = NonEmptyStringValueParser::new())] + pub run_uid: String, + /// UUIDv7 artifact identifier issued by Connect + #[arg(long = "artifact-uid", value_parser = NonEmptyStringValueParser::new())] + pub artifact_uid: String, + /// UUIDv7 consent identifier issued by Connect + #[arg(long = "consent-uid", value_parser = NonEmptyStringValueParser::new())] + pub consent_uid: String, + /// Consent policy revision bound to this capture + #[arg(long = "policy-revision")] + pub policy_revision: u64, + /// Consent expiry as UTC Unix seconds + #[arg(long = "consent-expires-at")] + pub consent_expires_at_unix: i64, + /// Diagnostic run expiry as UTC Unix seconds + #[arg(long = "run-expires-at")] + pub run_expires_at_unix: i64, + /// Monotonic sampling window in milliseconds + #[arg(long = "window-millis", default_value_t = 1_000)] + pub window_millis: u64, + /// Signed export validity in seconds + #[arg(long = "export-validity-seconds", default_value_t = 300)] + pub export_validity_seconds: u64, + /// Confirm this explicit local L3 diagnostic operation + #[arg(long = "acknowledge-l3", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_l3: bool, +} + +#[derive(Args, Clone)] +pub struct ConnectPerformanceOpts { + #[command(subcommand)] + pub command: ConnectPerformanceCommands, +} + +#[derive(Subcommand, Clone)] +pub enum ConnectPerformanceCommands { + /// Measure bounded client-to-deployment transfer performance + Client(Box), + /// Measure bounded S3 object throughput in a dedicated temporary namespace + Object(Box), + /// Measure bounded destination-confirmed site-replication performance + SiteReplication(Box), + /// Measure generated-file write and warm page-cache read performance + Drive(Box), +} + +#[derive(Args, Clone)] +pub struct ConnectSiteReplicationPerformanceOpts { + /// Directory containing an enrolled Connect device identity + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + /// Source RustFS deployment endpoint + #[arg(long = "source-endpoint", value_parser = NonEmptyStringValueParser::new())] + pub source_endpoint: String, + /// Destination RustFS deployment endpoint + #[arg(long = "destination-endpoint", value_parser = NonEmptyStringValueParser::new())] + pub destination_endpoint: String, + /// Optional PEM root certificate for the source endpoint + #[arg(long = "source-ca-file")] + pub source_ca_file: Option, + /// Optional PEM root certificate for the destination endpoint + #[arg(long = "destination-ca-file")] + pub destination_ca_file: Option, + /// Owner-readable file containing the source S3 access key + #[arg(long = "source-access-key-file")] + pub source_access_key_file: PathBuf, + /// Owner-readable file containing the source S3 secret key + #[arg(long = "source-secret-key-file")] + pub source_secret_key_file: PathBuf, + /// Optional owner-readable file containing a source S3 session token + #[arg(long = "source-session-token-file")] + pub source_session_token_file: Option, + /// Owner-readable file containing the destination S3 access key + #[arg(long = "destination-access-key-file")] + pub destination_access_key_file: PathBuf, + /// Owner-readable file containing the destination S3 secret key + #[arg(long = "destination-secret-key-file")] + pub destination_secret_key_file: PathBuf, + /// Optional owner-readable file containing a destination S3 session token + #[arg(long = "destination-session-token-file")] + pub destination_session_token_file: Option, + /// New local archive path; an existing file is never replaced + #[arg(long)] + pub output: PathBuf, + /// Negotiated producer schema version + #[arg(long = "schema-version", default_value_t = 1)] + pub schema_version: u16, + /// Negotiated producer capability + #[arg(long, default_value = "performance.siteReplication@1", value_parser = NonEmptyStringValueParser::new())] + pub capability: String, + /// Organization resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub organization: String, + /// Cluster resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub cluster: String, + /// Destination cluster resource name bound to the signed target pair + #[arg(long = "destination-cluster", value_parser = NonEmptyStringValueParser::new())] + pub destination_cluster: String, + /// Cluster-device resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub device: String, + /// UUIDv7 diagnostic run identifier issued by Connect + #[arg(long = "run-uid", value_parser = NonEmptyStringValueParser::new())] + pub run_uid: String, + /// UUIDv7 artifact identifier issued by Connect + #[arg(long = "artifact-uid", value_parser = NonEmptyStringValueParser::new())] + pub artifact_uid: String, + /// UUIDv7 consent identifier issued by Connect + #[arg(long = "consent-uid", value_parser = NonEmptyStringValueParser::new())] + pub consent_uid: String, + /// Consent policy revision bound to this measurement + #[arg(long = "policy-revision")] + pub policy_revision: u64, + /// Consent expiry as UTC Unix seconds + #[arg(long = "consent-expires-at")] + pub consent_expires_at_unix: i64, + /// Artifact expiry as UTC Unix seconds + #[arg(long = "expires-at")] + pub expires_at_unix: i64, + /// Generated transfer size in bytes + #[arg(long = "traffic-bytes", default_value_t = 65_536)] + pub traffic_bytes: u64, + /// Maximum wall-clock duration in milliseconds + #[arg(long = "duration-millis", default_value_t = 5_000)] + pub duration_millis: u64, + /// Stable opaque alias for the source deployment + #[arg(long = "source-alias", value_parser = NonEmptyStringValueParser::new())] + pub source_alias: String, + /// Source deployment identifier from site-replication configuration + #[arg(long = "source-deployment-id", value_parser = NonEmptyStringValueParser::new())] + pub source_deployment_id: String, + /// Stable opaque alias for the destination deployment + #[arg(long = "destination-alias", value_parser = NonEmptyStringValueParser::new())] + pub destination_alias: String, + /// Destination deployment identifier from site-replication configuration + #[arg(long = "destination-deployment-id", value_parser = NonEmptyStringValueParser::new())] + pub destination_deployment_id: String, + /// Existing versioned bucket dedicated to disposable performance objects + #[arg(long = "scratch-bucket", value_parser = NonEmptyStringValueParser::new())] + pub scratch_bucket: String, + /// Bounded cleanup window for versions that arrive after cancellation + #[arg(long = "late-arrival-cleanup-millis", default_value_t = 1_000)] + pub late_arrival_cleanup_millis: u64, + /// Confirm this explicit local L2 diagnostic operation + #[arg(long = "acknowledge-l2", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_l2: bool, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)] +pub enum ConnectClientPerformanceOperation { + Get, + Put, +} + +#[derive(Args, Clone)] +pub struct ConnectClientPerformanceOpts { + /// Directory containing an enrolled Connect device identity + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + + /// RustFS deployment endpoint + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub endpoint: String, + + /// Optional PEM root certificate for the deployment endpoint + #[arg(long = "ca-file")] + pub ca_file: Option, + + /// Optional explicit HTTP(S) proxy without embedded credentials + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub proxy: Option, + + /// Owner-readable file containing the S3 access key + #[arg(long = "access-key-file")] + pub access_key_file: PathBuf, + + /// Owner-readable file containing the S3 secret key + #[arg(long = "secret-key-file")] + pub secret_key_file: PathBuf, + + /// Optional owner-readable file containing an S3 session token + #[arg(long = "session-token-file")] + pub session_token_file: Option, + + /// New local archive path; an existing file is never replaced + #[arg(long)] + pub output: PathBuf, + + /// Negotiated producer schema version + #[arg(long = "schema-version", default_value_t = 1)] + pub schema_version: u16, + + /// Negotiated producer capability + #[arg(long, default_value = "performance.client@1", value_parser = NonEmptyStringValueParser::new())] + pub capability: String, + + /// Organization resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub organization: String, + + /// Cluster resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub cluster: String, + + /// Cluster-device resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub device: String, + + /// UUIDv7 diagnostic run identifier issued by Connect + #[arg(long = "run-uid", value_parser = NonEmptyStringValueParser::new())] + pub run_uid: String, + + /// UUIDv7 artifact identifier issued by Connect + #[arg(long = "artifact-uid", value_parser = NonEmptyStringValueParser::new())] + pub artifact_uid: String, + + /// UUIDv7 consent identifier issued by Connect + #[arg(long = "consent-uid", value_parser = NonEmptyStringValueParser::new())] + pub consent_uid: String, + + /// Consent policy revision bound to this measurement + #[arg(long = "policy-revision")] + pub policy_revision: u64, + + /// Consent expiry as UTC Unix seconds + #[arg(long = "consent-expires-at")] + pub consent_expires_at_unix: i64, + + /// Artifact expiry as UTC Unix seconds + #[arg(long = "expires-at")] + pub expires_at_unix: i64, + + /// Client transfer operation + #[arg(long, value_enum)] + pub operation: ConnectClientPerformanceOperation, + + /// Generated transfer size in bytes + #[arg(long = "traffic-bytes", default_value_t = 65_536)] + pub traffic_bytes: u64, + + /// Maximum wall-clock duration in milliseconds + #[arg(long = "duration-millis", default_value_t = 1_000)] + pub duration_millis: u64, + + /// Stable opaque alias for the deployment target + #[arg(long = "target-alias", default_value = "deployment-1", value_parser = NonEmptyStringValueParser::new())] + pub target_alias: String, + + /// Confirm this explicit local L1 diagnostic operation + #[arg(long = "acknowledge-l1", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_l1: bool, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)] +pub enum ConnectObjectPerformanceOperation { + Get, + Put, +} + +#[derive(Args, Clone)] +pub struct ConnectObjectPerformanceOpts { + /// Directory containing an enrolled Connect device identity + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + + /// RustFS deployment endpoint + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub endpoint: String, + + /// Optional PEM root certificate for the deployment endpoint + #[arg(long = "ca-file")] + pub ca_file: Option, + + /// Optional explicit HTTP(S) proxy without embedded credentials + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub proxy: Option, + + /// Owner-readable file containing the S3 access key + #[arg(long = "access-key-file")] + pub access_key_file: PathBuf, + + /// Owner-readable file containing the S3 secret key + #[arg(long = "secret-key-file")] + pub secret_key_file: PathBuf, + + /// Optional owner-readable file containing an S3 session token + #[arg(long = "session-token-file")] + pub session_token_file: Option, + + /// New local archive path; an existing file is never replaced + #[arg(long)] + pub output: PathBuf, + + /// Negotiated producer schema version + #[arg(long = "schema-version", default_value_t = 1)] + pub schema_version: u16, + + /// Negotiated producer capability + #[arg(long, default_value = "performance.object@1", value_parser = NonEmptyStringValueParser::new())] + pub capability: String, + + /// Organization resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub organization: String, + + /// Cluster resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub cluster: String, + + /// Cluster-device resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub device: String, + + /// UUIDv7 diagnostic run identifier issued by Connect + #[arg(long = "run-uid", value_parser = NonEmptyStringValueParser::new())] + pub run_uid: String, + + /// UUIDv7 artifact identifier issued by Connect + #[arg(long = "artifact-uid", value_parser = NonEmptyStringValueParser::new())] + pub artifact_uid: String, + + /// UUIDv7 consent identifier issued by Connect + #[arg(long = "consent-uid", value_parser = NonEmptyStringValueParser::new())] + pub consent_uid: String, + + /// Consent policy revision bound to this measurement + #[arg(long = "policy-revision")] + pub policy_revision: u64, + + /// Consent expiry as UTC Unix seconds + #[arg(long = "consent-expires-at")] + pub consent_expires_at_unix: i64, + + /// Artifact expiry as UTC Unix seconds + #[arg(long = "expires-at")] + pub expires_at_unix: i64, + + /// Object transfer operation + #[arg(long, value_enum)] + pub operation: ConnectObjectPerformanceOperation, + + /// Generated transfer size in bytes + #[arg(long = "traffic-bytes", default_value_t = 65_536)] + pub traffic_bytes: u64, + + /// Maximum wall-clock duration in milliseconds + #[arg(long = "duration-millis", default_value_t = 1_000)] + pub duration_millis: u64, + + /// Stable opaque alias for the deployment target + #[arg(long = "target-alias", default_value = "deployment-1", value_parser = NonEmptyStringValueParser::new())] + pub target_alias: String, + + /// Confirm this explicit local L1 diagnostic operation + #[arg(long = "acknowledge-l1", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_l1: bool, +} + +#[derive(Args, Clone)] +pub struct ConnectDrivePerformanceOpts { + /// Directory containing an enrolled Connect device identity + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + + /// Existing approved directory in which to create task-owned scratch state + #[arg(long = "scratch-dir")] + pub scratch_dir: PathBuf, + + /// New local archive path; an existing file is never replaced + #[arg(long)] + pub output: PathBuf, + + /// Negotiated producer schema version + #[arg(long = "schema-version", default_value_t = 1)] + pub schema_version: u16, + + /// Negotiated producer capability + #[arg(long, default_value = "performance.drive@1", value_parser = NonEmptyStringValueParser::new())] + pub capability: String, + + /// Organization resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub organization: String, + + /// Cluster resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub cluster: String, + + /// Cluster-device resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub device: String, + + /// UUIDv7 diagnostic run identifier issued by Connect + #[arg(long = "run-uid", value_parser = NonEmptyStringValueParser::new())] + pub run_uid: String, + + /// UUIDv7 artifact identifier issued by Connect + #[arg(long = "artifact-uid", value_parser = NonEmptyStringValueParser::new())] + pub artifact_uid: String, + + /// UUIDv7 consent identifier issued by Connect + #[arg(long = "consent-uid", value_parser = NonEmptyStringValueParser::new())] + pub consent_uid: String, + + /// Consent policy revision bound to this measurement + #[arg(long = "policy-revision")] + pub policy_revision: u64, + + /// Consent expiry as UTC Unix seconds + #[arg(long = "consent-expires-at")] + pub consent_expires_at_unix: i64, + + /// Artifact expiry as UTC Unix seconds + #[arg(long = "expires-at")] + pub expires_at_unix: i64, + + /// Maximum wall-clock duration in milliseconds + #[arg(long = "duration-millis", default_value_t = 1_000)] + pub duration_millis: u64, + + /// Generated scratch payload size in bytes + #[arg(long = "scratch-bytes", default_value_t = 32_768)] + pub scratch_bytes: u64, + + /// Individual read/write block size in bytes + #[arg(long = "block-bytes", default_value_t = 4_096)] + pub block_bytes: u64, + + /// Confirm this explicit local diagnostic operation + #[arg(long = "acknowledge-l1", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_l1: bool, +} + +#[derive(Args, Clone)] +pub struct ConnectTelemetryOpts { + #[command(subcommand)] + pub command: ConnectTelemetryCommands, +} + +#[derive(Subcommand, Clone)] +pub enum ConnectTelemetryCommands { + /// Capture process-local telemetry when an approved typed source is available + Record(ConnectTelemetryRecordOpts), + /// Forward an OTLP protobuf batch from stdin to a customer-approved collector + Otlp(ConnectTelemetryOtlpOpts), + /// Replay reviewed trace JSON read from stdin and write a signed export + Replay(ConnectTelemetryReplayOpts), +} + +#[derive(Args, Clone)] +pub struct ConnectTelemetryArtifactOpts { + /// Directory containing an enrolled Connect device identity + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + + /// New local archive path; an existing file is never replaced + #[arg(long)] + pub output: PathBuf, + + /// Organization resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub organization: String, + + /// Cluster resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub cluster: String, + + /// Cluster-device resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub device: String, + + /// UUIDv7 diagnostic run identifier issued by Connect + #[arg(long = "run-uid", value_parser = NonEmptyStringValueParser::new())] + pub run_uid: String, + + /// UUIDv7 artifact identifier issued by Connect + #[arg(long = "artifact-uid", value_parser = NonEmptyStringValueParser::new())] + pub artifact_uid: String, + + /// UUIDv7 consent identifier issued by Connect + #[arg(long = "consent-uid", value_parser = NonEmptyStringValueParser::new())] + pub consent_uid: String, + + /// Consent policy revision bound to this operation + #[arg(long = "policy-revision")] + pub policy_revision: u64, + + /// Consent expiry as UTC Unix seconds + #[arg(long = "consent-expires-at")] + pub consent_expires_at_unix: i64, + + /// Artifact expiry as UTC Unix seconds + #[arg(long = "expires-at")] + pub expires_at_unix: i64, + + /// Confirm this explicit local L3 telemetry operation + #[arg(long = "acknowledge-l3", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_l3: bool, +} + +#[derive(Args, Clone)] +pub struct ConnectTelemetryRecordOpts { + #[command(flatten)] + pub artifact: ConnectTelemetryArtifactOpts, + + /// Capture duration in milliseconds + #[arg(long = "duration-millis")] + pub duration_millis: u64, + + /// Maximum exported span count + #[arg(long = "max-spans", default_value_t = 1_024)] + pub max_spans: usize, +} + +#[derive(Args, Clone)] +pub struct ConnectTelemetryOtlpOpts { + #[command(flatten)] + pub artifact: ConnectTelemetryArtifactOpts, + + /// Customer-approved OTLP/HTTP traces endpoint + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub endpoint: String, + + /// Forward timeout in milliseconds + #[arg(long = "timeout-millis")] + pub timeout_millis: u64, + + /// Environment variable containing the local Authorization header value + #[arg(long = "authorization-env", value_parser = NonEmptyStringValueParser::new())] + pub authorization_env: Option, +} + +#[derive(Args, Clone)] +pub struct ConnectTelemetryReplayOpts { + #[command(flatten)] + pub artifact: ConnectTelemetryArtifactOpts, +} + +/// `connect logs` options. +#[derive(Args, Clone)] +pub struct ConnectLogsOpts { + /// Directory containing an enrolled Connect device identity + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + + /// New local archive path; an existing file is never replaced + #[arg(long)] + pub output: PathBuf, + + /// Capture the recent configured log window or tail new events + #[arg(long, value_enum, default_value = "batch")] + pub mode: ConnectLogsMode, + + /// Negotiated producer schema version + #[arg(long = "schema-version", default_value_t = 1)] + pub schema_version: u16, + + /// Negotiated producer capability + #[arg(long, default_value = "logs.capture@1")] + pub capability: String, + + /// Organization resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub organization: String, + + /// Cluster resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub cluster: String, + + /// Cluster-device resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub device: String, + + /// UUIDv7 diagnostic run identifier issued by Connect + #[arg(long = "run-uid", value_parser = NonEmptyStringValueParser::new())] + pub run_uid: String, + + /// UUIDv7 artifact identifier issued by Connect + #[arg(long = "artifact-uid", value_parser = NonEmptyStringValueParser::new())] + pub artifact_uid: String, + + /// UUIDv7 consent identifier issued by Connect + #[arg(long = "consent-uid", value_parser = NonEmptyStringValueParser::new())] + pub consent_uid: String, + + /// Consent policy revision bound to this capture + #[arg(long = "policy-revision")] + pub policy_revision: u64, + + /// Consent expiry as UTC Unix seconds + #[arg(long = "consent-expires-at")] + pub consent_expires_at_unix: i64, + + /// Artifact expiry as UTC Unix seconds + #[arg(long = "expires-at")] + pub expires_at_unix: i64, + + /// Batch lookback or live capture duration in milliseconds + #[arg(long = "duration-millis")] + pub duration_millis: u64, + + /// Maximum exported event count + #[arg(long = "max-events", default_value_t = 1_024)] + pub max_events: usize, + + /// Confirm this explicit local L3 log capture + #[arg(long = "acknowledge-l3", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_l3: bool, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)] +pub enum ConnectLogsMode { + Batch, + Live, +} + +/// `connect profile` options. +#[derive(Args, Clone)] +pub struct ConnectProfileOpts { + /// Directory containing an enrolled Connect device identity + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + + /// New local archive path; an existing file is never replaced + #[arg(long)] + pub output: PathBuf, + + /// Profile producer to run + #[arg(long, value_enum)] + pub tool: ConnectProfileTool, + + /// Thread source required by the threads producer + #[arg(long = "thread-scope", value_enum)] + pub thread_scope: Option, + + /// Negotiated producer schema version + #[arg(long = "schema-version", default_value_t = 1)] + pub schema_version: u16, + + /// Negotiated producer capability, such as profile.memory@1 + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub capability: String, + + /// Organization resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub organization: String, + + /// Cluster resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub cluster: String, + + /// Cluster-device resource name bound to the export + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub device: String, + + /// UUIDv7 diagnostic run identifier issued by Connect + #[arg(long = "run-uid", value_parser = NonEmptyStringValueParser::new())] + pub run_uid: String, + + /// UUIDv7 artifact identifier issued by Connect + #[arg(long = "artifact-uid", value_parser = NonEmptyStringValueParser::new())] + pub artifact_uid: String, + + /// UUIDv7 consent identifier issued by Connect + #[arg(long = "consent-uid", value_parser = NonEmptyStringValueParser::new())] + pub consent_uid: String, + + /// Consent policy revision bound to this capture + #[arg(long = "policy-revision")] + pub policy_revision: u64, + + /// Consent expiry as UTC Unix seconds + #[arg(long = "consent-expires-at")] + pub consent_expires_at_unix: i64, + + /// Artifact expiry as UTC Unix seconds + #[arg(long = "expires-at")] + pub expires_at_unix: i64, + + /// Maximum capture duration in milliseconds + #[arg(long = "duration-millis")] + pub duration_millis: u64, + + /// Sampling interval in microseconds + #[arg(long = "sample-period-micros")] + pub sample_period_micros: u64, + + /// Confirm this explicit local L3 profile capture + #[arg(long = "acknowledge-l3", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_l3: bool, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)] +pub enum ConnectProfileTool { + Cpu, + Memory, + Threads, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, ValueEnum)] +pub enum ConnectThreadProfileScope { + TokioRuntime, + NativeThreads, } /// `connect register` options @@ -147,6 +1133,138 @@ pub struct ConnectRegisterOpts { pub token_file: Option, } +/// Signed Connect service-license operations. +#[derive(Args, Clone)] +pub struct ConnectLicenseOpts { + #[command(subcommand)] + pub command: ConnectLicenseCommands, +} + +/// Local service-license operations. +#[derive(Subcommand, Clone)] +pub enum ConnectLicenseCommands { + /// Verify and atomically install a downloaded or hand-carried license file + Import(ConnectLicenseArtifactOpts), + /// Verify a license file without changing installed state + Verify(ConnectLicenseArtifactOpts), + /// Verify and display the installed license for one deployment and service + Show(ConnectLicenseScopeOpts), + /// Check Connect and install an operator-approved replacement license + Renew(ConnectLicenseRenewOpts), + /// Verify a license and write its reviewed relay envelope (Unix only) + RelayExport(ConnectLicenseRelayExportOpts), + /// Re-verify and install a reviewed relay envelope, then sign a destination receipt (Unix only) + RelayImport(ConnectLicenseRelayImportOpts), +} + +/// Online renewal transport plus local trust and scope pins. +#[derive(Args, Clone)] +pub struct ConnectLicenseRenewOpts { + /// HTTPS Connect agent API base URL + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub endpoint: String, + + /// PEM root CA file used only for this Connect endpoint + #[arg(long = "ca-file")] + pub ca_file: PathBuf, + + #[command(flatten)] + pub scope: ConnectLicenseScopeOpts, +} + +/// Trust and scope pins shared by service-license commands. +#[derive(Args, Clone)] +pub struct ConnectLicenseScopeOpts { + /// Directory containing local service-license state + #[arg(long = "state-dir")] + pub state_dir: PathBuf, + + /// File containing the pinned Ed25519 public key as canonical base64url + #[arg(long = "public-key-file")] + pub public_key_file: PathBuf, + + /// SHA-256 key ID for the pinned public key + #[arg(long = "key-id", value_parser = NonEmptyStringValueParser::new())] + pub key_id: String, + + /// Expected Connect license issuer + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub issuer: String, + + /// Expected RustFS license audience + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub audience: String, + + /// Expected organization resource name + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub organization: String, + + /// Expected deployment resource name + #[arg(long, value_parser = NonEmptyStringValueParser::new())] + pub deployment: String, + + /// Expected service code + #[arg(long = "service-code", value_parser = NonEmptyStringValueParser::new())] + pub service_code: String, +} + +/// A service-license file plus its local trust and scope pins. +#[derive(Args, Clone)] +pub struct ConnectLicenseArtifactOpts { + /// Downloaded or hand-carried signed license artifact + #[arg(long)] + pub artifact: PathBuf, + + #[command(flatten)] + pub scope: ConnectLicenseScopeOpts, +} + +/// A verified license artifact exported as an opaque relay envelope. +#[derive(Args, Clone)] +pub struct ConnectLicenseRelayExportOpts { + /// Downloaded signed license artifact to transfer unchanged + #[arg(long)] + pub artifact: PathBuf, + + /// New owner-only relay envelope file + #[arg(long)] + pub envelope: PathBuf, + + /// UUIDv7 identifying this exact relay attempt + #[arg(long = "transfer-uid", value_parser = NonEmptyStringValueParser::new())] + pub transfer_uid: String, + + #[command(flatten)] + pub scope: ConnectLicenseScopeOpts, + + /// Confirm the verified issuer, destination, digest, expiry, and license scope were reviewed + #[arg(long = "acknowledge-reviewed", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_reviewed: bool, +} + +/// A cluster-side relay import with a pre-bound destination receipt key. +#[derive(Args, Clone)] +pub struct ConnectLicenseRelayImportOpts { + /// Owner-only service-license relay envelope + #[arg(long)] + pub envelope: PathBuf, + + /// Owner-only Ed25519 seed for the pre-bound cluster receipt key + #[arg(long = "receipt-signing-key-file")] + pub receipt_signing_key_file: PathBuf, + + /// SHA-256 key ID of the pre-bound cluster receipt key + #[arg(long = "receipt-key-id", value_parser = NonEmptyStringValueParser::new())] + pub receipt_key_id: String, + + #[command(flatten)] + pub scope: ConnectLicenseScopeOpts, + + /// Confirm the verified issuer, destination, digest, expiry, and license scope were reviewed + #[arg(long = "acknowledge-reviewed", required = true, action = clap::ArgAction::SetTrue)] + pub acknowledge_reviewed: bool, +} + /// Offline inspection subcommand options #[derive(Args, Clone)] pub struct InspectOpts { @@ -446,6 +1564,32 @@ pub enum CommandResult { Inspect(InspectOpts), /// One-time Connect registration command ConnectRegister(ConnectRegisterOpts), + /// Local Connect service-license command + ConnectLicense(ConnectLicenseCommands), + /// Customer-operated relay of one reviewed signed artifact + ConnectRelay(Box), + /// Device-authenticated upload of one support report archive + ConnectReportUpload(ConnectReportUploadOpts), + /// Explicit local Connect environment inventory command + ConnectEnvironmentInventory(ConnectEnvironmentInventoryOpts), + /// Consent-bound local Connect drive performance export + ConnectDrivePerformance(ConnectDrivePerformanceOpts), + /// Consent-bound client-to-deployment performance export + ConnectClientPerformance(Box), + /// Consent-bound S3 object performance export + ConnectObjectPerformance(ConnectObjectPerformanceOpts), + /// Consent-bound site-replication performance export + ConnectSiteReplicationPerformance(Box), + /// Consent-bound local Connect profile export + ConnectProfile(ConnectProfileOpts), + /// Consent-bound local Connect log export + ConnectLogs(ConnectLogsOpts), + /// Consent-bound local Connect telemetry operation + ConnectTelemetry(ConnectTelemetryCommands), + /// Consent-bound local Connect top operation + ConnectTop(ConnectTopCommands), + /// Consent-bound local object integrity export + ConnectInspect(ConnectInspectObjectOpts), } /// Create default ServerOpts from environment variables @@ -485,7 +1629,10 @@ pub fn default_server_opts() -> ServerOpts { #[cfg(test)] mod tests { - use super::{Cli, Commands, ConnectCommands, InspectCommands, preprocess_args_for_legacy}; + use super::{ + Cli, Commands, ConnectCommands, ConnectInventoryCommands, ConnectLicenseCommands, ConnectRelayMaterialKind, + ConnectReportCommands, InspectCommands, preprocess_args_for_legacy, + }; use crate::version; use clap::error::ErrorKind; use clap::{CommandFactory, Parser}; @@ -582,13 +1729,132 @@ mod tests { let Some(Commands::Connect(connect)) = cli.command else { panic!("connect command expected"); }; - let ConnectCommands::Register(register) = connect.command; + let ConnectCommands::Register(register) = connect.command else { + panic!("connect register command expected"); + }; assert_eq!(register.endpoint, "https://connect.example/agent/"); assert_eq!(register.ca_file, std::path::Path::new("/etc/rustfs/connect-ca.pem")); assert_eq!(register.state_dir, std::path::Path::new("/var/lib/rustfs/connect")); assert!(register.token_file.is_none()); } + #[test] + fn connect_license_renew_requires_transport_and_scope_pins() { + let cli = Cli::try_parse_from([ + "rustfs", + "connect", + "license", + "renew", + "--endpoint", + "https://connect.example/agent/", + "--ca-file", + "/etc/rustfs/connect-ca.pem", + "--state-dir", + "/var/lib/rustfs/connect", + "--public-key-file", + "/etc/rustfs/connect-license.pub", + "--key-id", + "aabbcc", + "--issuer", + "connect", + "--audience", + "rustfs", + "--organization", + "organizations/018cc251-f400-7000-8000-000000000003", + "--deployment", + "organizations/018cc251-f400-7000-8000-000000000003/clusters/018cc251-f400-7000-8000-000000000004", + "--service-code", + "SUPPORT", + ]) + .expect("connect license renew arguments should parse"); + + let Some(Commands::Connect(connect)) = cli.command else { + panic!("connect command expected"); + }; + let ConnectCommands::License(license) = connect.command else { + panic!("connect license command expected"); + }; + let ConnectLicenseCommands::Renew(renew) = license.command else { + panic!("connect license renew command expected"); + }; + assert_eq!(renew.endpoint, "https://connect.example/agent/"); + assert_eq!(renew.scope.service_code, "SUPPORT"); + } + + #[test] + fn connect_relay_requires_protected_authentication_files_and_review() { + let cli = Cli::try_parse_from([ + "rustfs", "connect", "relay", "--endpoint", "https://connect.example/api/", "--ca-file", + "/etc/rustfs/connect-ca.pem", "--session-cookie-file", "/run/secrets/connect-cookie", "--csrf-token-file", + "/run/secrets/connect-csrf", "--organization-uid", "0198f3a1-4c00-7a10-8b21-0c1d2e3f4a50", + "--approval-reference", "0198f3a1-b100-7a10-8a11-001122334455", "--transfer-uid", + "0198f3a1-a200-7b20-8b22-112233445566", "--material-kind", "diagnostic-bundle-manifest", "--artifact", + "/var/lib/rustfs/relay/manifest.json", "--producer-name", + "organizations/0198f3a1-4c00-7a10-8b21-0c1d2e3f4a50/clusters/0198f3a1-5d00-7b20-9c31-1d2e3f4a5b61/clusterDevices/0198f3a1-6e00-7c30-ad41-2e3f4a5b6c72", + "--producer-key-id", "39ca24c8b02a559fd9beb2b1f5d18ced20c4bb246577b92914ae6814c3f70acf", + "--receipt-public-key-file", "/etc/rustfs/connect-relay.pub", "--receipt-key-id", + "aef7765496addd64bb9fcdd7b61682148622aed4856a7315326faea0aa86d53b", "--acknowledge-reviewed", + ]) + .expect("reviewed relay arguments should parse"); + let Some(Commands::Connect(connect)) = cli.command else { panic!("connect command expected") }; + let ConnectCommands::Relay(options) = connect.command else { panic!("relay command expected") }; + assert_eq!(options.material_kind, ConnectRelayMaterialKind::DiagnosticBundleManifest); + assert!(options.acknowledge_reviewed); + } + + #[test] + fn connect_report_upload_accepts_only_explicit_archive_and_transport_paths() { + let cli = Cli::try_parse_from([ + "rustfs", + "connect", + "report", + "upload", + "--endpoint", + "https://connect.example/agent/", + "--ca-file", + "/etc/rustfs/connect-ca.pem", + "--state-dir", + "/var/lib/rustfs/connect", + "--archive", + "/var/lib/rustfs/reports/support.tar.zst", + ]) + .expect("report upload arguments should parse"); + + let Some(Commands::Connect(connect)) = cli.command else { + panic!("connect command expected"); + }; + let ConnectCommands::Report(report) = connect.command else { + panic!("connect report command expected"); + }; + let ConnectReportCommands::Upload(upload) = report.command; + assert_eq!(upload.endpoint, "https://connect.example/agent/"); + assert_eq!(upload.archive, std::path::Path::new("/var/lib/rustfs/reports/support.tar.zst")); + assert_eq!(upload.upload_timeout_seconds, 600); + } + + #[test] + fn connect_report_upload_rejects_unbounded_timeout() { + let error = Cli::try_parse_from([ + "rustfs", + "connect", + "report", + "upload", + "--endpoint", + "https://connect.example/agent/", + "--ca-file", + "/etc/rustfs/connect-ca.pem", + "--state-dir", + "/var/lib/rustfs/connect", + "--archive", + "/var/lib/rustfs/reports/support.tar.zst", + "--upload-timeout-seconds", + "901", + ]) + .err() + .expect("unbounded upload timeout must fail"); + assert_eq!(error.kind(), ErrorKind::ValueValidation); + } + #[test] fn connect_register_has_no_token_value_or_environment_option() { for forbidden in ["--token", "--registration-token", "--token-env"] { @@ -623,6 +1889,165 @@ mod tests { assert!(help.to_string().contains("Unix only")); } + #[test] + fn connect_environment_inventory_requires_explicit_l1_acknowledgement() { + let error = Cli::try_parse_from([ + "rustfs", + "connect", + "inventory", + "environment", + "--state-dir", + "/var/lib/rustfs/connect", + ]) + .err() + .expect("unacknowledged L1 inventory must fail"); + assert_eq!(error.kind(), ErrorKind::MissingRequiredArgument); + assert!(error.to_string().contains("--acknowledge-l1")); + + let cli = Cli::try_parse_from([ + "rustfs", + "connect", + "inventory", + "environment", + "--state-dir", + "/var/lib/rustfs/connect", + "--acknowledge-l1", + ]) + .expect("acknowledged environment inventory parses"); + let Some(Commands::Connect(connect)) = cli.command else { + panic!("connect command expected"); + }; + let ConnectCommands::Inventory(inventory) = connect.command else { + panic!("inventory command expected"); + }; + let ConnectInventoryCommands::Environment(environment) = inventory.command; + assert_eq!(environment.schema_version, 1); + assert_eq!(environment.capability, "inventory.environment@1"); + assert_eq!(environment.timeout_seconds, 30); + assert!(environment.acknowledge_l1); + } + + #[test] + fn connect_profile_requires_explicit_l3_acknowledgement() { + let arguments = [ + "rustfs", + "connect", + "profile", + "--state-dir", + "/var/lib/rustfs/connect", + "--output", + "/tmp/profile.zip", + "--tool", + "memory", + "--capability", + "profile.memory@1", + "--organization", + "organizations/019e3ae0-0000-7000-8000-000000000001", + "--cluster", + "organizations/019e3ae0-0000-7000-8000-000000000001/clusters/019e3ae0-0000-7000-8000-000000000002", + "--device", + "organizations/019e3ae0-0000-7000-8000-000000000001/clusters/019e3ae0-0000-7000-8000-000000000002/clusterDevices/019e3ae0-0000-7000-8000-000000000003", + "--run-uid", + "019e3ae0-0000-7000-8000-000000000004", + "--artifact-uid", + "019e3ae0-0000-7000-8000-000000000005", + "--consent-uid", + "019e3ae0-0000-7000-8000-000000000006", + "--policy-revision", + "1", + "--consent-expires-at", + "4102444800", + "--expires-at", + "4102444700", + "--duration-millis", + "10", + "--sample-period-micros", + "1000", + ]; + let error = Cli::try_parse_from(arguments) + .err() + .expect("an incomplete unacknowledged profile must fail"); + assert_eq!(error.kind(), ErrorKind::MissingRequiredArgument); + assert!(error.to_string().contains("--acknowledge-l3")); + } + + #[test] + fn connect_logs_requires_explicit_l3_acknowledgement() { + let arguments = [ + "rustfs", + "connect", + "logs", + "--state-dir", + "/var/lib/rustfs/connect", + "--output", + "/tmp/logs.zip", + "--organization", + "organizations/019e3ae0-0000-7000-8000-000000000001", + "--cluster", + "organizations/019e3ae0-0000-7000-8000-000000000001/clusters/019e3ae0-0000-7000-8000-000000000002", + "--device", + "organizations/019e3ae0-0000-7000-8000-000000000001/clusters/019e3ae0-0000-7000-8000-000000000002/clusterDevices/019e3ae0-0000-7000-8000-000000000003", + "--run-uid", + "019e3ae0-0000-7000-8000-000000000004", + "--artifact-uid", + "019e3ae0-0000-7000-8000-000000000005", + "--consent-uid", + "019e3ae0-0000-7000-8000-000000000006", + "--policy-revision", + "1", + "--consent-expires-at", + "4102444800", + "--expires-at", + "4102444700", + "--duration-millis", + "1000", + ]; + let error = Cli::try_parse_from(arguments) + .err() + .expect("unacknowledged log capture must fail"); + assert_eq!(error.kind(), ErrorKind::MissingRequiredArgument); + assert!(error.to_string().contains("--acknowledge-l3")); + } + + #[test] + fn connect_telemetry_record_requires_explicit_l3_acknowledgement() { + let arguments = [ + "rustfs", + "connect", + "telemetry", + "record", + "--state-dir", + "/var/lib/rustfs/connect", + "--output", + "/tmp/telemetry.zip", + "--organization", + "organizations/019e3ae0-0000-7000-8000-000000000001", + "--cluster", + "organizations/019e3ae0-0000-7000-8000-000000000001/clusters/019e3ae0-0000-7000-8000-000000000002", + "--device", + "organizations/019e3ae0-0000-7000-8000-000000000001/clusters/019e3ae0-0000-7000-8000-000000000002/clusterDevices/019e3ae0-0000-7000-8000-000000000003", + "--run-uid", + "019e3ae0-0000-7000-8000-000000000004", + "--artifact-uid", + "019e3ae0-0000-7000-8000-000000000005", + "--consent-uid", + "019e3ae0-0000-7000-8000-000000000006", + "--policy-revision", + "1", + "--consent-expires-at", + "4102444800", + "--expires-at", + "4102444700", + "--duration-millis", + "10", + ]; + let error = Cli::try_parse_from(arguments) + .err() + .expect("unacknowledged telemetry capture must fail"); + assert_eq!(error.kind(), ErrorKind::MissingRequiredArgument); + assert!(error.to_string().contains("--acknowledge-l3")); + } + #[test] fn server_help_lists_allocator_reclaim_environment() { let result = Cli::try_parse_from(["rustfs", "server", "--help"]); diff --git a/rustfs/src/config/config_test.rs b/rustfs/src/config/config_test.rs index 058bb8baf..be764921b 100644 --- a/rustfs/src/config/config_test.rs +++ b/rustfs/src/config/config_test.rs @@ -126,6 +126,39 @@ mod tests { } } + #[test] + #[serial] + fn test_connect_inspect_object_parses() { + let result = Opt::parse_command([ + "rustfs", "connect", "inspect", "object", "--state-dir", "/state", "--output", "/tmp/inspect.zip", + "--organization", "organizations/019e3ae0-0000-7000-8000-000000000021", "--cluster", + "organizations/019e3ae0-0000-7000-8000-000000000021/clusters/019e3ae0-0000-7000-8000-000000000022", + "--device", + "organizations/019e3ae0-0000-7000-8000-000000000021/clusters/019e3ae0-0000-7000-8000-000000000022/clusterDevices/019e3ae0-0000-7000-8000-000000000023", + "--run-uid", "019e3ae0-0000-7000-8000-000000000024", "--artifact-uid", + "019e3ae0-0000-7000-8000-000000000025", "--consent-uid", + "019e3ae0-0000-7000-8000-000000000026", "--policy-revision", "3", "--consent-expires-at", + "2000000000", "--expires-at", "2000000000", "--path", "/data/drive-1", "--path", "/data/drive-2", + "--bucket", "customer-bucket", "--object", "private/report.bin", "--acknowledge-l3", + ]) + .expect("connect inspect object should parse"); + + match result { + CommandResult::ConnectInspect(opts) => { + assert_eq!( + opts.paths, + [ + std::path::PathBuf::from("/data/drive-1"), + std::path::PathBuf::from("/data/drive-2"), + ] + ); + assert_eq!(opts.bucket, "customer-bucket"); + assert_eq!(opts.object, "private/report.bin"); + } + _ => panic!("expected Connect inspect command result"), + } + } + #[test] #[serial] fn test_parse_from_non_server_commands_falls_back_without_panicking() { diff --git a/rustfs/src/config/mod.rs b/rustfs/src/config/mod.rs index 8c30d2d99..05d66ce52 100644 --- a/rustfs/src/config/mod.rs +++ b/rustfs/src/config/mod.rs @@ -50,7 +50,29 @@ mod snapshot; mod config_test; // Re-export public types +#[cfg(test)] +pub(crate) use cli::Cli; +pub use cli::ConnectSiteReplicationPerformanceOpts; pub use cli::{CommandResult, InfoOpts, InfoType}; +pub use cli::{ + ConnectClientPerformanceOperation, ConnectClientPerformanceOpts, ConnectDrivePerformanceOpts, ConnectPerformanceCommands, +}; +pub use cli::{ConnectEnvironmentInventoryOpts, ConnectInventoryCommands}; +pub use cli::{ConnectInspectCommands, ConnectInspectObjectOpts, ConnectInspectOpts}; +pub use cli::{ + ConnectLicenseArtifactOpts, ConnectLicenseCommands, ConnectLicenseRelayExportOpts, ConnectLicenseRelayImportOpts, + ConnectLicenseRenewOpts, ConnectLicenseScopeOpts, +}; +pub use cli::{ConnectLogsMode, ConnectLogsOpts}; +pub use cli::{ConnectObjectPerformanceOperation, ConnectObjectPerformanceOpts}; +pub use cli::{ConnectProfileOpts, ConnectProfileTool, ConnectThreadProfileScope}; +pub use cli::{ConnectRelayMaterialKind, ConnectRelayOpts}; +pub use cli::{ConnectReportCommands, ConnectReportOpts, ConnectReportUploadOpts}; +pub use cli::{ + ConnectTelemetryArtifactOpts, ConnectTelemetryCommands, ConnectTelemetryOtlpOpts, ConnectTelemetryRecordOpts, + ConnectTelemetryReplayOpts, +}; +pub use cli::{ConnectTopCaptureOpts, ConnectTopCommands}; pub use cli::{DiagnoseFormat, DiagnoseOpts}; pub use cli::{InspectBucketMetaOpts, InspectCommands, InspectOpts}; pub use cli::{TlsCommands, TlsInspectOpts, TlsOpts}; diff --git a/rustfs/src/config/opt.rs b/rustfs/src/config/opt.rs index e9dbbe42c..ab90ec5ee 100644 --- a/rustfs/src/config/opt.rs +++ b/rustfs/src/config/opt.rs @@ -18,7 +18,10 @@ //! and methods for parsing command line arguments. use super::Config; -use super::cli::{Cli, CommandResult, Commands, ConnectCommands, ServerOpts, default_server_opts, preprocess_args_for_legacy}; +use super::cli::{ + Cli, CommandResult, Commands, ConnectCommands, ConnectInspectCommands, ConnectInventoryCommands, ConnectPerformanceCommands, + ConnectReportCommands, ServerOpts, default_server_opts, preprocess_args_for_legacy, +}; use crate::apply_external_env_compat; use CommandResult::Server; use clap::Parser; @@ -139,6 +142,29 @@ impl Opt { Some(Commands::Inspect(opts)) => Ok(CommandResult::Inspect(opts)), Some(Commands::Connect(opts)) => match opts.command { ConnectCommands::Register(opts) => Ok(CommandResult::ConnectRegister(opts)), + ConnectCommands::License(opts) => Ok(CommandResult::ConnectLicense(opts.command)), + ConnectCommands::Relay(opts) => Ok(CommandResult::ConnectRelay(opts)), + ConnectCommands::Report(opts) => match opts.command { + ConnectReportCommands::Upload(opts) => Ok(CommandResult::ConnectReportUpload(opts)), + }, + ConnectCommands::Inventory(opts) => match opts.command { + ConnectInventoryCommands::Environment(opts) => Ok(CommandResult::ConnectEnvironmentInventory(opts)), + }, + ConnectCommands::Performance(opts) => match opts.command { + ConnectPerformanceCommands::Client(opts) => Ok(CommandResult::ConnectClientPerformance(opts)), + ConnectPerformanceCommands::Drive(opts) => Ok(CommandResult::ConnectDrivePerformance(*opts)), + ConnectPerformanceCommands::Object(opts) => Ok(CommandResult::ConnectObjectPerformance(*opts)), + ConnectPerformanceCommands::SiteReplication(opts) => { + Ok(CommandResult::ConnectSiteReplicationPerformance(opts)) + } + }, + ConnectCommands::Profile(opts) => Ok(CommandResult::ConnectProfile(opts)), + ConnectCommands::Logs(opts) => Ok(CommandResult::ConnectLogs(opts)), + ConnectCommands::Telemetry(opts) => Ok(CommandResult::ConnectTelemetry(opts.command)), + ConnectCommands::Top(opts) => Ok(CommandResult::ConnectTop(opts.command)), + ConnectCommands::Inspect(opts) => match opts.command { + ConnectInspectCommands::Object(opts) => Ok(CommandResult::ConnectInspect(opts)), + }, }, Some(Commands::Server(opts)) => Self::server_command_result(Self::from_server_opts(*opts)), None => { diff --git a/rustfs/src/connect/client.rs b/rustfs/src/connect/client.rs index 9d283460f..76d4f4e36 100644 --- a/rustfs/src/connect/client.rs +++ b/rustfs/src/connect/client.rs @@ -22,6 +22,7 @@ use serde::Deserialize; use uuid::Uuid; use zeroize::Zeroizing; +use super::config::{ProxyConfig, ProxyConfigError}; use super::credential_store::{ CompletedRegistration, CredentialLock, CredentialStore, CredentialStoreError, DeviceCredential, PendingRegistration, PendingRotation, @@ -46,6 +47,7 @@ pub struct ConnectConfig<'a> { pub endpoint: &'a str, pub root_ca_pem: &'a [u8], pub timeout: Duration, + pub proxy: Option<&'a ProxyConfig>, } pub struct ConnectClient { @@ -54,6 +56,7 @@ pub struct ConnectClient { root_certificates: Vec>, client: Client, timeout: Duration, + proxy: Option, } pub(crate) enum RotationAttempt { @@ -106,13 +109,14 @@ impl ConnectClient { return Err(ClientError::RootCertificate); } - let client = build_client(&root_certificates, config.timeout, None)?; + let client = build_client(&root_certificates, config.timeout, None, config.proxy)?; Ok(Self { endpoint, roots, root_certificates, client, timeout: config.timeout, + proxy: config.proxy.cloned(), }) } @@ -327,7 +331,7 @@ impl ConnectClient { identity_pem.push(b'\n'); identity_pem.extend_from_slice(private_key.as_bytes()); let tls_identity = reqwest::Identity::from_pem(&identity_pem).map_err(|_| ClientError::IdentityCertificate)?; - let client = build_client(&self.root_certificates, self.timeout, Some(tls_identity))?; + let client = build_client(&self.root_certificates, self.timeout, Some(tls_identity), self.proxy.as_ref())?; let path = format!("clusterDevices/{}:rotateCredential", credential.uid); let url = self.url(&path)?; let response = match self.send_once(StatusCode::OK, client.post(url).json(&body)).await? { @@ -474,6 +478,7 @@ impl ConnectClient { F: FnMut() -> reqwest::RequestBuilder, { let mut last_status = None; + let mut last_transport_failure = None; for attempt in 0..MAX_ATTEMPTS { match request().send().await { Ok(response) if response.status() == success => return decode_response(response).await, @@ -498,27 +503,40 @@ impl ConnectClient { let reason = decode_reason(response).await; return Err(ClientError::Rejected { status, reason }); } - Err(error) if !error.is_timeout() && !error.is_connect() => return Err(ClientError::Transport(error)), - Err(_) => {} + Err(error) => { + if let Some(failure) = classify_transport_failure(&error, self.proxy.is_some()) { + last_transport_failure = Some(failure); + } else if !error.is_timeout() && !error.is_connect() { + return Err(ClientError::Transport(error)); + } + } } if attempt + 1 < MAX_ATTEMPTS { tokio::time::sleep(Duration::from_millis(50 * (attempt as u64 + 1))).await; } } + if let Some(failure) = last_transport_failure { + return Err(failure.into()); + } Err(ClientError::Unavailable { status: last_status }) } async fn send_once(&self, success: StatusCode, request: reqwest::RequestBuilder) -> Result { let response = match request.send().await { Ok(response) => response, - Err(error) if error.is_timeout() || error.is_connect() => { - return Ok(SingleRequest::Unavailable { - status: None, - retry_after: None, - }); + Err(error) => { + if let Some(failure) = classify_transport_failure(&error, self.proxy.is_some()) { + return Err(failure.into()); + } + if error.is_timeout() || error.is_connect() { + return Ok(SingleRequest::Unavailable { + status: None, + retry_after: None, + }); + } + return Err(ClientError::Transport(error)); } - Err(error) => return Err(ClientError::Transport(error)), }; let status = response.status(); if status == success { @@ -600,26 +618,73 @@ fn retry_after(headers: &header::HeaderMap, now: DateTime) -> Option], timeout: Duration, identity: Option, + proxy: Option<&ProxyConfig>, ) -> Result { let certificates = roots .iter() .map(|root| reqwest::Certificate::from_der(root.as_ref())) .collect::, _>>()?; let mut builder = Client::builder() + .no_proxy() .https_only(true) .redirect(reqwest::redirect::Policy::none()) .timeout(timeout) .tls_certs_only(certificates); + if let Some(proxy) = proxy { + builder = proxy.apply(builder)?; + } if let Some(identity) = identity { builder = builder.identity(identity); } builder.build().map_err(ClientError::Transport) } +#[derive(Clone, Copy)] +pub(crate) enum TransportFailure { + ProxyAuthentication, + ProxyRejected, + TlsPeer, +} + +pub(crate) fn classify_transport_failure(error: &reqwest::Error, proxy_configured: bool) -> Option { + if proxy_configured && error.status() == Some(StatusCode::PROXY_AUTHENTICATION_REQUIRED) { + return Some(TransportFailure::ProxyAuthentication); + } + let mut source = std::error::Error::source(error); + while let Some(error) = source { + let message = error.to_string().to_ascii_lowercase(); + if proxy_configured && (message.contains("proxy authentication") || message.contains("proxy authorization required")) { + return Some(TransportFailure::ProxyAuthentication); + } + if message.contains("certificate") + || message.contains("unknown issuer") + || message.contains("invalid peer") + || message.contains("not valid for") + { + return Some(TransportFailure::TlsPeer); + } + if proxy_configured && message.contains("tunnel error: unsuccessful") { + return Some(TransportFailure::ProxyRejected); + } + source = error.source(); + } + None +} + +impl From for ClientError { + fn from(failure: TransportFailure) -> Self { + match failure { + TransportFailure::ProxyAuthentication => Self::ProxyAuthentication, + TransportFailure::ProxyRejected => Self::ProxyRejected, + TransportFailure::TlsPeer => Self::TlsPeer, + } + } +} + async fn decode_response(mut response: reqwest::Response) -> Result { let body = read_body(&mut response).await?; serde_json::from_slice(&body).map_err(|_| ClientError::Response) @@ -663,6 +728,16 @@ pub enum ClientError { Endpoint, #[error("Connect root CA configuration is invalid")] RootCertificate, + #[error("Connect proxy configuration is invalid")] + ProxyConfiguration(#[from] ProxyConfigError), + #[error("Connect proxy authentication failed; verify the configured proxy credentials")] + ProxyAuthentication, + #[error( + "Connect proxy connection failed; verify proxy availability, credentials, the proxy allow-list, and the Connect endpoint" + )] + ProxyRejected, + #[error("Connect TLS peer certificate validation failed; verify the endpoint and configured root CA")] + TlsPeer, #[error( "Connect registration has a pending attempt for a different token; restore the original protected token configuration" )] @@ -687,7 +762,7 @@ pub enum ClientError { AccessRevoked { status: StatusCode, reason: Option }, #[error("Connect rejected the request with HTTP {status}; reason={reason:?}")] Rejected { status: StatusCode, reason: Option }, - #[error("Connect remained unavailable after bounded retries; last_status={status:?}")] + #[error("Connect availability check failed after bounded retries; last_status={status:?}")] Unavailable { status: Option }, #[error("Connect response exceeded the 1 MiB credential-response limit")] ResponseTooLarge, @@ -704,3 +779,34 @@ pub enum ClientError { #[error(transparent)] Credential(#[from] CredentialValidationError), } + +#[cfg(test)] +mod tests { + use super::*; + + fn status_error(status: StatusCode, url: &str) -> reqwest::Error { + reqwest::Response::from(http::Response::builder().status(status).body("").expect("HTTP response")) + .error_for_status() + .expect_err("error status") + .with_url(Url::parse(url).expect("request URL")) + } + + #[test] + fn request_url_digits_do_not_imply_proxy_authentication() { + let error = status_error(StatusCode::SERVICE_UNAVAILABLE, "https://127.0.0.1:40701/upload/407"); + assert!(error.to_string().contains("407")); + for proxy_configured in [false, true] { + assert!(classify_transport_failure(&error, proxy_configured).is_none()); + } + } + + #[test] + fn proxy_authentication_status_requires_a_configured_proxy() { + let error = status_error(StatusCode::PROXY_AUTHENTICATION_REQUIRED, "https://localhost/upload"); + assert!(matches!( + classify_transport_failure(&error, true), + Some(TransportFailure::ProxyAuthentication) + )); + assert!(classify_transport_failure(&error, false).is_none()); + } +} diff --git a/rustfs/src/connect/config.rs b/rustfs/src/connect/config.rs index 8ecc6f04b..f7c3febcf 100644 --- a/rustfs/src/connect/config.rs +++ b/rustfs/src/connect/config.rs @@ -14,16 +14,275 @@ use std::env; use std::ffi::OsString; -#[cfg(target_os = "linux")] -use std::fs; +use std::fmt; +#[cfg(unix)] +use std::fs::{self, OpenOptions}; +#[cfg(unix)] +use std::io::Read as _; +#[cfg(unix)] +use std::os::unix::fs::{MetadataExt as _, OpenOptionsExt as _, PermissionsExt as _}; use std::path::PathBuf; use std::time::Duration; -use super::{CredentialStore, IdentityStore}; +use reqwest::{ClientBuilder, NoProxy, Proxy, Url}; +use zeroize::Zeroizing; + +use super::{CredentialStore, IdentityStore, TrustedDiagnosticJobSigner}; pub const ENV_CONNECT_ENDPOINT: &str = "RUSTFS_CONNECT_ENDPOINT"; pub const ENV_CONNECT_ROOT_CA_FILE: &str = "RUSTFS_CONNECT_ROOT_CA_FILE"; pub const ENV_CONNECT_STATE_DIR: &str = "RUSTFS_CONNECT_STATE_DIR"; +pub const ENV_CONNECT_PROXY_URL: &str = "RUSTFS_CONNECT_PROXY_URL"; +pub const ENV_CONNECT_PROXY_BYPASS: &str = "RUSTFS_CONNECT_PROXY_BYPASS"; +pub const ENV_CONNECT_PROXY_USERNAME_FILE: &str = "RUSTFS_CONNECT_PROXY_USERNAME_FILE"; +pub const ENV_CONNECT_PROXY_PASSWORD_FILE: &str = "RUSTFS_CONNECT_PROXY_PASSWORD_FILE"; +pub const ENV_CONNECT_JOB_SIGNING_KEY_ID: &str = "RUSTFS_CONNECT_JOB_SIGNING_KEY_ID"; +pub const ENV_CONNECT_JOB_SIGNING_PUBLIC_KEY_FILE: &str = "RUSTFS_CONNECT_JOB_SIGNING_PUBLIC_KEY_FILE"; + +const MAX_PROXY_BYPASS_BYTES: usize = 2048; +const MAX_PROXY_USERNAME_BYTES: usize = 256; +const MAX_PROXY_PASSWORD_BYTES: usize = 4096; + +/// Explicit HTTP CONNECT proxy configuration for RustFS Connect traffic. +#[derive(Clone)] +pub struct ProxyConfig { + url: Url, + bypass: Option, + username: Option>, + password: Option>, +} + +impl ProxyConfig { + /// Creates an unauthenticated proxy configuration. + pub fn new(url: &str, bypass: Option<&str>) -> Result { + let url = proxy_url(url)?; + let bypass = proxy_bypass(bypass)?; + Ok(Self { + url, + bypass, + username: None, + password: None, + }) + } + + /// Adds HTTP Basic authentication without placing credentials in the proxy URL. + pub fn with_basic_auth(mut self, username: &str, password: &str) -> Result { + validate_proxy_secret(username, MAX_PROXY_USERNAME_BYTES)?; + validate_proxy_secret(password, MAX_PROXY_PASSWORD_BYTES)?; + self.username = Some(Zeroizing::new(username.to_owned())); + self.password = Some(Zeroizing::new(password.to_owned())); + Ok(self) + } + + /// Loads an explicit proxy and optional protected Basic-auth files from RustFS-specific environment variables. + pub fn from_env() -> Result, ProxyConfigError> { + Self::from_env_values( + env::var_os(ENV_CONNECT_PROXY_URL), + env::var_os(ENV_CONNECT_PROXY_BYPASS), + env::var_os(ENV_CONNECT_PROXY_USERNAME_FILE), + env::var_os(ENV_CONNECT_PROXY_PASSWORD_FILE), + ) + } + + pub(crate) fn apply(&self, builder: ClientBuilder) -> Result { + let mut proxy = Proxy::https(self.url.clone()).map_err(|_| ProxyConfigError::Url)?; + if let Some(bypass) = self.bypass.as_deref() { + proxy = proxy.no_proxy(NoProxy::from_string(bypass)); + } + if let (Some(username), Some(password)) = (&self.username, &self.password) { + proxy = proxy.basic_auth(username, password); + } + Ok(builder.proxy(proxy)) + } + + #[cfg(unix)] + fn from_env_values( + url: Option, + bypass: Option, + username_file: Option, + password_file: Option, + ) -> Result, ProxyConfigError> { + let configured = url.is_some() || bypass.is_some() || username_file.is_some() || password_file.is_some(); + if !configured { + return Ok(None); + } + let Some(url) = url else { + return Err(ProxyConfigError::Partial); + }; + if username_file.is_some() != password_file.is_some() { + return Err(ProxyConfigError::Partial); + } + let url = url.into_string().map_err(|_| ProxyConfigError::Encoding)?; + let bypass = bypass + .map(|value| value.into_string().map_err(|_| ProxyConfigError::Encoding)) + .transpose()?; + let mut config = Self::new(&url, bypass.as_deref())?; + if let (Some(username_file), Some(password_file)) = (username_file, password_file) { + let username = read_proxy_secret(PathBuf::from(username_file), MAX_PROXY_USERNAME_BYTES)?; + let password = read_proxy_secret(PathBuf::from(password_file), MAX_PROXY_PASSWORD_BYTES)?; + config = config.with_basic_auth(&username, &password)?; + } + Ok(Some(config)) + } + + #[cfg(not(unix))] + fn from_env_values( + url: Option, + bypass: Option, + username_file: Option, + password_file: Option, + ) -> Result, ProxyConfigError> { + if url.is_none() && bypass.is_none() && username_file.is_none() && password_file.is_none() { + Ok(None) + } else { + Err(ProxyConfigError::PlatformSecurity) + } + } +} + +impl fmt::Debug for ProxyConfig { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("ProxyConfig") + .field("configured", &true) + .field("bypass_configured", &self.bypass.is_some()) + .field("authentication_configured", &self.username.is_some()) + .finish() + } +} + +fn proxy_url(value: &str) -> Result { + let url = Url::parse(value).map_err(|_| ProxyConfigError::Url)?; + if url.scheme() != "http" + || url.host_str().is_none() + || url.cannot_be_a_base() + || !url.username().is_empty() + || url.password().is_some() + || !matches!(url.path(), "" | "/") + || url.query().is_some() + || url.fragment().is_some() + { + return Err(ProxyConfigError::Url); + } + Ok(url) +} + +fn proxy_bypass(value: Option<&str>) -> Result, ProxyConfigError> { + let Some(value) = value else { + return Ok(None); + }; + if value.is_empty() + || value.len() > MAX_PROXY_BYPASS_BYTES + || !value.is_ascii() + || value.split(',').any(|entry| !valid_bypass_entry(entry.trim())) + { + return Err(ProxyConfigError::Bypass); + } + Ok(Some(value.to_owned())) +} + +fn valid_bypass_entry(value: &str) -> bool { + if value == "*" || value.parse::().is_ok() { + return true; + } + if let Some((network, prefix)) = value.split_once('/') { + let Ok(address) = network.parse::() else { + return false; + }; + let Ok(prefix) = prefix.parse::() else { + return false; + }; + return prefix <= if address.is_ipv4() { 32 } else { 128 }; + } + let domain = value.strip_prefix('.').unwrap_or(value); + !domain.is_empty() + && domain.len() <= 253 + && domain.split('.').all(|label| { + !label.is_empty() + && label.len() <= 63 + && !label.starts_with('-') + && !label.ends_with('-') + && label.bytes().all(|byte| byte.is_ascii_alphanumeric() || byte == b'-') + }) +} + +fn validate_proxy_secret(value: &str, maximum: usize) -> Result<(), ProxyConfigError> { + if value.is_empty() || value.len() > maximum || value.chars().any(char::is_control) { + return Err(ProxyConfigError::Authentication); + } + Ok(()) +} + +#[cfg(unix)] +fn read_proxy_secret(path: PathBuf, maximum: usize) -> Result, ProxyConfigError> { + let initial = fs::symlink_metadata(&path).map_err(|source| ProxyConfigError::SecretFile { + path: path.clone(), + source, + })?; + if !initial.file_type().is_file() || initial.permissions().mode() & 0o077 != 0 { + return Err(ProxyConfigError::SecretFileSecurity { path }); + } + let mut options = OpenOptions::new(); + options.read(true).custom_flags(libc::O_NOFOLLOW | libc::O_CLOEXEC); + let mut file = options.open(&path).map_err(|source| ProxyConfigError::SecretFile { + path: path.clone(), + source, + })?; + let opened = file.metadata().map_err(|source| ProxyConfigError::SecretFile { + path: path.clone(), + source, + })?; + if !opened.is_file() + || opened.uid() != process_uid() + || opened.dev() != initial.dev() + || opened.ino() != initial.ino() + || opened.len() > maximum as u64 + 2 + { + return Err(ProxyConfigError::SecretFileSecurity { path }); + } + let mut bytes = Zeroizing::new(Vec::with_capacity(opened.len() as usize)); + file.read_to_end(&mut bytes).map_err(|source| ProxyConfigError::SecretFile { + path: path.clone(), + source, + })?; + while matches!(bytes.last(), Some(b'\n' | b'\r')) { + bytes.pop(); + } + let value = Zeroizing::new(String::from_utf8(std::mem::take(&mut *bytes)).map_err(|_| ProxyConfigError::Authentication)?); + validate_proxy_secret(&value, maximum)?; + Ok(value) +} + +#[cfg(unix)] +// SAFETY: geteuid has no pointer arguments or caller preconditions. +#[allow(unsafe_code)] +fn process_uid() -> u32 { + unsafe { libc::geteuid() } +} + +#[derive(Debug, thiserror::Error)] +pub enum ProxyConfigError { + #[error("Connect proxy configuration requires RUSTFS_CONNECT_PROXY_URL and both or neither authentication files")] + Partial, + #[error("Connect proxy configuration is not valid UTF-8")] + Encoding, + #[error("Connect proxy must be an HTTP base URL without credentials, path, query, or fragment")] + Url, + #[error("Connect proxy bypass rules are invalid")] + Bypass, + #[error("Connect proxy credentials are invalid")] + Authentication, + #[error("Connect proxy credential file must be an owner-only regular file: {path}")] + SecretFileSecurity { path: PathBuf }, + #[error("Connect proxy credential file could not be read: {path}")] + SecretFile { + path: PathBuf, + #[source] + source: std::io::Error, + }, + #[error("Connect proxy credential files require Unix filesystem security guarantees")] + PlatformSecurity, +} #[derive(Clone, Copy, Debug)] pub struct HeartbeatSchedule { @@ -54,6 +313,8 @@ pub struct HeartbeatConfig { pub credential_store: CredentialStore, pub state_path: PathBuf, pub schedule: HeartbeatSchedule, + pub proxy: Option, + pub diagnostic_job_signer: Option, } impl HeartbeatConfig { @@ -72,6 +333,8 @@ impl HeartbeatConfig { credential_store, state_path, schedule: HeartbeatSchedule::default(), + proxy: None, + diagnostic_job_signer: None, } } @@ -84,6 +347,8 @@ impl HeartbeatConfig { credential_store: CredentialStore::new(state_root.join("credential")), state_path: state_root.join("heartbeat/state.json"), schedule: HeartbeatSchedule::default(), + proxy: None, + diagnostic_job_signer: None, } } @@ -96,19 +361,48 @@ impl HeartbeatConfig { } pub fn from_env() -> Result, HeartbeatConfigError> { - Self::from_env_values( + let mut config = Self::from_env_values( env::var_os(ENV_CONNECT_ENDPOINT), env::var_os(ENV_CONNECT_ROOT_CA_FILE), env::var_os(ENV_CONNECT_STATE_DIR), - ) + env::var_os(ENV_CONNECT_PROXY_URL), + env::var_os(ENV_CONNECT_PROXY_BYPASS), + env::var_os(ENV_CONNECT_PROXY_USERNAME_FILE), + env::var_os(ENV_CONNECT_PROXY_PASSWORD_FILE), + )?; + let key_id = env::var_os(ENV_CONNECT_JOB_SIGNING_KEY_ID); + let public_key_file = env::var_os(ENV_CONNECT_JOB_SIGNING_PUBLIC_KEY_FILE); + if key_id.is_some() != public_key_file.is_some() { + return Err(HeartbeatConfigError::DiagnosticJobTrust); + } + if let (Some(config), Some(key_id), Some(public_key_file)) = (&mut config, key_id, public_key_file) { + let key_id = key_id.into_string().map_err(|_| HeartbeatConfigError::DiagnosticJobTrust)?; + config.diagnostic_job_signer = Some( + TrustedDiagnosticJobSigner::from_public_key_file(&PathBuf::from(public_key_file), key_id) + .map_err(|_| HeartbeatConfigError::DiagnosticJobTrust)?, + ); + } + Ok(config) } fn from_env_values( endpoint: Option, root_ca_file: Option, state_dir: Option, + proxy_url: Option, + proxy_bypass: Option, + proxy_username_file: Option, + proxy_password_file: Option, ) -> Result, HeartbeatConfigError> { - let configured = endpoint.is_some() || root_ca_file.is_some() || state_dir.is_some(); + let proxy_configured = + proxy_url.is_some() || proxy_bypass.is_some() || proxy_username_file.is_some() || proxy_password_file.is_some(); + let configured = endpoint.is_some() + || root_ca_file.is_some() + || state_dir.is_some() + || proxy_url.is_some() + || proxy_bypass.is_some() + || proxy_username_file.is_some() + || proxy_password_file.is_some(); if !configured { return Ok(None); } @@ -116,7 +410,10 @@ impl HeartbeatConfig { return Err(HeartbeatConfigError::Partial); }; let state_dir = PathBuf::from(state_dir); - if state_dir.as_os_str().is_empty() || endpoint.is_some() != root_ca_file.is_some() { + if state_dir.as_os_str().is_empty() + || endpoint.is_some() != root_ca_file.is_some() + || (proxy_configured && endpoint.is_none()) + { return Err(HeartbeatConfigError::Partial); } #[cfg(not(target_os = "linux"))] @@ -139,13 +436,19 @@ impl HeartbeatConfig { source, })?; #[cfg(target_os = "linux")] - Ok(Some(Self::new( - endpoint, - root_ca_pem, - IdentityStore::new(state_dir.join("identity")), - CredentialStore::new(state_dir.join("credential")), - state_dir.join("heartbeat/state.json"), - ))) + let proxy = ProxyConfig::from_env_values(proxy_url, proxy_bypass, proxy_username_file, proxy_password_file)?; + #[cfg(target_os = "linux")] + { + let mut config = Self::new( + endpoint, + root_ca_pem, + IdentityStore::new(state_dir.join("identity")), + CredentialStore::new(state_dir.join("credential")), + state_dir.join("heartbeat/state.json"), + ); + config.proxy = proxy; + Ok(Some(config)) + } } } @@ -165,17 +468,82 @@ pub enum HeartbeatConfigError { }, #[error("Connect inventory persistence requires Linux filesystem security guarantees")] PlatformSecurity, + #[error(transparent)] + Proxy(#[from] ProxyConfigError), + #[error("Connect diagnostic job signing trust configuration is invalid")] + DiagnosticJobTrust, } #[cfg(test)] mod tests { - use super::{HeartbeatConfig, HeartbeatConfigError}; + use super::{HeartbeatConfig, HeartbeatConfigError, ProxyConfig, ProxyConfigError}; use std::ffi::OsString; + #[test] + fn proxy_rejects_implicit_credentials_and_non_http_transport() { + assert!(matches!( + ProxyConfig::new("http://user:secret@proxy.example:8080", None), + Err(ProxyConfigError::Url) + )); + assert!(matches!(ProxyConfig::new("https://proxy.example:8443", None), Err(ProxyConfigError::Url))); + assert!(matches!( + ProxyConfig::new("http://proxy.example:8080/tunnel", None), + Err(ProxyConfigError::Url) + )); + } + + #[test] + fn proxy_debug_output_contains_no_endpoint_or_credentials() { + let proxy = ProxyConfig::new("http://sensitive-proxy.example:8080", Some("private.example")) + .expect("proxy") + .with_basic_auth("sensitive-user", "sensitive-password") + .expect("authentication"); + let debug = format!("{proxy:?}"); + + for secret in ["sensitive-proxy", "private.example", "sensitive-user", "sensitive-password"] { + assert!(!debug.contains(secret)); + } + assert!(debug.contains("authentication_configured: true")); + } + + #[test] + #[cfg(unix)] + fn proxy_authentication_requires_owner_only_regular_files() { + use std::os::unix::fs::PermissionsExt as _; + + let temp = tempfile::tempdir().expect("tempdir"); + let username = temp.path().join("username"); + let password = temp.path().join("password"); + std::fs::write(&username, b"proxy-user\n").expect("username"); + std::fs::write(&password, b"proxy-password\n").expect("password"); + std::fs::set_permissions(&username, std::fs::Permissions::from_mode(0o600)).expect("username mode"); + std::fs::set_permissions(&password, std::fs::Permissions::from_mode(0o644)).expect("password mode"); + + assert!(matches!( + ProxyConfig::from_env_values( + Some(OsString::from("http://proxy.example:8080")), + None, + Some(username.clone().into_os_string()), + Some(password.clone().into_os_string()), + ), + Err(ProxyConfigError::SecretFileSecurity { .. }) + )); + std::fs::set_permissions(&password, std::fs::Permissions::from_mode(0o600)).expect("private password mode"); + let proxy = ProxyConfig::from_env_values( + Some(OsString::from("http://proxy.example:8080")), + Some(OsString::from("localhost,127.0.0.1")), + Some(username.into_os_string()), + Some(password.into_os_string()), + ) + .expect("valid proxy") + .expect("configured proxy"); + assert!(!format!("{proxy:?}").contains("proxy-password")); + } + #[test] fn absent_environment_is_disabled_without_side_effects() { assert!( - HeartbeatConfig::from_env_values(None, None, None) + HeartbeatConfig::from_env_values(None, None, None, None, None, None, None) .expect("absent config") .is_none() ); @@ -184,7 +552,15 @@ mod tests { #[test] fn partial_environment_is_rejected() { assert!(matches!( - HeartbeatConfig::from_env_values(Some(OsString::from("https://connect.example/agent/")), None, None), + HeartbeatConfig::from_env_values( + Some(OsString::from("https://connect.example/agent/")), + None, + None, + None, + None, + None, + None, + ), Err(HeartbeatConfigError::Partial) )); assert!(matches!( @@ -192,11 +568,23 @@ mod tests { Some(OsString::from("https://connect.example/agent/")), Some(OsString::from("root.pem")), None, + None, + None, + None, + None, ), Err(HeartbeatConfigError::Partial) )); assert!(matches!( - HeartbeatConfig::from_env_values(None, Some(OsString::from("root.pem")), Some(OsString::from("state"))), + HeartbeatConfig::from_env_values( + None, + Some(OsString::from("root.pem")), + Some(OsString::from("state")), + None, + None, + None, + None, + ), Err(HeartbeatConfigError::Partial) )); } @@ -205,7 +593,7 @@ mod tests { #[cfg(target_os = "linux")] fn state_directory_alone_enables_local_inventory_without_transport() { let state = tempfile::tempdir().expect("tempdir").keep(); - let config = HeartbeatConfig::from_env_values(None, None, Some(state.clone().into_os_string())) + let config = HeartbeatConfig::from_env_values(None, None, Some(state.clone().into_os_string()), None, None, None, None) .expect("state-only config") .expect("enabled config"); @@ -224,6 +612,10 @@ mod tests { Some(OsString::from("https://connect.example/agent/")), Some(root.into_os_string()), Some(state.clone().into_os_string()), + None, + None, + None, + None, ) .expect("complete config") .expect("enabled config"); @@ -239,7 +631,7 @@ mod tests { #[cfg(not(target_os = "linux"))] fn configured_inventory_fails_without_linux_filesystem_guarantees() { assert!(matches!( - HeartbeatConfig::from_env_values(None, None, Some(OsString::from("state"))), + HeartbeatConfig::from_env_values(None, None, Some(OsString::from("state")), None, None, None, None), Err(HeartbeatConfigError::PlatformSecurity) )); assert!(matches!( @@ -247,6 +639,10 @@ mod tests { Some(OsString::from("https://connect.example/agent/")), Some(OsString::from("missing-root.pem")), Some(OsString::from("state")), + None, + None, + None, + None, ), Err(HeartbeatConfigError::PlatformSecurity) )); diff --git a/rustfs/src/connect/diagnostics/inspect.rs b/rustfs/src/connect/diagnostics/inspect.rs new file mode 100644 index 000000000..962f2c96b --- /dev/null +++ b/rustfs/src/connect/diagnostics/inspect.rs @@ -0,0 +1,1412 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Local-only, bounded object integrity analysis for `inspect.object@1`. +//! +//! Object names, versions, metadata, shards, hashes, paths, and reconstructed +//! bytes stay inside this module. The signed artifact contains only the closed +//! rule conclusions defined by the Connect Inspect contract. + +use std::collections::{BTreeMap, BTreeSet}; +use std::fs::{self, File, OpenOptions}; +use std::io::{Cursor, Read as _, Write as _}; +use std::path::{Component, Path, PathBuf}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use base64_simd::URL_SAFE_NO_PAD; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use rustfs_filemeta::{FileInfo, FileMeta}; +use rustfs_utils::HashAlgorithm; +use serde::Serialize; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use crate::connect::DeviceIdentity; +use crate::storage_api::inspect::{BitrotReader, Erasure, check_valid_bucket_name_strict, file_info_quorum_hash}; + +pub const INSPECT_SCHEMA_VERSION: u16 = 1; +pub const INSPECT_CAPABILITY: &str = "inspect.object@1"; +pub const MAX_INSPECT_DURATION: Duration = Duration::from_secs(30); +pub const MAX_LOCAL_READ_BYTES: u64 = 256 * 1024 * 1024; +pub const MAX_WORKING_MEMORY_BYTES: u64 = 64 * 1024 * 1024; +const MAX_XL_META_BYTES: u64 = 4 * 1024 * 1024; +const MAX_FINDINGS: usize = 3; +const MAX_RESULT_BYTES: usize = 64 * 1024; +const MAX_ENVELOPE_BYTES: usize = 16 * 1024; +const MAX_ARCHIVE_BYTES: usize = 96 * 1024; +const MAX_VALIDITY_SECONDS: i64 = 30 * 24 * 60 * 60; +const MAX_FUTURE_SKEW_SECONDS: i64 = 300; +const OUTPUT_MODE: u32 = 0o600; +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1\0"; +const RESULT_PATH: &str = "result.json"; +const ENVELOPE_PATH: &str = "envelope.json"; +const SIGNATURE_PATH: &str = "envelope.sig"; +static INSPECT_ACTIVE: AtomicBool = AtomicBool::new(false); + +#[derive(Clone, Debug)] +pub struct InspectRequest { + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub schema_version: u16, + pub capability: String, + pub consent: InspectArtifactConsent, + pub produced_at_unix: i64, + pub expires_at_unix: i64, + pub nonce: [u8; 32], + pub drive_roots: Vec, + pub bucket: String, + pub object: String, + pub version_id: Option, + pub rules: Vec, + pub max_duration: Duration, + pub max_read_bytes: u64, + pub max_memory_bytes: u64, + pub provenance: InspectProvenance, +} + +#[derive(Clone, Debug)] +pub struct InspectArtifactConsent { + pub consent_uid: String, + pub policy_revision: u64, + pub expires_at_unix: i64, + pub confirmed: bool, +} + +#[derive(Clone, Debug, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct InspectProvenance { + repository: &'static str, + source_commit: String, + executable_sha256: String, + rustfs_version: String, + os_family: OsFamily, + architecture: Architecture, + build_features: Vec, +} + +impl InspectProvenance { + pub fn new(source_commit: String, executable_sha256: String, rustfs_version: String, build_features: Vec) -> Self { + Self { + repository: "rustfs/rustfs", + source_commit, + executable_sha256, + rustfs_version, + os_family: OsFamily::current(), + architecture: Architecture::current(), + build_features, + } + } +} + +#[derive(Clone, Copy, Debug, Eq, Ord, PartialEq, PartialOrd)] +pub enum InspectRule { + ShardBitrot, + ShardAvailability, + MetadataIdentity, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum InspectOutcome { + Succeeded, + Partial, + Failed, + Unsupported, + Cancelled, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum InspectReasonCode { + Complete, + LimitExceeded, + SourceUnavailable, + PermissionDenied, + UnsupportedVersion, + Cancelled, + CollectionFailed, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum InspectRuleOutcome { + Pass, + Fail, + Indeterminate, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum InspectReason { + Verified, + CorruptShard, + MissingShard, + IdentityMismatch, + InvalidMetadata, + NoUsableMetadata, + UnsupportedFormat, + ReadDenied, + LimitExceeded, + Cancelled, + SourceChanged, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum Reconstruction { + Possible, + Impossible, + Unknown, +} + +#[derive(Clone, Debug, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct InspectFinding { + resource_alias: &'static str, + rule_id: RuleId, + outcome: InspectRuleOutcome, + reason: InspectReason, + #[serde(skip_serializing_if = "Option::is_none")] + missing_shard_count: Option, + #[serde(skip_serializing_if = "Option::is_none")] + corrupt_shard_count: Option, + #[serde(skip_serializing_if = "Option::is_none")] + metadata_match: Option, + reconstruction: Reconstruction, +} + +impl InspectFinding { + pub fn rule(&self) -> InspectRule { + self.rule_id.into() + } + + pub fn outcome(&self) -> InspectRuleOutcome { + self.outcome + } + + pub fn reason(&self) -> InspectReason { + self.reason + } + + pub fn missing_shard_count(&self) -> Option { + self.missing_shard_count + } + + pub fn corrupt_shard_count(&self) -> Option { + self.corrupt_shard_count + } + + pub fn metadata_match(&self) -> Option { + self.metadata_match + } + + pub fn reconstruction(&self) -> Reconstruction { + self.reconstruction + } +} + +#[derive(Clone, Debug, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct InspectDiagnosticResult { + schema_version: u16, + run_uid: String, + tool_id: &'static str, + capability: &'static str, + outcome: InspectOutcome, + reason_code: InspectReasonCode, + duration_millis: u64, + provenance: InspectProvenance, + coverage: Coverage, + data: Option, +} + +impl InspectDiagnosticResult { + pub fn outcome(&self) -> InspectOutcome { + self.outcome + } + + pub fn reason_code(&self) -> InspectReasonCode { + self.reason_code + } + + pub fn findings(&self) -> &[InspectFinding] { + self.data.as_ref().map_or(&[], |data| data.findings.as_slice()) + } +} + +#[derive(Clone, Debug)] +pub enum InspectRun { + Signed(SignedInspectExport), + Terminal(InspectDiagnosticResult), +} + +#[derive(Clone, Debug)] +pub struct SignedInspectExport { + pub artifact_uid: String, + pub envelope_json: Vec, + pub envelope_signature: Vec, + pub result_json: Vec, + pub archive_bytes: Vec, + pub archive_sha256: String, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct SavedInspectExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +#[derive(Debug, Error)] +pub enum InspectError { + #[error("inspect_invalid_request")] + InvalidRequest, + #[error("inspect_consent_required")] + ConsentRequired, + #[error("inspect_consent_expired")] + ConsentExpired, + #[error("inspect_expired")] + Expired, + #[error("inspect_unsupported_version")] + UnsupportedVersion, + #[error("inspect_unsupported_capability")] + UnsupportedCapability, + #[error("inspect_limit_exceeded")] + LimitExceeded, + #[error("inspect_cancelled")] + Cancelled, + #[error("inspect_already_running")] + Busy, + #[error("inspect_signing_failed")] + Signing, + #[error("inspect_export_exists")] + AlreadyExists, + #[error("inspect_io_failed")] + Io(#[source] std::io::Error), + #[error("inspect_encoding_failed")] + Encoding, + #[error("inspect_durability_failed_after_commit")] + DurabilityAfterCommit(#[source] std::io::Error), +} + +pub fn export_inspect_summary( + request: &InspectRequest, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + let _lease = CollectorLease::acquire()?; + let started = Instant::now(); + let result = analyze(request, cancel, started)?; + if !matches!(result.outcome, InspectOutcome::Succeeded | InspectOutcome::Partial) { + return Ok(InspectRun::Terminal(result)); + } + Ok(InspectRun::Signed(sign_result(request, result, key, cancel)?)) +} + +fn analyze( + request: &InspectRequest, + cancel: &CancellationToken, + started: Instant, +) -> Result { + if cancel.is_cancelled() { + return Ok(terminal(request, InspectOutcome::Cancelled, InspectReasonCode::Cancelled, started)); + } + let mut budget = ReadBudget::new(request.max_read_bytes); + let mut readable = Vec::new(); + let mut retained_bytes = 0u64; + let mut saw_permission_denied = false; + let mut saw_unsupported = false; + let mut saw_invalid = false; + + for root in &request.drive_roots { + if cancel.is_cancelled() { + return Ok(terminal(request, InspectOutcome::Cancelled, InspectReasonCode::Cancelled, started)); + } + if started.elapsed() >= request.max_duration { + return Ok(terminal(request, InspectOutcome::Failed, InspectReasonCode::LimitExceeded, started)); + } + let memory_remaining = request.max_memory_bytes.saturating_sub(retained_bytes); + match read_drive(root, request, &mut budget, memory_remaining, cancel, started) { + Ok(shard) => { + let Some(next_retained_bytes) = retained_bytes + .checked_add(shard.retained_bytes()) + .filter(|used| *used <= request.max_memory_bytes) + else { + return Ok(terminal(request, InspectOutcome::Failed, InspectReasonCode::LimitExceeded, started)); + }; + retained_bytes = next_retained_bytes; + readable.push(shard); + } + Err(ReadFailure::Missing) => {} + Err(ReadFailure::PermissionDenied) => saw_permission_denied = true, + Err(ReadFailure::Unsupported) => saw_unsupported = true, + Err(ReadFailure::Invalid) => saw_invalid = true, + Err(ReadFailure::Limit) => { + return Ok(terminal(request, InspectOutcome::Failed, InspectReasonCode::LimitExceeded, started)); + } + Err(ReadFailure::Changed) => { + return Ok(terminal(request, InspectOutcome::Failed, InspectReasonCode::SourceUnavailable, started)); + } + Err(ReadFailure::Cancelled) => { + return Ok(terminal(request, InspectOutcome::Cancelled, InspectReasonCode::Cancelled, started)); + } + } + } + + if readable.is_empty() { + if saw_permission_denied { + return Ok(terminal(request, InspectOutcome::Failed, InspectReasonCode::PermissionDenied, started)); + } + if saw_unsupported { + return Ok(terminal( + request, + InspectOutcome::Unsupported, + InspectReasonCode::UnsupportedVersion, + started, + )); + } + let findings = request + .rules + .iter() + .copied() + .map(|rule| { + indeterminate( + rule, + if saw_invalid { + InspectReason::InvalidMetadata + } else { + InspectReason::NoUsableMetadata + }, + ) + }) + .collect(); + return Ok(success_result(request, findings, started)); + } + + let incomplete_reason = if saw_permission_denied { + Some(InspectReason::ReadDenied) + } else if saw_unsupported { + Some(InspectReason::UnsupportedFormat) + } else if saw_invalid { + Some(InspectReason::InvalidMetadata) + } else { + None + }; + let findings = match evaluate(request, readable, retained_bytes, incomplete_reason, cancel, started) { + Ok(findings) => findings, + Err(InspectError::LimitExceeded) => { + return Ok(terminal(request, InspectOutcome::Failed, InspectReasonCode::LimitExceeded, started)); + } + Err(InspectError::Cancelled) => { + return Ok(terminal(request, InspectOutcome::Cancelled, InspectReasonCode::Cancelled, started)); + } + Err(error) => return Err(error), + }; + Ok(success_result(request, findings, started)) +} + +fn evaluate( + request: &InspectRequest, + shards: Vec, + retained_bytes: u64, + incomplete_reason: Option, + cancel: &CancellationToken, + started: Instant, +) -> Result, InspectError> { + let mut groups: BTreeMap<[u8; 32], Vec> = BTreeMap::new(); + for shard in shards { + groups.entry(file_info_quorum_hash(&shard.file_info)).or_default().push(shard); + } + let metadata_mismatch = groups.len() > 1; + let mut selected = groups + .into_values() + .max_by_key(Vec::len) + .ok_or(InspectError::InvalidRequest)?; + selected.sort_by_key(|shard| shard.index); + selected.dedup_by_key(|shard| shard.index); + let file_info = selected.first().ok_or(InspectError::InvalidRequest)?.file_info.clone(); + let total = file_info + .erasure + .data_blocks + .checked_add(file_info.erasure.parity_blocks) + .ok_or(InspectError::LimitExceeded)?; + let total = u16::try_from(total).map_err(|_| InspectError::LimitExceeded)?; + if total == 0 || total > 256 { + return Err(InspectError::LimitExceeded); + } + let indices = selected.iter().map(|shard| shard.index).collect::>(); + let corrupt = selected + .iter() + .filter(|shard| matches!(shard.state, ShardState::Corrupt)) + .count(); + let locally_missing = selected + .iter() + .filter(|shard| matches!(shard.state, ShardState::Missing)) + .count(); + let verified = selected + .iter() + .filter(|shard| matches!(shard.state, ShardState::Verified(_))) + .count(); + let absent = usize::from(total).saturating_sub(indices.len()); + let missing = absent.saturating_add(locally_missing); + let reconstruction = reconstruct( + &file_info, + &selected, + retained_bytes, + request.max_memory_bytes, + request.max_duration, + cancel, + started, + )?; + let missing = u16::try_from(missing).map_err(|_| InspectError::LimitExceeded)?; + let corrupt = u16::try_from(corrupt).map_err(|_| InspectError::LimitExceeded)?; + let mut findings = Vec::with_capacity(request.rules.len()); + for rule in &request.rules { + findings.push(match rule { + InspectRule::ShardBitrot if corrupt > 0 => finding( + *rule, + InspectRuleOutcome::Fail, + InspectReason::CorruptShard, + Some(missing), + Some(corrupt), + None, + Reconstruction::Unknown, + ), + InspectRule::ShardBitrot if let Some(reason) = incomplete_reason => indeterminate(*rule, reason), + InspectRule::ShardBitrot => finding( + *rule, + InspectRuleOutcome::Pass, + InspectReason::Verified, + Some(missing), + Some(0), + None, + Reconstruction::Unknown, + ), + InspectRule::ShardAvailability if let Some(reason) = incomplete_reason => indeterminate(*rule, reason), + InspectRule::ShardAvailability if missing > 0 => finding( + *rule, + InspectRuleOutcome::Fail, + InspectReason::MissingShard, + Some(missing), + Some(corrupt), + None, + reconstruction, + ), + InspectRule::ShardAvailability if verified == usize::from(total) => finding( + *rule, + InspectRuleOutcome::Pass, + InspectReason::Verified, + Some(0), + Some(0), + None, + reconstruction, + ), + InspectRule::ShardAvailability => indeterminate(*rule, InspectReason::NoUsableMetadata), + InspectRule::MetadataIdentity if metadata_mismatch => finding( + *rule, + InspectRuleOutcome::Fail, + InspectReason::IdentityMismatch, + None, + None, + Some(false), + Reconstruction::Unknown, + ), + InspectRule::MetadataIdentity if let Some(reason) = incomplete_reason => indeterminate(*rule, reason), + InspectRule::MetadataIdentity => finding( + *rule, + InspectRuleOutcome::Pass, + InspectReason::Verified, + None, + None, + Some(true), + Reconstruction::Unknown, + ), + }); + } + Ok(findings) +} + +fn reconstruct( + file_info: &FileInfo, + shards: &[DriveShard], + retained_bytes: u64, + memory_limit: u64, + max_duration: Duration, + cancel: &CancellationToken, + started: Instant, +) -> Result { + check_bounds(cancel, started, max_duration)?; + let k = file_info.erasure.data_blocks; + let m = file_info.erasure.parity_blocks; + let verified = shards + .iter() + .filter(|shard| matches!(shard.state, ShardState::Verified(_))) + .count(); + if verified < k { + return Ok(Reconstruction::Impossible); + } + let size = usize::try_from(file_info.size).map_err(|_| InspectError::InvalidRequest)?; + let erasure = Erasure::try_new_with_options(k, m, file_info.erasure.block_size, file_info.uses_legacy_checksum) + .map_err(|_| InspectError::InvalidRequest)?; + let total = k.checked_add(m).ok_or(InspectError::LimitExceeded)?; + let shard_total = erasure.shard_file_offset(0, size, size); + let mut object_offset = 0usize; + let mut shard_offset = 0usize; + while object_offset < size { + check_bounds(cancel, started, max_duration)?; + let block_len = (size - object_offset).min(erasure.block_size); + let block_shard_len = if object_offset + block_len >= size { + shard_total.checked_sub(shard_offset).ok_or(InspectError::InvalidRequest)? + } else { + erasure.shard_size() + }; + let verified_slots = shards + .iter() + .filter(|shard| matches!(shard.state, ShardState::Verified(_))) + .count(); + let slot_bytes = total + .checked_mul(std::mem::size_of::>>()) + .and_then(|overhead| { + block_shard_len + .checked_mul(verified_slots) + .and_then(|data| overhead.checked_add(data)) + }) + .and_then(|bytes| u64::try_from(bytes).ok()) + .ok_or(InspectError::LimitExceeded)?; + if retained_bytes.checked_add(slot_bytes).is_none_or(|peak| peak > memory_limit) { + return Err(InspectError::LimitExceeded); + } + let mut slots = vec![None; total]; + for shard in shards { + let ShardState::Verified(bytes) = &shard.state else { continue }; + let end = shard_offset.checked_add(block_shard_len).ok_or(InspectError::LimitExceeded)?; + if shard.index > 0 && shard.index <= total && end <= bytes.len() { + slots[shard.index - 1] = Some(bytes[shard_offset..end].to_vec()); + } + } + erasure.decode_data(&mut slots).map_err(|_| InspectError::InvalidRequest)?; + for slot in slots.iter().take(k) { + if slot.is_none() { + return Ok(Reconstruction::Impossible); + } + } + object_offset = object_offset.checked_add(block_len).ok_or(InspectError::LimitExceeded)?; + shard_offset = shard_offset.checked_add(block_shard_len).ok_or(InspectError::LimitExceeded)?; + } + Ok(Reconstruction::Possible) +} + +fn read_drive( + root: &Path, + request: &InspectRequest, + budget: &mut ReadBudget, + memory_remaining: u64, + cancel: &CancellationToken, + started: Instant, +) -> Result { + check_read_bounds(cancel, started, request.max_duration)?; + let object_dir = object_dir(root, &request.bucket, &request.object)?; + let xl_path = object_dir.join("xl.meta"); + let before = fs::metadata(&xl_path).map_err(map_read_error)?; + if before.len() > MAX_XL_META_BYTES { + return Err(ReadFailure::Limit); + } + let bytes = budget.read(&xl_path, MAX_XL_META_BYTES)?; + if bytes.len() >= 6 && &bytes[..4] == b"XL2 " && u16::from_le_bytes([bytes[4], bytes[5]]) > 1 { + return Err(ReadFailure::Unsupported); + } + let metadata = FileMeta::load(&bytes).map_err(|_| ReadFailure::Invalid)?; + let version = request.version_id.as_deref().unwrap_or_default(); + let mut file_info = metadata + .into_fileinfo(&request.bucket, &request.object, version, true, false, true) + .map_err(|_| ReadFailure::Invalid)?; + file_info.validate_for_metadata_read().map_err(|_| ReadFailure::Invalid)?; + if file_info.parts.len() != 1 || file_info.parts.first().is_none_or(|part| part.number != 1) { + return Err(ReadFailure::Unsupported); + } + let index = file_info.erasure.index; + let state = + match read_verified_shard(&object_dir, &file_info, budget, memory_remaining, request.max_duration, cancel, started) { + Ok(bytes) => ShardState::Verified(bytes), + Err(ReadFailure::Missing) => ShardState::Missing, + Err(ReadFailure::Invalid) => ShardState::Corrupt, + Err(error) => return Err(error), + }; + file_info.data = None; + let after = fs::metadata(&xl_path).map_err(map_read_error)?; + if before.len() != after.len() || before.modified().ok() != after.modified().ok() { + return Err(ReadFailure::Changed); + } + Ok(DriveShard { index, file_info, state }) +} + +fn read_verified_shard( + object_dir: &Path, + file_info: &FileInfo, + budget: &mut ReadBudget, + memory_limit: u64, + max_duration: Duration, + cancel: &CancellationToken, + started: Instant, +) -> Result, ReadFailure> { + check_read_bounds(cancel, started, max_duration)?; + let checksum = file_info.erasure.get_checksum_info(1); + let algorithm = if file_info.uses_legacy_checksum && checksum.algorithm == HashAlgorithm::HighwayHash256S { + HashAlgorithm::HighwayHash256SLegacy + } else { + checksum.algorithm + }; + let erasure = Erasure::try_new_with_options( + file_info.erasure.data_blocks, + file_info.erasure.parity_blocks, + file_info.erasure.block_size, + file_info.uses_legacy_checksum, + ) + .map_err(|_| ReadFailure::Invalid)?; + let size = usize::try_from(file_info.size).map_err(|_| ReadFailure::Invalid)?; + let expected = erasure.shard_file_offset(0, size, size); + let source = if let Some(inline) = file_info.data.as_ref() { + let inline_bytes = u64::try_from(inline.len()).map_err(|_| ReadFailure::Limit)?; + if inline_bytes.checked_mul(2).is_none_or(|peak| peak > memory_limit) { + return Err(ReadFailure::Limit); + } + budget.charge(inline_bytes)?; + inline.to_vec() + } else { + let data_dir = file_info.data_dir.ok_or(ReadFailure::Invalid)?; + budget.read(&object_dir.join(data_dir.to_string()).join("part.1"), memory_limit)? + }; + let streaming = matches!(algorithm, HashAlgorithm::HighwayHash256S | HashAlgorithm::HighwayHash256SLegacy); + if !streaming { + if checksum.hash.len() != algorithm.size() || algorithm.hash_encode(&source).as_ref() != checksum.hash.as_ref() { + return Err(ReadFailure::Invalid); + } + return Ok(source); + } + let transient_bytes = u64::try_from(source.len()) + .ok() + .and_then(|source_len| source_len.checked_add(u64::try_from(expected).ok()?)) + .ok_or(ReadFailure::Limit)?; + let inline_bytes = file_info + .data + .as_ref() + .map_or(0, |data| u64::try_from(data.len()).unwrap_or(u64::MAX)); + if transient_bytes + .checked_add(inline_bytes) + .is_none_or(|peak| peak > memory_limit) + { + return Err(ReadFailure::Limit); + } + let blocks = if expected == 0 { + 0 + } else { + expected.div_ceil(erasure.shard_size().max(1)) + }; + let framed = expected + .checked_add(blocks.checked_mul(algorithm.size()).ok_or(ReadFailure::Limit)?) + .ok_or(ReadFailure::Limit)?; + let (expected, blocks) = if framed == source.len() { + (expected, blocks) + } else if source.len() >= algorithm.size() && blocks <= 1 { + (source.len() - algorithm.size(), 1) + } else { + return Err(ReadFailure::Invalid); + }; + let frame_size = if blocks <= 1 { + expected.max(1) + } else { + erasure.shard_size().max(1) + }; + let mut reader = BitrotReader::new(Cursor::new(source), frame_size, algorithm, false); + let mut output = vec![0; expected]; + let mut offset = 0usize; + for _ in 0..blocks { + check_read_bounds(cancel, started, max_duration)?; + let wanted = (expected - offset).min(frame_size); + let read = + futures::executor::block_on(reader.read(&mut output[offset..offset + wanted])).map_err(|_| ReadFailure::Invalid)?; + offset = offset.checked_add(read).ok_or(ReadFailure::Limit)?; + } + if offset != expected { + return Err(ReadFailure::Invalid); + } + Ok(output) +} + +fn object_dir(root: &Path, bucket: &str, object: &str) -> Result { + if check_valid_bucket_name_strict(bucket).is_err() + || object.is_empty() + || object.len() > 1_024 + || Path::new(object).is_absolute() + || Path::new(object) + .components() + .any(|component| !matches!(component, Component::Normal(_))) + { + return Err(ReadFailure::Invalid); + } + let canonical = root.canonicalize().map_err(map_read_error)?; + let path = canonical.join(bucket).join(object); + let parent = path.parent().ok_or(ReadFailure::Invalid)?; + let resolved = parent.canonicalize().map_err(map_read_error)?; + if !resolved.starts_with(&canonical) { + return Err(ReadFailure::Invalid); + } + Ok(path) +} + +struct DriveShard { + index: usize, + file_info: FileInfo, + state: ShardState, +} + +impl DriveShard { + fn retained_bytes(&self) -> u64 { + match &self.state { + ShardState::Verified(bytes) => u64::try_from(bytes.len()).unwrap_or(u64::MAX), + ShardState::Missing | ShardState::Corrupt => 0, + } + } +} + +enum ShardState { + Verified(Vec), + Missing, + Corrupt, +} + +struct ReadBudget { + remaining: u64, +} + +impl ReadBudget { + fn new(limit: u64) -> Self { + Self { remaining: limit } + } + + fn charge(&mut self, amount: u64) -> Result<(), ReadFailure> { + self.remaining = self.remaining.checked_sub(amount).ok_or(ReadFailure::Limit)?; + Ok(()) + } + + fn read(&mut self, path: &Path, ceiling: u64) -> Result, ReadFailure> { + let before = fs::metadata(path).map_err(map_read_error)?; + if before.len() > ceiling { + return Err(ReadFailure::Limit); + } + self.charge(before.len())?; + let mut options = OpenOptions::new(); + options.read(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.custom_flags(libc::O_NOFOLLOW); + } + let file = options.open(path).map_err(map_read_error)?; + let capacity = usize::try_from(before.len()).map_err(|_| ReadFailure::Limit)?; + let mut bytes = Vec::with_capacity(capacity); + file.take(ceiling.saturating_add(1)) + .read_to_end(&mut bytes) + .map_err(map_read_error)?; + let after = fs::metadata(path).map_err(map_read_error)?; + if u64::try_from(bytes.len()).map_err(|_| ReadFailure::Limit)? != before.len() + || before.len() != after.len() + || before.modified().ok() != after.modified().ok() + { + return Err(ReadFailure::Changed); + } + Ok(bytes) + } +} + +#[derive(Clone, Copy)] +enum ReadFailure { + Missing, + PermissionDenied, + Unsupported, + Invalid, + Limit, + Changed, + Cancelled, +} + +fn map_read_error(error: std::io::Error) -> ReadFailure { + match error.kind() { + std::io::ErrorKind::NotFound => ReadFailure::Missing, + std::io::ErrorKind::PermissionDenied => ReadFailure::PermissionDenied, + _ => ReadFailure::Invalid, + } +} + +fn check_read_cancel(cancel: &CancellationToken) -> Result<(), ReadFailure> { + if cancel.is_cancelled() { + Err(ReadFailure::Cancelled) + } else { + Ok(()) + } +} + +fn check_read_bounds(cancel: &CancellationToken, started: Instant, max_duration: Duration) -> Result<(), ReadFailure> { + check_read_cancel(cancel)?; + if started.elapsed() >= max_duration { + Err(ReadFailure::Limit) + } else { + Ok(()) + } +} + +fn check_bounds(cancel: &CancellationToken, started: Instant, max_duration: Duration) -> Result<(), InspectError> { + check_cancel(cancel)?; + if started.elapsed() >= max_duration { + Err(InspectError::LimitExceeded) + } else { + Ok(()) + } +} + +fn success_result(request: &InspectRequest, findings: Vec, started: Instant) -> InspectDiagnosticResult { + InspectDiagnosticResult { + schema_version: INSPECT_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: "inspect.object", + capability: INSPECT_CAPABILITY, + outcome: InspectOutcome::Succeeded, + reason_code: InspectReasonCode::Complete, + duration_millis: duration_millis(started.elapsed()), + provenance: request.provenance.clone(), + coverage: Coverage { + requested_units: u32::try_from(request.rules.len()).unwrap_or(0), + completed_units: u32::try_from(findings.len()).unwrap_or(0), + unit: "CHECK", + }, + data: Some(InspectData { + scope: "LOCAL_OBJECT_SUMMARY", + findings, + }), + } +} + +fn terminal( + request: &InspectRequest, + outcome: InspectOutcome, + reason_code: InspectReasonCode, + started: Instant, +) -> InspectDiagnosticResult { + InspectDiagnosticResult { + schema_version: INSPECT_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: "inspect.object", + capability: INSPECT_CAPABILITY, + outcome, + reason_code, + duration_millis: duration_millis(started.elapsed()), + provenance: request.provenance.clone(), + coverage: Coverage { + requested_units: u32::try_from(request.rules.len()).unwrap_or(0), + completed_units: 0, + unit: "CHECK", + }, + data: None, + } +} + +fn finding( + rule: InspectRule, + outcome: InspectRuleOutcome, + reason: InspectReason, + missing: Option, + corrupt: Option, + metadata_match: Option, + reconstruction: Reconstruction, +) -> InspectFinding { + InspectFinding { + resource_alias: "object-1", + rule_id: rule.into(), + outcome, + reason, + missing_shard_count: missing, + corrupt_shard_count: corrupt, + metadata_match, + reconstruction, + } +} + +fn indeterminate(rule: InspectRule, reason: InspectReason) -> InspectFinding { + finding(rule, InspectRuleOutcome::Indeterminate, reason, None, None, None, Reconstruction::Unknown) +} + +fn sign_result( + request: &InspectRequest, + result: InspectDiagnosticResult, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + if unix_now()? >= request.expires_at_unix { + return Err(InspectError::Expired); + } + let result_json = serde_json::to_vec(&result).map_err(|_| InspectError::Encoding)?; + if result_json.is_empty() || result_json.len() > MAX_RESULT_BYTES { + return Err(InspectError::LimitExceeded); + } + let device_key_id = hex_lower(&Sha256::digest(key.public_key_der())); + let envelope = Envelope { + format_version: "rustfs.connect.diagnosticEnvelope/1", + protocol_version: "v1", + organization_name: &request.organization_name, + cluster_name: &request.cluster_name, + device_name: &request.device_name, + run_uid: &request.run_uid, + artifact_uid: &request.artifact_uid, + tool_id: "inspect.object", + schema_version: INSPECT_SCHEMA_VERSION, + classification: "L3", + consent_uid: &request.consent.consent_uid, + policy_revision: request.consent.policy_revision, + produced_at: timestamp(request.produced_at_unix)?, + expires_at: timestamp(request.expires_at_unix)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(request.nonce), + device_key_id: &device_key_id, + payload: Payload { + path: RESULT_PATH, + media_type: "application/json", + size_bytes: u64::try_from(result_json.len()).map_err(|_| InspectError::LimitExceeded)?, + sha256: hex_lower(&Sha256::digest(&result_json)), + }, + }; + let envelope_json = serde_json::to_vec(&envelope).map_err(|_| InspectError::Encoding)?; + if envelope_json.is_empty() || envelope_json.len() > MAX_ENVELOPE_BYTES { + return Err(InspectError::LimitExceeded); + } + let envelope_signature = signature_document(key, &device_key_id, &envelope_json)?; + check_cancel(cancel)?; + let archive_bytes = archive(&envelope_json, &envelope_signature, &result_json)?; + if archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(InspectError::LimitExceeded); + } + Ok(SignedInspectExport { + artifact_uid: request.artifact_uid.clone(), + envelope_json, + envelope_signature, + result_json, + archive_sha256: hex_lower(&Sha256::digest(&archive_bytes)), + archive_bytes, + }) +} + +pub fn save_signed_inspect_export( + output: &Path, + export: &SignedInspectExport, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + if !uuid7(&export.artifact_uid) { + return Err(InspectError::InvalidRequest); + } + let parent = output + .parent() + .filter(|path| !path.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let filename = output.file_name().ok_or(InspectError::InvalidRequest)?.to_string_lossy(); + let temporary = parent.join(format!(".{filename}.{}.partial", export.artifact_uid)); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + let mut file = options.open(&temporary).map_err(map_create_error)?; + let saved = (|| { + file.write_all(&export.archive_bytes).map_err(InspectError::Io)?; + check_cancel(cancel)?; + file.sync_all().map_err(InspectError::Io)?; + check_cancel(cancel)?; + fs::hard_link(&temporary, output).map_err(map_publish_error)?; + fs::remove_file(&temporary).map_err(InspectError::DurabilityAfterCommit)?; + #[cfg(unix)] + File::open(parent) + .and_then(|directory| directory.sync_all()) + .map_err(InspectError::DurabilityAfterCommit)?; + Ok(SavedInspectExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: u64::try_from(export.archive_bytes.len()).map_err(|_| InspectError::LimitExceeded)?, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if saved.is_err() { + let _ = fs::remove_file(&temporary); + } + saved +} + +impl InspectRequest { + fn validate(&self, now: i64) -> Result<(), InspectError> { + if self.schema_version != INSPECT_SCHEMA_VERSION { + return Err(InspectError::UnsupportedVersion); + } + if self.capability != INSPECT_CAPABILITY { + return Err(InspectError::UnsupportedCapability); + } + if !self.consent.confirmed || self.consent.policy_revision == 0 { + return Err(InspectError::ConsentRequired); + } + if self.consent.expires_at_unix <= now || self.expires_at_unix > self.consent.expires_at_unix { + return Err(InspectError::ConsentExpired); + } + let validity = self + .expires_at_unix + .checked_sub(self.produced_at_unix) + .ok_or(InspectError::Expired)?; + if self.produced_at_unix > now.saturating_add(MAX_FUTURE_SKEW_SECONDS) + || validity <= 0 + || validity > MAX_VALIDITY_SECONDS + || self.expires_at_unix <= now + { + return Err(InspectError::Expired); + } + if self.drive_roots.is_empty() + || self.drive_roots.len() > 256 + || self.rules.is_empty() + || self.rules.len() > MAX_FINDINGS + || self.rules.iter().copied().collect::>().len() != self.rules.len() + || self.max_duration.is_zero() + || self.max_duration > MAX_INSPECT_DURATION + || self.max_read_bytes == 0 + || self.max_read_bytes > MAX_LOCAL_READ_BYTES + || self.max_memory_bytes == 0 + || self.max_memory_bytes > MAX_WORKING_MEMORY_BYTES + { + return Err(InspectError::LimitExceeded); + } + if check_valid_bucket_name_strict(&self.bucket).is_err() + || self.object.is_empty() + || self.object.len() > 1_024 + || Path::new(&self.object).is_absolute() + || Path::new(&self.object) + .components() + .any(|component| !matches!(component, Component::Normal(_))) + || self + .version_id + .as_deref() + .is_some_and(|value| Uuid::parse_str(value).is_err()) + { + return Err(InspectError::InvalidRequest); + } + if !uuid7(&self.run_uid) + || !uuid7(&self.artifact_uid) + || !uuid7(&self.consent.consent_uid) + || !resource_names_match(self) + || !lower_hex(&self.provenance.source_commit, 40) + || !lower_hex(&self.provenance.executable_sha256, 64) + || !version(&self.provenance.rustfs_version) + || self.provenance.build_features.len() > 64 + || !self.provenance.build_features.iter().all(|feature| build_feature(feature)) + { + return Err(InspectError::InvalidRequest); + } + Ok(()) + } +} + +struct CollectorLease; + +impl CollectorLease { + fn acquire() -> Result { + INSPECT_ACTIVE + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map(|_| Self) + .map_err(|_| InspectError::Busy) + } +} + +impl Drop for CollectorLease { + fn drop(&mut self) { + INSPECT_ACTIVE.store(false, Ordering::Release); + } +} + +#[derive(Clone, Debug, Serialize)] +#[serde(rename_all = "camelCase")] +struct Coverage { + requested_units: u32, + completed_units: u32, + unit: &'static str, +} + +#[derive(Clone, Debug, Serialize)] +#[serde(rename_all = "camelCase")] +struct InspectData { + scope: &'static str, + findings: Vec, +} + +#[derive(Clone, Copy, Debug, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum RuleId { + ShardBitrot, + ShardAvailability, + MetadataIdentity, +} + +impl From for RuleId { + fn from(value: InspectRule) -> Self { + match value { + InspectRule::ShardBitrot => Self::ShardBitrot, + InspectRule::ShardAvailability => Self::ShardAvailability, + InspectRule::MetadataIdentity => Self::MetadataIdentity, + } + } +} + +impl From for InspectRule { + fn from(value: RuleId) -> Self { + match value { + RuleId::ShardBitrot => Self::ShardBitrot, + RuleId::ShardAvailability => Self::ShardAvailability, + RuleId::MetadataIdentity => Self::MetadataIdentity, + } + } +} + +#[derive(Clone, Copy, Debug, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum OsFamily { + Linux, + Darwin, + Windows, + Freebsd, + Other, +} + +impl OsFamily { + const fn current() -> Self { + if cfg!(target_os = "linux") { + Self::Linux + } else if cfg!(target_os = "macos") { + Self::Darwin + } else if cfg!(target_os = "windows") { + Self::Windows + } else if cfg!(target_os = "freebsd") { + Self::Freebsd + } else { + Self::Other + } + } +} + +#[derive(Clone, Copy, Debug, Serialize)] +enum Architecture { + #[serde(rename = "x86_64")] + X86_64, + #[serde(rename = "aarch64")] + Aarch64, + #[serde(rename = "other")] + Other, +} + +impl Architecture { + const fn current() -> Self { + if cfg!(target_arch = "x86_64") { + Self::X86_64 + } else if cfg!(target_arch = "aarch64") { + Self::Aarch64 + } else { + Self::Other + } + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct Envelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: &'static str, + schema_version: u16, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: &'a str, + payload: Payload, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct Payload { + path: &'static str, + media_type: &'static str, + size_bytes: u64, + sha256: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct SignatureDocument<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +fn signature_document(key: &DeviceIdentity, key_id: &str, envelope: &[u8]) -> Result, InspectError> { + let pkcs8 = key.to_pkcs8_der().map_err(|_| InspectError::Signing)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| InspectError::Signing)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + envelope.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.extend_from_slice(envelope); + let signature: Signature = signing_key.sign(&input); + serde_json::to_vec(&SignatureDocument { + algorithm: "ES256", + key_id, + value: URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()), + }) + .map_err(|_| InspectError::Encoding) +} + +fn archive(envelope: &[u8], signature: &[u8], result: &[u8]) -> Result, InspectError> { + let cursor = Cursor::new(Vec::with_capacity(envelope.len() + signature.len() + result.len() + 512)); + let mut writer = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(OUTPUT_MODE); + for (name, bytes) in [(ENVELOPE_PATH, envelope), (SIGNATURE_PATH, signature), (RESULT_PATH, result)] { + writer.start_file(name, options).map_err(|_| InspectError::Encoding)?; + writer.write_all(bytes).map_err(InspectError::Io)?; + } + writer + .finish() + .map(|cursor| cursor.into_inner()) + .map_err(|_| InspectError::Encoding) +} + +fn map_create_error(error: std::io::Error) -> InspectError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + InspectError::AlreadyExists + } else { + InspectError::Io(error) + } +} + +fn map_publish_error(error: std::io::Error) -> InspectError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + InspectError::AlreadyExists + } else { + InspectError::Io(error) + } +} + +fn resource_names_match(request: &InspectRequest) -> bool { + let Some(organization_uid) = request.organization_name.strip_prefix("organizations/") else { + return false; + }; + let cluster_prefix = format!("{}/clusters/", request.organization_name); + let Some(cluster_uid) = request.cluster_name.strip_prefix(&cluster_prefix) else { return false }; + let device_prefix = format!("{}/clusterDevices/", request.cluster_name); + let Some(device_uid) = request.device_name.strip_prefix(&device_prefix) else { return false }; + uuid7(organization_uid) && uuid7(cluster_uid) && uuid7(device_uid) +} + +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn lower_hex(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn version(value: &str) -> bool { + if value.is_empty() + || value.len() > 64 + || !value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'-')) + { + return false; + } + let (core, suffix) = value + .split_once('-') + .map_or((value, None), |(core, suffix)| (core, Some(suffix))); + if suffix.is_some_and(str::is_empty) { + return false; + } + let mut parts = core.split('.'); + parts.clone().count() == 3 && parts.all(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) +} + +fn build_feature(value: &str) -> bool { + !value.is_empty() + && value.len() <= 64 + && value.as_bytes()[0].is_ascii_lowercase() + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'_' | b'-')) +} + +fn timestamp(unix: i64) -> Result { + OffsetDateTime::from_unix_timestamp(unix) + .map_err(|_| InspectError::InvalidRequest)? + .format(&Rfc3339) + .map_err(|_| InspectError::InvalidRequest) +} + +fn unix_now() -> Result { + let duration = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| InspectError::InvalidRequest)?; + i64::try_from(duration.as_secs()).map_err(|_| InspectError::InvalidRequest) +} + +fn check_cancel(cancel: &CancellationToken) -> Result<(), InspectError> { + if cancel.is_cancelled() { + Err(InspectError::Cancelled) + } else { + Ok(()) + } +} + +fn duration_millis(duration: Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).min(30_000) +} + +fn hex_lower(bytes: &[u8]) -> String { + let mut value = String::with_capacity(bytes.len() * 2); + for byte in bytes { + use std::fmt::Write as _; + write!(&mut value, "{byte:02x}").expect("writing hexadecimal to a string cannot fail"); + } + value +} diff --git a/rustfs/src/connect/diagnostics/job.rs b/rustfs/src/connect/diagnostics/job.rs new file mode 100644 index 000000000..62466dd3c --- /dev/null +++ b/rustfs/src/connect/diagnostics/job.rs @@ -0,0 +1,1697 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Verification and execution of the small allow-list of Connect diagnostic jobs. + +use std::time::Duration; +use std::{fs, io::Read as _, path::Path}; + +#[cfg(unix)] +use std::os::unix::fs::{DirBuilderExt as _, MetadataExt as _, OpenOptionsExt as _, PermissionsExt as _}; + +use base64_simd::URL_SAFE_NO_PAD; +use chrono::{DateTime, Utc}; +use ed25519_dalek::{Signature, Verifier as _, VerifyingKey}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; + +use super::{ + CPU_PROFILE_CAPABILITY, DRIVE_CAPABILITY, DRIVE_SCHEMA_VERSION, DriveOutcome, DrivePerformanceError, DrivePerformanceRequest, + DriveProvenance, LocalDriveConsent, LocalNetworkConsent, LocalProfileConsent, LocalTopConsent, MAX_NETWORK_TRAFFIC_BYTES, + MAX_TOP_EXPORT_VALIDITY, NETWORK_CAPABILITY, NETWORK_SCHEMA_VERSION, NetworkOutcome, NetworkPerformanceError, + NetworkPerformanceRequest, NetworkProvenance, NetworkReasonCode, PROFILE_SCHEMA_VERSION, ProfileCaptureRequest, + ProfileOutcome, ProfileProvenance, THREAD_PROFILE_CAPABILITY, TOP_API_CAPABILITY, TOP_CLASSIFICATION, TOP_LOCKS_CAPABILITY, + TOP_RPC_CAPABILITY, TOP_SCHEMA_VERSION, ThreadProfileScope, TopApiOperation, TopCaptureLimits, TopCaptureRequest, + TopCaptureScope, TopOutcome, capture_cpu_profile, capture_thread_profile, capture_top_api, capture_top_locks, + capture_top_rpc, encode_signed_profile_export, measure_drive, measure_network, runtime_network_peer_aliases, + sign_drive_export, sign_network_export, sign_top_export_with_nonce, +}; +use crate::connect::DeviceIdentity; + +const PROTOCOL_VERSION: &str = "v1"; +const PROFILE_CPU_JOB_TYPE: &str = "profile.cpu"; +const PROFILE_THREADS_JOB_TYPE: &str = "profile.threads"; +const PERFORMANCE_DRIVE_JOB_TYPE: &str = "performance.drive"; +const PERFORMANCE_NETWORK_JOB_TYPE: &str = "performance.network"; +const TOP_API_JOB_TYPE: &str = "top.api"; +const TOP_LOCKS_JOB_TYPE: &str = "top.locks"; +const TOP_RPC_JOB_TYPE: &str = "top.rpc"; +pub const DIAGNOSTIC_JOB_SIGNATURE_DOMAIN: &[u8] = b"rustfs-connect-agent-job-v1\0"; +const MAX_JOB_LIFETIME_SECONDS: i64 = 1_800; +const MAX_FUTURE_SKEW_SECONDS: i64 = 300; +const MAX_OUTPUT_BYTES: u64 = 524_288; +const MAX_MEMORY_BYTES: u64 = 64 * 1024 * 1024; +const MAX_CPU_MILLIS: u64 = 30_000; +const MAX_NETWORK_CPU_MILLIS: u64 = 5_000; +const MIN_NETWORK_MEMORY_BYTES: u64 = 1_048_576; +const DRIVE_TARGET_ALIAS: &str = "drive-1"; +const DRIVE_DURATION_MILLIS: u64 = 5_000; +const DRIVE_SCRATCH_BYTES: u64 = 524_288; +const DRIVE_BLOCK_BYTES: u64 = 65_536; +const DRIVE_SAMPLE_PERIOD_MICROS: u64 = 10_000; +const DRIVE_SCRATCH_DIRECTORY: &str = ".rustfs-connect-drive-scratch"; +const MAX_TOP_API_CPU_MILLIS: u64 = 5_000; +const MIN_TOP_API_MEMORY_BYTES: u64 = 1_048_576; +const MAX_TOP_LOCKS_CPU_MILLIS: u64 = 5_000; +const MIN_TOP_LOCKS_MEMORY_BYTES: u64 = 1_048_576; +const MAX_TOP_RPC_CPU_MILLIS: u64 = 5_000; +const MIN_TOP_RPC_MEMORY_BYTES: u64 = 1_048_576; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum DiagnosticJobKind { + ProfileCpu, + ProfileThreads, + PerformanceDrive, + PerformanceNetwork, + TopApi, + TopLocks, + TopRpc, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct DiagnosticJobTarget { + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, +} + +#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct DiagnosticJobLimits { + pub timeout_seconds: u64, + pub max_output_bytes: u64, + pub max_memory_bytes: u64, + pub max_cpu_millis: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct DiagnosticJobParameters { + pub artifact_uid: String, + pub consent_uid: String, + pub consent_policy_revision: u64, + pub consent_expires_at: String, + pub duration_millis: u64, + pub sample_period_micros: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub traffic_bytes: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub target_alias: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub scratch_bytes: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub block_bytes: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct DiagnosticJobSignature { + algorithm: String, + key_id: String, + value: String, +} + +#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct DiagnosticJobAuthorization { + actor_type: String, + actor_name: String, + request_id: String, +} + +#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct DiagnosticJobEnvelope { + pub job_id: String, + pub protocol_version: String, + pub job_type: String, + pub schema_version: u16, + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub create_time: String, + pub expire_time: String, + pub nonce: String, + pub required_capabilities: Vec, + pub authorization: DiagnosticJobAuthorization, + pub limits: DiagnosticJobLimits, + pub parameters: DiagnosticJobParameters, + pub signature: DiagnosticJobSignature, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct UnsignedDiagnosticJob<'a> { + job_id: &'a str, + protocol_version: &'a str, + job_type: &'a str, + schema_version: u16, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + create_time: &'a str, + expire_time: &'a str, + nonce: &'a str, + required_capabilities: &'a [String], + authorization: &'a DiagnosticJobAuthorization, + limits: &'a DiagnosticJobLimits, + parameters: &'a DiagnosticJobParameters, +} + +#[derive(Clone, Debug)] +pub struct TrustedDiagnosticJobSigner { + key_id: String, + key: VerifyingKey, +} + +impl TrustedDiagnosticJobSigner { + pub fn new(key_id: String, public_key: [u8; 32]) -> Result { + if !lower_hex(&key_id, 64) || hex_lower(&Sha256::digest(public_key)) != key_id { + return Err(DiagnosticJobError::TrustInvalid); + } + let key = VerifyingKey::from_bytes(&public_key).map_err(|_| DiagnosticJobError::TrustInvalid)?; + Ok(Self { key_id, key }) + } + + pub fn from_public_key_file(path: &Path, key_id: String) -> Result { + #[cfg(not(unix))] + { + let _ = (path, key_id); + return Err(DiagnosticJobError::TrustInvalid); + } + #[cfg(unix)] + { + let initial = fs::symlink_metadata(path).map_err(|_| DiagnosticJobError::TrustInvalid)?; + if !initial.file_type().is_file() || initial.permissions().mode() & 0o077 != 0 || initial.len() > 256 { + return Err(DiagnosticJobError::TrustInvalid); + } + let mut options = fs::OpenOptions::new(); + options.read(true).custom_flags(libc::O_NOFOLLOW | libc::O_CLOEXEC); + let file = options.open(path).map_err(|_| DiagnosticJobError::TrustInvalid)?; + let opened = file.metadata().map_err(|_| DiagnosticJobError::TrustInvalid)?; + if !opened.is_file() + || opened.uid() != rustix::process::geteuid().as_raw() + || opened.dev() != initial.dev() + || opened.ino() != initial.ino() + { + return Err(DiagnosticJobError::TrustInvalid); + } + let mut encoded = Vec::new(); + file.take(257) + .read_to_end(&mut encoded) + .map_err(|_| DiagnosticJobError::TrustInvalid)?; + if encoded.len() > 256 { + return Err(DiagnosticJobError::TrustInvalid); + } + let encoded = std::str::from_utf8(&encoded) + .map_err(|_| DiagnosticJobError::TrustInvalid)? + .trim(); + let public_key = URL_SAFE_NO_PAD + .decode_to_vec(encoded.as_bytes()) + .map_err(|_| DiagnosticJobError::TrustInvalid)?; + if URL_SAFE_NO_PAD.encode_to_string(&public_key) != encoded { + return Err(DiagnosticJobError::TrustInvalid); + } + Self::new(key_id, public_key.try_into().map_err(|_| DiagnosticJobError::TrustInvalid)?) + } + } + + pub fn verify( + &self, + envelope: &DiagnosticJobEnvelope, + target: &DiagnosticJobTarget, + now: DateTime, + ) -> Result { + envelope.validate(target, now)?; + if envelope.signature.algorithm != "Ed25519" || envelope.signature.key_id != self.key_id { + return Err(DiagnosticJobError::SignerUntrusted); + } + let encoded = URL_SAFE_NO_PAD + .decode_to_vec(envelope.signature.value.as_bytes()) + .map_err(|_| DiagnosticJobError::SignatureInvalid)?; + let signature = Signature::from_slice(&encoded).map_err(|_| DiagnosticJobError::SignatureInvalid)?; + let payload = envelope.signing_payload()?; + self.key + .verify(&payload, &signature) + .map_err(|_| DiagnosticJobError::SignatureInvalid)?; + let nonce = URL_SAFE_NO_PAD + .decode_to_vec(envelope.nonce.as_bytes()) + .map_err(|_| DiagnosticJobError::Invalid)? + .try_into() + .map_err(|_| DiagnosticJobError::Invalid)?; + Ok(VerifiedDiagnosticJob { + envelope: envelope.clone(), + nonce, + }) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct VerifiedDiagnosticJob { + envelope: DiagnosticJobEnvelope, + nonce: [u8; 32], +} + +impl VerifiedDiagnosticJob { + pub fn job_id(&self) -> &str { + &self.envelope.job_id + } + + pub(crate) fn expire_time(&self) -> &str { + &self.envelope.expire_time + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Deserialize, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct DiagnosticJobExecution { + pub job_id: String, + pub outcome: String, + pub reason: String, + pub artifact_uid: Option, + pub artifact_sha256: Option, + pub artifact_bytes: Option>, +} + +#[derive(Debug, Error, PartialEq, Eq)] +pub enum DiagnosticJobError { + #[error("connect_diagnostic_job_invalid")] + Invalid, + #[error("connect_diagnostic_job_target_mismatch")] + TargetMismatch, + #[error("connect_diagnostic_job_expired")] + Expired, + #[error("connect_diagnostic_job_unsupported")] + Unsupported, + #[error("connect_diagnostic_job_limit_exceeded")] + LimitExceeded, + #[error("connect_diagnostic_job_trust_invalid")] + TrustInvalid, + #[error("connect_diagnostic_job_signer_untrusted")] + SignerUntrusted, + #[error("connect_diagnostic_job_signature_invalid")] + SignatureInvalid, + #[error("connect_diagnostic_job_encoding_failed")] + Encoding, + #[error("connect_diagnostic_job_cancelled")] + Cancelled, + #[error("connect_diagnostic_job_collection_failed")] + CollectionFailed, + #[error("connect_diagnostic_job_profile_source_unavailable")] + ProfileSourceUnavailable, + #[error("connect_diagnostic_job_export_failed")] + ExportFailed, +} + +impl DiagnosticJobError { + pub const fn reason(&self) -> &'static str { + match self { + Self::Invalid => "INVALID", + Self::TargetMismatch => "TARGET_MISMATCH", + Self::Expired => "EXPIRED", + Self::Unsupported => "UNSUPPORTED", + Self::LimitExceeded => "LIMIT_EXCEEDED", + Self::TrustInvalid => "TRUST_INVALID", + Self::SignerUntrusted => "SIGNER_UNTRUSTED", + Self::SignatureInvalid => "SIGNATURE_INVALID", + Self::Encoding => "ENCODING_FAILED", + Self::Cancelled => "CANCELLED", + Self::CollectionFailed => "COLLECTION_FAILED", + Self::ProfileSourceUnavailable => "PROFILE_SOURCE_UNAVAILABLE", + Self::ExportFailed => "EXPORT_FAILED", + } + } +} + +impl DiagnosticJobEnvelope { + fn unsigned(&self) -> UnsignedDiagnosticJob<'_> { + UnsignedDiagnosticJob { + job_id: &self.job_id, + protocol_version: &self.protocol_version, + job_type: &self.job_type, + schema_version: self.schema_version, + organization_name: &self.organization_name, + cluster_name: &self.cluster_name, + device_name: &self.device_name, + create_time: &self.create_time, + expire_time: &self.expire_time, + nonce: &self.nonce, + required_capabilities: &self.required_capabilities, + authorization: &self.authorization, + limits: &self.limits, + parameters: &self.parameters, + } + } + + pub fn signing_payload(&self) -> Result, DiagnosticJobError> { + let payload = serde_json::to_vec(&self.unsigned()).map_err(|_| DiagnosticJobError::Encoding)?; + let mut signed = Vec::with_capacity(DIAGNOSTIC_JOB_SIGNATURE_DOMAIN.len() + payload.len()); + signed.extend_from_slice(DIAGNOSTIC_JOB_SIGNATURE_DOMAIN); + signed.extend_from_slice(&payload); + Ok(signed) + } + + fn validate(&self, target: &DiagnosticJobTarget, now: DateTime) -> Result<(), DiagnosticJobError> { + let kind = self.kind()?; + if self.protocol_version != PROTOCOL_VERSION || self.schema_version != PROFILE_SCHEMA_VERSION { + return Err(DiagnosticJobError::Unsupported); + } + if self.organization_name != target.organization_name + || self.cluster_name != target.cluster_name + || self.device_name != target.device_name + { + return Err(DiagnosticJobError::TargetMismatch); + } + if !uuid7(&self.job_id) + || !uuid7(&self.parameters.artifact_uid) + || !uuid7(&self.parameters.consent_uid) + || self.authorization.actor_type != "BROWSER_USER" + || !self.authorization.actor_name.strip_prefix("users/").is_some_and(uuid7) + || !Uuid::parse_str(&self.authorization.request_id) + .is_ok_and(|value| value.get_version_num() == 4 && value.to_string() == self.authorization.request_id) + || self.parameters.consent_policy_revision == 0 + { + return Err(DiagnosticJobError::Invalid); + } + let create = parse_time(&self.create_time)?; + let expire = parse_time(&self.expire_time)?; + let consent_expire = parse_time(&self.parameters.consent_expires_at)?; + if create > now + chrono::Duration::seconds(MAX_FUTURE_SKEW_SECONDS) + || expire <= now + || expire <= create + || expire > create + chrono::Duration::seconds(MAX_JOB_LIFETIME_SECONDS) + || consent_expire < expire + { + return Err(DiagnosticJobError::Expired); + } + let nonce = URL_SAFE_NO_PAD + .decode_to_vec(self.nonce.as_bytes()) + .map_err(|_| DiagnosticJobError::Invalid)?; + if nonce.len() != 32 + || URL_SAFE_NO_PAD.encode_to_string(&nonce) != self.nonce + || self.limits.timeout_seconds == 0 + || self.limits.timeout_seconds > 30 + || self.limits.max_output_bytes == 0 + || self.limits.max_output_bytes > MAX_OUTPUT_BYTES + || self.limits.max_memory_bytes == 0 + || self.limits.max_memory_bytes > MAX_MEMORY_BYTES + || self.limits.max_cpu_millis == 0 + || self.limits.max_cpu_millis > MAX_CPU_MILLIS + || self.parameters.duration_millis == 0 + || self.parameters.duration_millis > self.limits.timeout_seconds.saturating_mul(1_000) + || self.parameters.duration_millis > self.limits.max_cpu_millis + || self.parameters.sample_period_micros == 0 + || self.parameters.sample_period_micros > self.parameters.duration_millis.saturating_mul(1_000) + { + return Err(DiagnosticJobError::LimitExceeded); + } + if kind == DiagnosticJobKind::TopApi + && (self.limits.max_cpu_millis > MAX_TOP_API_CPU_MILLIS || self.limits.max_memory_bytes < MIN_TOP_API_MEMORY_BYTES) + { + return Err(DiagnosticJobError::LimitExceeded); + } + match (kind, self.parameters.traffic_bytes) { + (DiagnosticJobKind::PerformanceNetwork, Some(1..=MAX_NETWORK_TRAFFIC_BYTES)) => { + if self.limits.max_cpu_millis > MAX_NETWORK_CPU_MILLIS || self.limits.max_memory_bytes < MIN_NETWORK_MEMORY_BYTES + { + return Err(DiagnosticJobError::LimitExceeded); + } + } + (DiagnosticJobKind::PerformanceNetwork, _) => return Err(DiagnosticJobError::LimitExceeded), + (_, None) => {} + (_, Some(_)) => return Err(DiagnosticJobError::Invalid), + } + let has_drive_parameters = self.parameters.target_alias.is_some() + || self.parameters.scratch_bytes.is_some() + || self.parameters.block_bytes.is_some(); + if kind == DiagnosticJobKind::PerformanceDrive { + if self.parameters.target_alias.as_deref() != Some(DRIVE_TARGET_ALIAS) { + return Err(DiagnosticJobError::Invalid); + } + if self.parameters.scratch_bytes != Some(DRIVE_SCRATCH_BYTES) + || self.parameters.block_bytes != Some(DRIVE_BLOCK_BYTES) + || self.parameters.duration_millis != DRIVE_DURATION_MILLIS + || self.parameters.sample_period_micros != DRIVE_SAMPLE_PERIOD_MICROS + || self.limits.timeout_seconds != 30 + || self.limits.max_output_bytes != MAX_OUTPUT_BYTES + || self.limits.max_memory_bytes != MAX_MEMORY_BYTES + || self.limits.max_cpu_millis != DRIVE_DURATION_MILLIS + { + return Err(DiagnosticJobError::LimitExceeded); + } + } else if has_drive_parameters { + return Err(DiagnosticJobError::Invalid); + } + if kind == DiagnosticJobKind::TopLocks + && (self.limits.max_cpu_millis > MAX_TOP_LOCKS_CPU_MILLIS + || self.limits.max_memory_bytes < MIN_TOP_LOCKS_MEMORY_BYTES) + { + return Err(DiagnosticJobError::LimitExceeded); + } + if kind == DiagnosticJobKind::TopRpc + && (self.limits.max_cpu_millis > MAX_TOP_RPC_CPU_MILLIS || self.limits.max_memory_bytes < MIN_TOP_RPC_MEMORY_BYTES) + { + return Err(DiagnosticJobError::LimitExceeded); + } + Ok(()) + } + + fn kind(&self) -> Result { + match (self.job_type.as_str(), self.required_capabilities.as_slice(), self.schema_version) { + (PROFILE_CPU_JOB_TYPE, [capability], PROFILE_SCHEMA_VERSION) if capability == CPU_PROFILE_CAPABILITY => { + Ok(DiagnosticJobKind::ProfileCpu) + } + (PROFILE_THREADS_JOB_TYPE, [capability], PROFILE_SCHEMA_VERSION) if capability == THREAD_PROFILE_CAPABILITY => { + Ok(DiagnosticJobKind::ProfileThreads) + } + (PERFORMANCE_DRIVE_JOB_TYPE, [capability], DRIVE_SCHEMA_VERSION) if capability == DRIVE_CAPABILITY => { + Ok(DiagnosticJobKind::PerformanceDrive) + } + (PERFORMANCE_NETWORK_JOB_TYPE, [capability], NETWORK_SCHEMA_VERSION) if capability == NETWORK_CAPABILITY => { + Ok(DiagnosticJobKind::PerformanceNetwork) + } + (TOP_API_JOB_TYPE, [capability], version) + if capability == TOP_API_CAPABILITY && version == u16::from(TOP_SCHEMA_VERSION) => + { + Ok(DiagnosticJobKind::TopApi) + } + (TOP_LOCKS_JOB_TYPE, [capability], version) + if capability == TOP_LOCKS_CAPABILITY && version == u16::from(TOP_SCHEMA_VERSION) => + { + Ok(DiagnosticJobKind::TopLocks) + } + (TOP_RPC_JOB_TYPE, [capability], version) + if capability == TOP_RPC_CAPABILITY && version == u16::from(TOP_SCHEMA_VERSION) => + { + Ok(DiagnosticJobKind::TopRpc) + } + _ => Err(DiagnosticJobError::Unsupported), + } + } +} + +pub async fn execute_diagnostic_job( + job: VerifiedDiagnosticJob, + identity: &DeviceIdentity, + provenance: ProfileProvenance, + cancel: &CancellationToken, +) -> Result { + if cancel.is_cancelled() { + return Err(DiagnosticJobError::Cancelled); + } + let nonce = job.nonce; + let envelope = job.envelope; + match envelope.kind()? { + DiagnosticJobKind::ProfileCpu => execute_profile_cpu_job(envelope, nonce, identity, provenance, cancel).await, + DiagnosticJobKind::ProfileThreads => execute_profile_threads_job(envelope, nonce, identity, provenance, cancel).await, + DiagnosticJobKind::PerformanceDrive => { + let Some(scratch_root) = runtime_drive_scratch_root() else { + return Ok(failed_drive_execution(&envelope.job_id, "SOURCE_UNAVAILABLE")); + }; + if !ensure_drive_scratch_root(&scratch_root) { + return Ok(failed_drive_execution(&envelope.job_id, "SOURCE_UNAVAILABLE")); + } + execute_performance_drive_job(envelope, nonce, identity, provenance, &scratch_root, cancel).await + } + DiagnosticJobKind::PerformanceNetwork => { + execute_performance_network_job(envelope, nonce, identity, provenance, cancel).await + } + DiagnosticJobKind::TopApi => execute_top_api_job(envelope, nonce, identity, provenance, cancel).await, + DiagnosticJobKind::TopLocks => execute_top_locks_job(envelope, nonce, identity, provenance, cancel).await, + DiagnosticJobKind::TopRpc => execute_top_rpc_job(envelope, nonce, identity, provenance, cancel).await, + } +} + +fn runtime_drive_scratch_root() -> Option { + let endpoint_pools = crate::runtime_sources::current_endpoints_handle()?; + drive_scratch_root_from_endpoints(&endpoint_pools) +} + +fn drive_scratch_root_from_endpoints( + endpoint_pools: &crate::storage_api::cluster::EndpointServerPools, +) -> Option { + endpoint_pools + .as_ref() + .iter() + .flat_map(|pool| pool.endpoints.as_ref()) + .find(|endpoint| endpoint.is_local) + .map(|endpoint| std::path::PathBuf::from(endpoint.get_file_path()).join(DRIVE_SCRATCH_DIRECTORY)) +} + +fn ensure_drive_scratch_root(path: &Path) -> bool { + let created = if path.exists() { + true + } else { + let mut builder = fs::DirBuilder::new(); + #[cfg(unix)] + builder.mode(0o700); + builder.create(path).is_ok() + }; + if !created { + return false; + } + let Ok(metadata) = fs::symlink_metadata(path) else { + return false; + }; + if metadata.file_type().is_symlink() || !metadata.is_dir() { + return false; + } + #[cfg(unix)] + if metadata.uid() != rustix::process::geteuid().as_raw() || metadata.permissions().mode() & 0o077 != 0 { + return false; + } + true +} + +fn failed_drive_execution(job_id: &str, reason: &str) -> DiagnosticJobExecution { + DiagnosticJobExecution { + job_id: job_id.to_owned(), + outcome: "FAILED".to_owned(), + reason: reason.to_owned(), + artifact_uid: None, + artifact_sha256: None, + artifact_bytes: None, + } +} + +async fn execute_performance_drive_job( + envelope: DiagnosticJobEnvelope, + nonce: [u8; 32], + identity: &DeviceIdentity, + provenance: ProfileProvenance, + scratch_root: &Path, + cancel: &CancellationToken, +) -> Result { + let expire = parse_time(&envelope.expire_time)?; + let consent_expire = parse_time(&envelope.parameters.consent_expires_at)?; + let request = DrivePerformanceRequest { + organization_name: envelope.organization_name, + cluster_name: envelope.cluster_name, + device_name: envelope.device_name, + run_uid: envelope.job_id.clone(), + artifact_uid: envelope.parameters.artifact_uid, + schema_version: envelope.schema_version, + capability: DRIVE_CAPABILITY.to_owned(), + consent: LocalDriveConsent { + consent_uid: envelope.parameters.consent_uid, + policy_revision: envelope.parameters.consent_policy_revision, + expires_at_unix: consent_expire.timestamp(), + confirmed: true, + }, + produced_at_unix: Utc::now().timestamp(), + expires_at_unix: expire.timestamp(), + nonce, + duration: Duration::from_millis(envelope.parameters.duration_millis), + target_alias: DRIVE_TARGET_ALIAS.to_owned(), + scratch_root: scratch_root.to_path_buf(), + scratch_bytes: envelope.parameters.scratch_bytes.ok_or(DiagnosticJobError::LimitExceeded)?, + block_bytes: envelope.parameters.block_bytes.ok_or(DiagnosticJobError::LimitExceeded)?, + provenance: DriveProvenance::new( + provenance.source_commit(), + provenance.executable_sha256(), + provenance.rustfs_version(), + provenance.build_features().to_vec(), + ), + }; + let measurement = measure_drive(&request, cancel).await.map_err(drive_capture_failure)?; + let outcome = measurement.result.outcome(); + let reason = measurement.result.reason_code(); + if outcome != DriveOutcome::Succeeded { + return Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome: outcome.as_str().to_owned(), + reason: reason.as_str().to_owned(), + artifact_uid: None, + artifact_sha256: None, + artifact_bytes: None, + }); + } + let export = sign_drive_export(&request, &measurement, identity, cancel).map_err(drive_export_failure)?; + if export.archive_bytes.len() > usize::try_from(envelope.limits.max_output_bytes).unwrap_or(usize::MAX) { + return Err(DiagnosticJobError::LimitExceeded); + } + Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome: outcome.as_str().to_owned(), + reason: reason.as_str().to_owned(), + artifact_uid: Some(export.artifact_uid), + artifact_sha256: Some(export.archive_sha256), + artifact_bytes: Some(export.archive_bytes), + }) +} + +async fn execute_performance_network_job( + envelope: DiagnosticJobEnvelope, + nonce: [u8; 32], + identity: &DeviceIdentity, + provenance: ProfileProvenance, + cancel: &CancellationToken, +) -> Result { + let Some(peer_aliases) = runtime_network_peer_aliases() else { + return Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome: "FAILED".to_owned(), + reason: "SOURCE_UNAVAILABLE".to_owned(), + artifact_uid: None, + artifact_sha256: None, + artifact_bytes: None, + }); + }; + let traffic_bytes = envelope.parameters.traffic_bytes.ok_or(DiagnosticJobError::LimitExceeded)?; + let peer_count = u64::try_from(peer_aliases.len()).map_err(|_| DiagnosticJobError::LimitExceeded)?; + let traffic_bytes_per_peer = traffic_bytes + .checked_div(peer_count) + .filter(|value| *value > 0) + .ok_or(DiagnosticJobError::LimitExceeded)?; + let expire = parse_time(&envelope.expire_time)?; + let consent_expire = parse_time(&envelope.parameters.consent_expires_at)?; + let request = NetworkPerformanceRequest { + organization_name: envelope.organization_name, + cluster_name: envelope.cluster_name, + device_name: envelope.device_name, + run_uid: envelope.job_id.clone(), + artifact_uid: envelope.parameters.artifact_uid, + schema_version: envelope.schema_version, + capability: NETWORK_CAPABILITY.to_owned(), + consent: LocalNetworkConsent { + consent_uid: envelope.parameters.consent_uid, + policy_revision: envelope.parameters.consent_policy_revision, + expires_at_unix: consent_expire.timestamp(), + confirmed: true, + }, + produced_at_unix: Utc::now().timestamp(), + expires_at_unix: expire.timestamp(), + nonce, + duration: Duration::from_millis(envelope.parameters.duration_millis), + peer_aliases, + traffic_bytes_per_peer, + provenance: NetworkProvenance::new( + provenance.source_commit(), + provenance.executable_sha256(), + provenance.rustfs_version(), + provenance.build_features().to_vec(), + ), + }; + let measurement = measure_network(&request, cancel).await.map_err(network_capture_failure)?; + let measured_outcome = measurement.result.outcome(); + let measured_reason = measurement.result.reason_code(); + let outcome = measured_outcome.as_str().to_owned(); + let reason = measured_reason.as_str().to_owned(); + if !matches!(measured_outcome, NetworkOutcome::Succeeded | NetworkOutcome::Partial) { + let (outcome, reason) = match (measured_outcome, measured_reason) { + (NetworkOutcome::Unsupported, NetworkReasonCode::SourceUnavailable) => { + ("FAILED".to_owned(), "SOURCE_UNAVAILABLE".to_owned()) + } + (NetworkOutcome::Unsupported, _) => ("FAILED".to_owned(), "COLLECTION_FAILED".to_owned()), + _ => (outcome, reason), + }; + return Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome, + reason, + artifact_uid: None, + artifact_sha256: None, + artifact_bytes: None, + }); + } + let export = sign_network_export(&request, &measurement, identity, cancel).map_err(network_export_failure)?; + if export.archive_bytes.len() > usize::try_from(envelope.limits.max_output_bytes).unwrap_or(usize::MAX) { + return Err(DiagnosticJobError::LimitExceeded); + } + Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome, + reason, + artifact_uid: Some(export.artifact_uid), + artifact_sha256: Some(export.archive_sha256), + artifact_bytes: Some(export.archive_bytes), + }) +} + +async fn execute_profile_threads_job( + envelope: DiagnosticJobEnvelope, + nonce: [u8; 32], + identity: &DeviceIdentity, + provenance: ProfileProvenance, + cancel: &CancellationToken, +) -> Result { + let expire = parse_time(&envelope.expire_time)?; + let consent_expire = parse_time(&envelope.parameters.consent_expires_at)?; + let request = ProfileCaptureRequest { + organization_name: envelope.organization_name, + cluster_name: envelope.cluster_name, + device_name: envelope.device_name, + run_uid: envelope.job_id.clone(), + artifact_uid: envelope.parameters.artifact_uid.clone(), + schema_version: envelope.schema_version, + capability: THREAD_PROFILE_CAPABILITY.to_owned(), + consent: LocalProfileConsent { + consent_uid: envelope.parameters.consent_uid, + policy_revision: envelope.parameters.consent_policy_revision, + expires_at_unix: consent_expire.timestamp(), + confirmed: true, + }, + produced_at_unix: Utc::now().timestamp(), + expires_at_unix: expire.timestamp(), + nonce, + duration: Duration::from_millis(envelope.parameters.duration_millis), + sample_period: Duration::from_micros(envelope.parameters.sample_period_micros), + provenance, + }; + let owned_request = request.clone(); + let owned_cancel = cancel.clone(); + let result = tokio::task::spawn_blocking(move || { + capture_thread_profile(&owned_request, ThreadProfileScope::NativeThreads, &owned_cancel) + }) + .await + .map_err(|_| DiagnosticJobError::CollectionFailed)? + .map_err(capture_failure)?; + if result.outcome() == ProfileOutcome::Unsupported { + return Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome: result.outcome().as_str().to_owned(), + reason: result.reason_code().as_str().to_owned(), + artifact_uid: None, + artifact_sha256: None, + artifact_bytes: None, + }); + } + let export = encode_signed_profile_export(&request, &result, identity, cancel).map_err(export_failure)?; + if export.archive_bytes.len() > usize::try_from(envelope.limits.max_output_bytes).unwrap_or(usize::MAX) { + return Err(DiagnosticJobError::LimitExceeded); + } + Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome: result.outcome().as_str().to_owned(), + reason: result.reason_code().as_str().to_owned(), + artifact_uid: Some(export.artifact_uid), + artifact_sha256: Some(export.archive_sha256), + artifact_bytes: Some(export.archive_bytes), + }) +} + +async fn execute_profile_cpu_job( + envelope: DiagnosticJobEnvelope, + nonce: [u8; 32], + identity: &DeviceIdentity, + provenance: ProfileProvenance, + cancel: &CancellationToken, +) -> Result { + let expire = parse_time(&envelope.expire_time)?; + let consent_expire = parse_time(&envelope.parameters.consent_expires_at)?; + let request = ProfileCaptureRequest { + organization_name: envelope.organization_name, + cluster_name: envelope.cluster_name, + device_name: envelope.device_name, + run_uid: envelope.job_id.clone(), + artifact_uid: envelope.parameters.artifact_uid.clone(), + schema_version: envelope.schema_version, + capability: CPU_PROFILE_CAPABILITY.to_owned(), + consent: LocalProfileConsent { + consent_uid: envelope.parameters.consent_uid, + policy_revision: envelope.parameters.consent_policy_revision, + expires_at_unix: consent_expire.timestamp(), + confirmed: true, + }, + produced_at_unix: Utc::now().timestamp(), + expires_at_unix: expire.timestamp(), + nonce, + duration: Duration::from_millis(envelope.parameters.duration_millis), + sample_period: Duration::from_micros(envelope.parameters.sample_period_micros), + provenance, + }; + let result = capture_cpu_profile(&request, cancel).await.map_err(capture_failure)?; + let export = encode_signed_profile_export(&request, &result, identity, cancel).map_err(export_failure)?; + if export.archive_bytes.len() > usize::try_from(envelope.limits.max_output_bytes).unwrap_or(usize::MAX) { + return Err(DiagnosticJobError::LimitExceeded); + } + let outcome = result.outcome().as_str(); + let reason = result.reason_code().as_str(); + Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome: outcome.to_owned(), + reason: reason.to_owned(), + artifact_uid: Some(export.artifact_uid), + artifact_sha256: Some(export.archive_sha256), + artifact_bytes: Some(export.archive_bytes), + }) +} + +async fn execute_top_api_job( + envelope: DiagnosticJobEnvelope, + nonce: [u8; 32], + identity: &DeviceIdentity, + provenance: ProfileProvenance, + cancel: &CancellationToken, +) -> Result { + let expire = parse_time(&envelope.expire_time)?; + let consent_expire = parse_time(&envelope.parameters.consent_expires_at)?; + let request = TopCaptureRequest { + scope: TopCaptureScope { + organization_name: envelope.organization_name, + cluster_name: envelope.cluster_name, + device_name: envelope.device_name, + run_uid: envelope.job_id.clone(), + artifact_uid: envelope.parameters.artifact_uid, + policy_revision: envelope.parameters.consent_policy_revision, + run_expires_at_unix: expire.timestamp(), + executable_sha256: provenance.executable_sha256().to_owned(), + build_features: provenance.build_features().to_vec(), + consent: LocalTopConsent { + uid: envelope.parameters.consent_uid, + tool_id: TOP_API_JOB_TYPE.to_owned(), + classification: TOP_CLASSIFICATION.to_owned(), + active: true, + expires_at_unix: consent_expire.timestamp(), + }, + }, + limits: TopCaptureLimits { + max_duration_millis: envelope.parameters.duration_millis, + max_working_memory_bytes: envelope.limits.max_memory_bytes, + max_cpu_millis: envelope.limits.max_cpu_millis, + ..TopCaptureLimits::default() + }, + window: Duration::from_millis(envelope.parameters.duration_millis), + export_validity: MAX_TOP_EXPORT_VALIDITY, + }; + let result = capture_top_api(&request, TopApiOperation::GetObject, cancel) + .await + .map_err(top_capture_failure)?; + let outcome = result.outcome.as_str().to_owned(); + let reason = result.reason_code.as_str().to_owned(); + if !matches!(result.outcome, TopOutcome::Succeeded | TopOutcome::Partial) { + return Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome, + reason, + artifact_uid: None, + artifact_sha256: None, + artifact_bytes: None, + }); + } + let export = sign_top_export_with_nonce(&request, &result, identity, cancel, nonce).map_err(top_export_failure)?; + if export.archive_bytes.len() > usize::try_from(envelope.limits.max_output_bytes).unwrap_or(usize::MAX) { + return Err(DiagnosticJobError::LimitExceeded); + } + Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome, + reason, + artifact_uid: Some(export.artifact_uid), + artifact_sha256: Some(export.archive_sha256), + artifact_bytes: Some(export.archive_bytes), + }) +} + +async fn execute_top_locks_job( + envelope: DiagnosticJobEnvelope, + nonce: [u8; 32], + identity: &DeviceIdentity, + provenance: ProfileProvenance, + cancel: &CancellationToken, +) -> Result { + let expire = parse_time(&envelope.expire_time)?; + let consent_expire = parse_time(&envelope.parameters.consent_expires_at)?; + let request = TopCaptureRequest { + scope: TopCaptureScope { + organization_name: envelope.organization_name, + cluster_name: envelope.cluster_name, + device_name: envelope.device_name, + run_uid: envelope.job_id.clone(), + artifact_uid: envelope.parameters.artifact_uid, + policy_revision: envelope.parameters.consent_policy_revision, + run_expires_at_unix: expire.timestamp(), + executable_sha256: provenance.executable_sha256().to_owned(), + build_features: provenance.build_features().to_vec(), + consent: LocalTopConsent { + uid: envelope.parameters.consent_uid, + tool_id: TOP_LOCKS_JOB_TYPE.to_owned(), + classification: TOP_CLASSIFICATION.to_owned(), + active: true, + expires_at_unix: consent_expire.timestamp(), + }, + }, + limits: TopCaptureLimits { + max_duration_millis: envelope.parameters.duration_millis, + max_working_memory_bytes: envelope.limits.max_memory_bytes, + max_cpu_millis: envelope.limits.max_cpu_millis, + ..TopCaptureLimits::default() + }, + window: Duration::from_millis(envelope.parameters.duration_millis), + export_validity: MAX_TOP_EXPORT_VALIDITY, + }; + let result = capture_top_locks(&request, cancel).await.map_err(top_capture_failure)?; + let outcome = result.outcome.as_str().to_owned(); + let reason = result.reason_code.as_str().to_owned(); + if !matches!(result.outcome, TopOutcome::Succeeded | TopOutcome::Partial) { + return Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome, + reason, + artifact_uid: None, + artifact_sha256: None, + artifact_bytes: None, + }); + } + let export = sign_top_export_with_nonce(&request, &result, identity, cancel, nonce).map_err(top_export_failure)?; + if export.archive_bytes.len() > usize::try_from(envelope.limits.max_output_bytes).unwrap_or(usize::MAX) { + return Err(DiagnosticJobError::LimitExceeded); + } + Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome, + reason, + artifact_uid: Some(export.artifact_uid), + artifact_sha256: Some(export.archive_sha256), + artifact_bytes: Some(export.archive_bytes), + }) +} + +async fn execute_top_rpc_job( + envelope: DiagnosticJobEnvelope, + nonce: [u8; 32], + identity: &DeviceIdentity, + provenance: ProfileProvenance, + cancel: &CancellationToken, +) -> Result { + let expire = parse_time(&envelope.expire_time)?; + let consent_expire = parse_time(&envelope.parameters.consent_expires_at)?; + let request = TopCaptureRequest { + scope: TopCaptureScope { + organization_name: envelope.organization_name, + cluster_name: envelope.cluster_name, + device_name: envelope.device_name, + run_uid: envelope.job_id.clone(), + artifact_uid: envelope.parameters.artifact_uid, + policy_revision: envelope.parameters.consent_policy_revision, + run_expires_at_unix: expire.timestamp(), + executable_sha256: provenance.executable_sha256().to_owned(), + build_features: provenance.build_features().to_vec(), + consent: LocalTopConsent { + uid: envelope.parameters.consent_uid, + tool_id: TOP_RPC_JOB_TYPE.to_owned(), + classification: TOP_CLASSIFICATION.to_owned(), + active: true, + expires_at_unix: consent_expire.timestamp(), + }, + }, + limits: TopCaptureLimits { + max_duration_millis: envelope.parameters.duration_millis, + max_working_memory_bytes: envelope.limits.max_memory_bytes, + max_cpu_millis: envelope.limits.max_cpu_millis, + ..TopCaptureLimits::default() + }, + window: Duration::from_millis(envelope.parameters.duration_millis), + export_validity: MAX_TOP_EXPORT_VALIDITY, + }; + let result = capture_top_rpc(&request, cancel).await.map_err(top_capture_failure)?; + let outcome = result.outcome.as_str().to_owned(); + let reason = result.reason_code.as_str().to_owned(); + if !matches!(result.outcome, TopOutcome::Succeeded | TopOutcome::Partial) { + return Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome, + reason, + artifact_uid: None, + artifact_sha256: None, + artifact_bytes: None, + }); + } + let export = sign_top_export_with_nonce(&request, &result, identity, cancel, nonce).map_err(top_export_failure)?; + if export.archive_bytes.len() > usize::try_from(envelope.limits.max_output_bytes).unwrap_or(usize::MAX) { + return Err(DiagnosticJobError::LimitExceeded); + } + Ok(DiagnosticJobExecution { + job_id: envelope.job_id, + outcome, + reason, + artifact_uid: Some(export.artifact_uid), + artifact_sha256: Some(export.archive_sha256), + artifact_bytes: Some(export.archive_bytes), + }) +} + +fn capture_failure(error: super::ProfileError) -> DiagnosticJobError { + match error { + super::ProfileError::Cancelled => DiagnosticJobError::Cancelled, + super::ProfileError::LimitExceeded | super::ProfileError::TimedOut => DiagnosticJobError::LimitExceeded, + super::ProfileError::SourceUnavailable => DiagnosticJobError::ProfileSourceUnavailable, + _ => DiagnosticJobError::CollectionFailed, + } +} + +fn export_failure(error: super::ProfileError) -> DiagnosticJobError { + match error { + super::ProfileError::Cancelled => DiagnosticJobError::Cancelled, + super::ProfileError::LimitExceeded => DiagnosticJobError::LimitExceeded, + _ => DiagnosticJobError::ExportFailed, + } +} + +fn top_capture_failure(error: super::TopCaptureError) -> DiagnosticJobError { + match error { + super::TopCaptureError::Cancelled => DiagnosticJobError::Cancelled, + super::TopCaptureError::Limits | super::TopCaptureError::ResultTooLarge => DiagnosticJobError::LimitExceeded, + _ => DiagnosticJobError::CollectionFailed, + } +} + +fn network_capture_failure(error: NetworkPerformanceError) -> DiagnosticJobError { + match error { + NetworkPerformanceError::Cancelled => DiagnosticJobError::Cancelled, + NetworkPerformanceError::LimitExceeded | NetworkPerformanceError::Busy => DiagnosticJobError::LimitExceeded, + _ => DiagnosticJobError::CollectionFailed, + } +} + +fn network_export_failure(error: NetworkPerformanceError) -> DiagnosticJobError { + match error { + NetworkPerformanceError::Cancelled => DiagnosticJobError::Cancelled, + NetworkPerformanceError::LimitExceeded => DiagnosticJobError::LimitExceeded, + _ => DiagnosticJobError::ExportFailed, + } +} + +fn drive_capture_failure(error: DrivePerformanceError) -> DiagnosticJobError { + match error { + DrivePerformanceError::Cancelled => DiagnosticJobError::Cancelled, + DrivePerformanceError::LimitExceeded | DrivePerformanceError::Busy => DiagnosticJobError::LimitExceeded, + _ => DiagnosticJobError::CollectionFailed, + } +} + +fn drive_export_failure(error: DrivePerformanceError) -> DiagnosticJobError { + match error { + DrivePerformanceError::Cancelled => DiagnosticJobError::Cancelled, + DrivePerformanceError::LimitExceeded => DiagnosticJobError::LimitExceeded, + _ => DiagnosticJobError::ExportFailed, + } +} + +fn top_export_failure(error: super::TopCaptureError) -> DiagnosticJobError { + match error { + super::TopCaptureError::Cancelled => DiagnosticJobError::Cancelled, + super::TopCaptureError::Limits | super::TopCaptureError::ResultTooLarge => DiagnosticJobError::LimitExceeded, + _ => DiagnosticJobError::ExportFailed, + } +} + +fn parse_time(value: &str) -> Result, DiagnosticJobError> { + DateTime::parse_from_rfc3339(value) + .map(|value| value.with_timezone(&Utc)) + .map_err(|_| DiagnosticJobError::Invalid) +} + +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uid| { + uid.get_variant() == Variant::RFC4122 && uid.get_version() == Some(Version::SortRand) && uid.to_string() == value + }) +} + +fn lower_hex(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn hex_lower(bytes: &[u8]) -> String { + hex_simd::encode_to_string(bytes, hex_simd::AsciiCase::Lower) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::storage_api::cluster::{Endpoint, EndpointServerPools, Endpoints, PoolEndpoints}; + use ed25519_dalek::{Signer as _, SigningKey}; + + fn envelope() -> DiagnosticJobEnvelope { + DiagnosticJobEnvelope { + job_id: "018cc251-f400-7abc-8def-0123456789ab".to_owned(), + protocol_version: "v1".to_owned(), + job_type: "profile.cpu".to_owned(), + schema_version: 1, + organization_name: "organizations/018cc251-f400-7abc-8def-0123456789ab".to_owned(), + cluster_name: "organizations/018cc251-f400-7abc-8def-0123456789ab/clusters/018cc251-f400-7abc-8def-0123456789ac".to_owned(), + device_name: "organizations/018cc251-f400-7abc-8def-0123456789ab/clusters/018cc251-f400-7abc-8def-0123456789ac/clusterDevices/018cc251-f400-7abc-8def-0123456789ad".to_owned(), + create_time: "2030-01-01T00:00:00Z".to_owned(), + expire_time: "2030-01-01T00:00:30Z".to_owned(), + nonce: URL_SAFE_NO_PAD.encode_to_string([7_u8; 32]), + required_capabilities: vec![CPU_PROFILE_CAPABILITY.to_owned()], + authorization: DiagnosticJobAuthorization { + actor_type: "BROWSER_USER".to_owned(), + actor_name: "users/018cc251-f400-7abc-8def-0123456789ab".to_owned(), + request_id: "123e4567-e89b-42d3-a456-426614174001".to_owned(), + }, + limits: DiagnosticJobLimits { + timeout_seconds: 30, + max_output_bytes: 524_288, + max_memory_bytes: 64 * 1024 * 1024, + max_cpu_millis: 30_000, + }, + parameters: DiagnosticJobParameters { + artifact_uid: "018cc251-f400-7abc-8def-0123456789ae".to_owned(), + consent_uid: "018cc251-f400-7abc-8def-0123456789af".to_owned(), + consent_policy_revision: 1, + consent_expires_at: "2030-01-01T00:01:00Z".to_owned(), + duration_millis: 1_000, + sample_period_micros: 10_000, + traffic_bytes: None, + target_alias: None, + scratch_bytes: None, + block_bytes: None, + }, + signature: DiagnosticJobSignature { + algorithm: "Ed25519".to_owned(), + key_id: String::new(), + value: String::new(), + }, + } + } + + fn signed_envelope(mut envelope: DiagnosticJobEnvelope) -> (DiagnosticJobEnvelope, TrustedDiagnosticJobSigner) { + let signing = SigningKey::from_bytes(&[9_u8; 32]); + let key_id = hex_lower(&Sha256::digest(signing.verifying_key().as_bytes())); + let trusted = + TrustedDiagnosticJobSigner::new(key_id.clone(), *signing.verifying_key().as_bytes()).expect("trusted signer"); + envelope.signature.key_id = key_id; + envelope.signature.value = + URL_SAFE_NO_PAD.encode_to_string(signing.sign(&envelope.signing_payload().expect("payload")).to_bytes()); + (envelope, trusted) + } + + fn signed() -> (DiagnosticJobEnvelope, TrustedDiagnosticJobSigner) { + signed_envelope(envelope()) + } + + fn target(envelope: &DiagnosticJobEnvelope) -> DiagnosticJobTarget { + DiagnosticJobTarget { + organization_name: envelope.organization_name.clone(), + cluster_name: envelope.cluster_name.clone(), + device_name: envelope.device_name.clone(), + } + } + + #[test] + fn advertised_diagnostic_capabilities_have_execution_paths() { + use crate::config::Cli; + use clap::CommandFactory; + + let expected = [ + ("performance.client@1", &["performance", "client"][..]), + ("performance.drive@1", &["performance", "drive"][..]), + ("performance.network@1", &[][..]), + ("performance.object@1", &["performance", "object"][..]), + ("performance.siteReplication@1", &["performance", "site-replication"][..]), + ("logs.capture@1", &["logs"][..]), + ("profile.cpu@1", &["profile"][..]), + ("profile.memory@1", &["profile"][..]), + ("profile.threads@1", &["profile"][..]), + ("telemetry.record@1", &["telemetry", "record"][..]), + ("telemetry.otlp@1", &["telemetry", "otlp"][..]), + ("telemetry.replay@1", &["telemetry", "replay"][..]), + ("top.api@1", &["top", "api"][..]), + ("top.disk@1", &["top", "disk"][..]), + ("top.locks@1", &["top", "locks"][..]), + ("top.net@1", &["top", "net"][..]), + ("top.rpc@1", &["top", "rpc"][..]), + ("inspect.object@1", &["inspect", "object"][..]), + ]; + assert_eq!( + super::super::CONNECT_DIAGNOSTIC_CAPABILITIES, + expected.iter().map(|(capability, _)| *capability).collect::>() + ); + + let command = Cli::command(); + let connect = command.find_subcommand("connect").expect("connect command"); + for (capability, path) in expected { + if path.is_empty() { + // Network probes use the authenticated service dispatcher and + // locally resolved peers, not a standalone CLI command. + let mut job = envelope(); + job.job_type = PERFORMANCE_NETWORK_JOB_TYPE.to_owned(); + job.required_capabilities = vec![capability.to_owned()]; + job.schema_version = NETWORK_SCHEMA_VERSION; + assert_eq!(job.kind(), Ok(DiagnosticJobKind::PerformanceNetwork)); + continue; + } + let mut command = connect; + for segment in path { + command = command + .find_subcommand(segment) + .unwrap_or_else(|| panic!("{capability} is missing CLI dispatch at {segment}")); + } + } + } + + #[test] + fn accepts_a_bounded_signed_profile_job_for_the_exact_device() { + let (envelope, signer) = signed(); + signer + .verify(&envelope, &target(&envelope), "2030-01-01T00:00:10Z".parse().expect("time")) + .expect("valid job"); + } + + #[test] + fn accepts_only_the_thread_profile_capability_pair() { + let mut threads = envelope(); + threads.job_type = PROFILE_THREADS_JOB_TYPE.to_owned(); + threads.required_capabilities = vec![THREAD_PROFILE_CAPABILITY.to_owned()]; + let (threads, signer) = signed_envelope(threads); + signer + .verify(&threads, &target(&threads), "2030-01-01T00:00:10Z".parse().expect("time")) + .expect("valid profile.threads job"); + + let mut mismatched = threads; + mismatched.required_capabilities = vec![CPU_PROFILE_CAPABILITY.to_owned()]; + assert_eq!( + signer.verify(&mismatched, &target(&mismatched), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::Unsupported) + ); + } + + #[tokio::test] + async fn executes_thread_profile_jobs_against_the_service_process() { + let now = Utc::now(); + let mut envelope = envelope(); + envelope.job_type = PROFILE_THREADS_JOB_TYPE.to_owned(); + envelope.required_capabilities = vec![THREAD_PROFILE_CAPABILITY.to_owned()]; + envelope.create_time = now.to_rfc3339_opts(chrono::SecondsFormat::Secs, true); + envelope.expire_time = (now + chrono::Duration::seconds(30)).to_rfc3339_opts(chrono::SecondsFormat::Secs, true); + envelope.parameters.consent_expires_at = + (now + chrono::Duration::seconds(60)).to_rfc3339_opts(chrono::SecondsFormat::Secs, true); + let execution = execute_diagnostic_job( + VerifiedDiagnosticJob { + envelope, + nonce: [7_u8; 32], + }, + &DeviceIdentity::generate(), + ProfileProvenance::new("a".repeat(40), "b".repeat(64), "1.0.0", vec![]), + &CancellationToken::new(), + ) + .await + .expect("native thread profile job should execute"); + + #[cfg(target_os = "linux")] + { + assert_eq!(execution.outcome, "SUCCEEDED"); + assert_eq!(execution.reason, "COMPLETE"); + assert!(execution.artifact_bytes.is_some_and(|bytes| !bytes.is_empty())); + } + #[cfg(not(target_os = "linux"))] + { + assert_eq!(execution.outcome, "UNSUPPORTED"); + assert_eq!(execution.reason, "UNSUPPORTED_PLATFORM"); + assert!(execution.artifact_bytes.is_none()); + } + } + + #[test] + fn accepts_only_the_bounded_top_api_capability_pair() { + let mut top = envelope(); + top.job_type = TOP_API_JOB_TYPE.to_owned(); + top.required_capabilities = vec![TOP_API_CAPABILITY.to_owned()]; + top.limits.max_cpu_millis = MAX_TOP_API_CPU_MILLIS; + let (top, signer) = signed_envelope(top); + signer + .verify(&top, &target(&top), "2030-01-01T00:00:10Z".parse().expect("time")) + .expect("valid top.api job"); + + let mut mismatched = top.clone(); + mismatched.required_capabilities = vec![CPU_PROFILE_CAPABILITY.to_owned()]; + assert_eq!( + signer.verify(&mismatched, &target(&mismatched), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::Unsupported) + ); + + let mut unbounded = top; + unbounded.limits.max_cpu_millis += 1; + assert_eq!( + signer.verify(&unbounded, &target(&unbounded), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::LimitExceeded) + ); + } + + #[test] + fn accepts_only_a_bounded_network_traffic_budget() { + let mut network = envelope(); + network.job_type = PERFORMANCE_NETWORK_JOB_TYPE.to_owned(); + network.required_capabilities = vec![NETWORK_CAPABILITY.to_owned()]; + network.limits.max_cpu_millis = MAX_NETWORK_CPU_MILLIS; + network.parameters.duration_millis = MAX_NETWORK_CPU_MILLIS; + network.parameters.traffic_bytes = Some(MAX_NETWORK_TRAFFIC_BYTES); + let (network, signer) = signed_envelope(network); + signer + .verify(&network, &target(&network), "2030-01-01T00:00:10Z".parse().expect("time")) + .expect("valid performance.network job"); + + let mut missing = network.clone(); + missing.parameters.traffic_bytes = None; + assert_eq!( + signer.verify(&missing, &target(&missing), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::LimitExceeded) + ); + + let mut zero = network.clone(); + zero.parameters.traffic_bytes = Some(0); + assert_eq!( + signer.verify(&zero, &target(&zero), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::LimitExceeded) + ); + + let mut unbounded = network.clone(); + unbounded.parameters.traffic_bytes = Some(MAX_NETWORK_TRAFFIC_BYTES + 1); + assert_eq!( + signer.verify(&unbounded, &target(&unbounded), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::LimitExceeded) + ); + + let mut profile = signed().0; + profile.parameters.traffic_bytes = Some(1); + assert_eq!( + signer.verify(&profile, &target(&profile), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::Invalid) + ); + + let unsigned_network = serde_json::to_value(network.unsigned()).expect("network envelope"); + let unsigned_profile = serde_json::to_value(envelope().unsigned()).expect("profile envelope"); + assert_eq!(unsigned_network["parameters"]["trafficBytes"], MAX_NETWORK_TRAFFIC_BYTES); + assert!(unsigned_profile["parameters"].get("trafficBytes").is_none()); + } + + #[test] + fn accepts_only_the_fixed_drive_target_and_budget_without_a_path() { + let mut drive = envelope(); + drive.job_type = PERFORMANCE_DRIVE_JOB_TYPE.to_owned(); + drive.required_capabilities = vec![DRIVE_CAPABILITY.to_owned()]; + drive.limits.max_cpu_millis = DRIVE_DURATION_MILLIS; + drive.parameters.duration_millis = DRIVE_DURATION_MILLIS; + drive.parameters.target_alias = Some(DRIVE_TARGET_ALIAS.to_owned()); + drive.parameters.scratch_bytes = Some(DRIVE_SCRATCH_BYTES); + drive.parameters.block_bytes = Some(DRIVE_BLOCK_BYTES); + let (drive, signer) = signed_envelope(drive); + signer + .verify(&drive, &target(&drive), "2030-01-01T00:00:10Z".parse().expect("time")) + .expect("valid performance.drive job"); + + let mut wrong_target = drive.clone(); + wrong_target.parameters.target_alias = Some("../../customer-data".to_owned()); + assert_eq!( + signer.verify(&wrong_target, &target(&wrong_target), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::Invalid) + ); + + let mut unbounded = drive.clone(); + unbounded.parameters.scratch_bytes = Some(DRIVE_SCRATCH_BYTES + 1); + assert_eq!( + signer.verify(&unbounded, &target(&unbounded), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::LimitExceeded) + ); + + let unsigned = serde_json::to_value(drive.unsigned()).expect("drive envelope"); + assert_eq!(unsigned["parameters"]["targetAlias"], DRIVE_TARGET_ALIAS); + assert_eq!(unsigned["parameters"]["scratchBytes"], DRIVE_SCRATCH_BYTES); + assert_eq!(unsigned["parameters"]["blockBytes"], DRIVE_BLOCK_BYTES); + assert!(unsigned["parameters"].get("scratchRoot").is_none()); + + let mut with_path = serde_json::to_value(&drive).expect("drive envelope"); + with_path["parameters"]["scratchRoot"] = serde_json::Value::String("/customer/data".to_owned()); + assert!(serde_json::from_value::(with_path).is_err()); + } + + #[tokio::test] + async fn drive_adapter_exports_only_successful_measurements() { + let now = Utc::now(); + let mut drive = envelope(); + drive.job_type = PERFORMANCE_DRIVE_JOB_TYPE.to_owned(); + drive.required_capabilities = vec![DRIVE_CAPABILITY.to_owned()]; + drive.create_time = now.to_rfc3339(); + drive.expire_time = (now + chrono::Duration::seconds(30)).to_rfc3339(); + drive.parameters.consent_expires_at = (now + chrono::Duration::seconds(60)).to_rfc3339(); + drive.limits.max_cpu_millis = DRIVE_DURATION_MILLIS; + drive.parameters.duration_millis = DRIVE_DURATION_MILLIS; + drive.parameters.target_alias = Some(DRIVE_TARGET_ALIAS.to_owned()); + drive.parameters.scratch_bytes = Some(DRIVE_SCRATCH_BYTES); + drive.parameters.block_bytes = Some(DRIVE_BLOCK_BYTES); + let (drive, signer) = signed_envelope(drive); + let verified = signer.verify(&drive, &target(&drive), now).expect("valid drive job"); + let provenance = ProfileProvenance::new("a".repeat(40), "b".repeat(64), "1.0.0", Vec::new()); + let data_drive = tempfile::tempdir().expect("data drive"); + let scratch = data_drive.path().join(DRIVE_SCRATCH_DIRECTORY); + assert!(ensure_drive_scratch_root(&scratch)); + let result = execute_performance_drive_job( + verified.envelope, + verified.nonce, + &DeviceIdentity::generate(), + provenance.clone(), + &scratch, + &CancellationToken::new(), + ) + .await + .expect("drive execution"); + assert_eq!(result.outcome, "SUCCEEDED"); + assert!(result.artifact_bytes.is_some()); + assert_eq!(scratch.read_dir().expect("scratch contents").count(), 0); + + let verified = signer.verify(&drive, &target(&drive), now).expect("valid drive job"); + let missing_root = data_drive.path().join("not-configured"); + let result = execute_performance_drive_job( + verified.envelope, + verified.nonce, + &DeviceIdentity::generate(), + provenance, + &missing_root, + &CancellationToken::new(), + ) + .await + .expect("terminal drive execution"); + assert_eq!(result.outcome, "FAILED"); + assert_eq!(result.reason, "SOURCE_UNAVAILABLE"); + assert!(result.artifact_uid.is_none()); + assert!(result.artifact_sha256.is_none()); + assert!(result.artifact_bytes.is_none()); + } + + #[test] + fn drive_alias_resolves_to_the_first_local_storage_endpoint() { + let local_drive = tempfile::tempdir().expect("local drive"); + let mut remote = Endpoint::try_from("http://node-b.example:9000/remote-drive").expect("remote endpoint"); + remote.set_pool_index(0); + remote.set_set_index(0); + remote.set_disk_index(0); + let mut local = Endpoint::try_from(local_drive.path().to_str().expect("local path")).expect("local endpoint"); + local.set_pool_index(0); + local.set_set_index(0); + local.set_disk_index(1); + let pools = EndpointServerPools::from(vec![PoolEndpoints { + legacy: false, + set_count: 1, + drives_per_set: 2, + endpoints: Endpoints::from(vec![remote, local]), + cmd_line: "test storage endpoints".to_owned(), + platform: "test".to_owned(), + }]); + + assert_eq!( + drive_scratch_root_from_endpoints(&pools), + Some(local_drive.path().join(DRIVE_SCRATCH_DIRECTORY)) + ); + } + + #[test] + fn accepts_only_the_bounded_top_locks_capability_pair() { + let mut top = envelope(); + top.job_type = TOP_LOCKS_JOB_TYPE.to_owned(); + top.required_capabilities = vec![TOP_LOCKS_CAPABILITY.to_owned()]; + top.limits.max_cpu_millis = MAX_TOP_LOCKS_CPU_MILLIS; + let (top, signer) = signed_envelope(top); + signer + .verify(&top, &target(&top), "2030-01-01T00:00:10Z".parse().expect("time")) + .expect("valid top.locks job"); + + let mut mismatched = top.clone(); + mismatched.required_capabilities = vec![TOP_API_CAPABILITY.to_owned()]; + assert_eq!( + signer.verify(&mismatched, &target(&mismatched), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::Unsupported) + ); + + let mut unbounded = top; + unbounded.limits.max_cpu_millis += 1; + assert_eq!( + signer.verify(&unbounded, &target(&unbounded), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::LimitExceeded) + ); + } + + #[tokio::test] + #[serial_test::serial] + async fn top_locks_job_preserves_the_signed_job_nonce() { + rustfs_lock::get_global_lock_manager(); + let now = Utc::now(); + let mut top = envelope(); + top.job_type = TOP_LOCKS_JOB_TYPE.to_owned(); + top.required_capabilities = vec![TOP_LOCKS_CAPABILITY.to_owned()]; + top.limits.max_cpu_millis = MAX_TOP_LOCKS_CPU_MILLIS; + top.parameters.duration_millis = 1; + top.create_time = now.to_rfc3339_opts(chrono::SecondsFormat::Secs, true); + top.expire_time = (now + chrono::Duration::seconds(30)).to_rfc3339_opts(chrono::SecondsFormat::Secs, true); + top.parameters.consent_expires_at = + (now + chrono::Duration::seconds(60)).to_rfc3339_opts(chrono::SecondsFormat::Secs, true); + let nonce = [7_u8; 32]; + let execution = execute_diagnostic_job( + VerifiedDiagnosticJob { envelope: top, nonce }, + &DeviceIdentity::generate(), + ProfileProvenance::new("a".repeat(40), "b".repeat(64), "1.0.0", vec![]), + &CancellationToken::new(), + ) + .await + .expect("top.locks job should execute"); + + let bytes = execution.artifact_bytes.expect("top.locks artifact"); + let mut archive = zip::ZipArchive::new(std::io::Cursor::new(bytes)).expect("top.locks archive"); + let mut envelope = String::new(); + std::io::Read::read_to_string(&mut archive.by_name("envelope.json").expect("top.locks envelope"), &mut envelope) + .expect("read top.locks envelope"); + let envelope: serde_json::Value = serde_json::from_str(&envelope).expect("valid top.locks envelope"); + assert_eq!(envelope["nonce"], URL_SAFE_NO_PAD.encode_to_string(nonce)); + } + + #[test] + fn accepts_only_the_bounded_top_rpc_capability_pair() { + let mut top = envelope(); + top.job_type = TOP_RPC_JOB_TYPE.to_owned(); + top.required_capabilities = vec![TOP_RPC_CAPABILITY.to_owned()]; + top.limits.max_cpu_millis = MAX_TOP_RPC_CPU_MILLIS; + let (top, signer) = signed_envelope(top); + signer + .verify(&top, &target(&top), "2030-01-01T00:00:10Z".parse().expect("time")) + .expect("valid top.rpc job"); + + let mut mismatched = top.clone(); + mismatched.required_capabilities = vec![TOP_LOCKS_CAPABILITY.to_owned()]; + assert_eq!( + signer.verify(&mismatched, &target(&mismatched), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::Unsupported) + ); + + let mut unbounded = top; + unbounded.limits.max_cpu_millis += 1; + assert_eq!( + signer.verify(&unbounded, &target(&unbounded), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::LimitExceeded) + ); + } + + #[tokio::test] + #[serial_test::serial] + async fn top_rpc_job_captures_service_process_events_and_exports_a_signed_artifact() { + use rustfs_common::trace_bus::{ + TelemetryTraceEvent, TelemetryTraceOperation, TelemetryTraceStatus, telemetry_trace_emit, + }; + + let mut top = envelope(); + top.job_type = TOP_RPC_JOB_TYPE.to_owned(); + top.required_capabilities = vec![TOP_RPC_CAPABILITY.to_owned()]; + top.limits.max_cpu_millis = MAX_TOP_RPC_CPU_MILLIS; + top.parameters.duration_millis = 50; + + let emit = async { + tokio::time::sleep(Duration::from_millis(10)).await; + assert!(telemetry_trace_emit(|| { + TelemetryTraceEvent::new( + TelemetryTraceOperation::InternalRpc, + Duration::from_micros(37), + TelemetryTraceStatus::Ok, + ) + })); + }; + let identity = DeviceIdentity::generate(); + let cancellation = CancellationToken::new(); + let execute = execute_diagnostic_job( + VerifiedDiagnosticJob { + envelope: top, + nonce: [7_u8; 32], + }, + &identity, + ProfileProvenance::new("a".repeat(40), "b".repeat(64), "1.0.0", vec![]), + &cancellation, + ); + let (execution, ()) = tokio::join!(execute, emit); + let execution = execution.expect("top.rpc job should execute"); + + assert_eq!(execution.outcome, "SUCCEEDED"); + assert_eq!(execution.reason, "COMPLETE"); + assert!(execution.artifact_bytes.is_some_and(|bytes| !bytes.is_empty())); + } + + #[test] + fn rejects_tampering_cross_device_replay_and_expiry() { + let (envelope, signer) = signed(); + let mut tampered = envelope.clone(); + tampered.parameters.duration_millis = 2_000; + assert_eq!( + signer.verify(&tampered, &target(&tampered), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::SignatureInvalid) + ); + let mut wrong_target = target(&envelope); + wrong_target.device_name.push('0'); + assert_eq!( + signer.verify(&envelope, &wrong_target, "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::TargetMismatch) + ); + assert_eq!( + signer.verify(&envelope, &target(&envelope), "2030-01-01T00:00:30Z".parse().expect("time")), + Err(DiagnosticJobError::Expired) + ); + let (mut actor_tampered, signer) = signed(); + actor_tampered.authorization.actor_name.push('0'); + assert_eq!( + signer.verify(&actor_tampered, &target(&actor_tampered), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::Invalid) + ); + } + + #[test] + fn rejects_unbounded_and_non_allow_listed_jobs_before_signature_use() { + let (mut envelope, signer) = signed(); + envelope.limits.max_output_bytes += 1; + assert_eq!( + signer.verify(&envelope, &target(&envelope), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::LimitExceeded) + ); + envelope = signed().0; + envelope.job_type = "shell.exec".to_owned(); + assert_eq!( + signer.verify(&envelope, &target(&envelope), "2030-01-01T00:00:10Z".parse().expect("time")), + Err(DiagnosticJobError::Unsupported) + ); + } + + #[test] + fn exposes_only_allow_listed_profile_failure_reasons() { + assert_eq!( + capture_failure(super::super::ProfileError::SourceUnavailable), + DiagnosticJobError::ProfileSourceUnavailable + ); + assert_eq!(capture_failure(super::super::ProfileError::TimedOut), DiagnosticJobError::LimitExceeded); + assert_eq!(capture_failure(super::super::ProfileError::Cancelled), DiagnosticJobError::Cancelled); + assert_eq!(export_failure(super::super::ProfileError::Encoding), DiagnosticJobError::ExportFailed); + assert_eq!(DiagnosticJobError::ProfileSourceUnavailable.reason(), "PROFILE_SOURCE_UNAVAILABLE"); + assert_eq!(DiagnosticJobError::ExportFailed.reason(), "EXPORT_FAILED"); + } +} diff --git a/rustfs/src/connect/diagnostics/job_delivery.rs b/rustfs/src/connect/diagnostics/job_delivery.rs new file mode 100644 index 000000000..d29f6a18a --- /dev/null +++ b/rustfs/src/connect/diagnostics/job_delivery.rs @@ -0,0 +1,821 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::{BTreeMap, HashSet}; +use std::fs::{self, File, OpenOptions}; +use std::io::{Read as _, Write as _}; +use std::path::{Path, PathBuf}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +#[cfg(unix)] +use std::os::unix::fs::{MetadataExt as _, OpenOptionsExt as _, PermissionsExt as _}; + +use chrono::Utc; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use tokio_util::sync::CancellationToken; +use uuid::Uuid; + +use super::{ + DiagnosticJobEnvelope, DiagnosticJobExecution, DiagnosticJobTarget, ProfileProvenance, TrustedDiagnosticJobSigner, + execute_diagnostic_job, +}; +use crate::connect::config::HeartbeatConfig; +use crate::connect::report_upload::ReportUploadClient; +use crate::connect::telemetry::{TelemetryDelivery, TelemetryTransport}; + +const PROTOCOL_VERSION: &str = "v1"; +const MAX_EXECUTABLE_BYTES: u64 = 2_147_483_648; +const INITIAL_DELIVERY_BACKOFF: Duration = Duration::from_secs(1); +const MAX_DELIVERY_BACKOFF: Duration = Duration::from_secs(30); +const MAX_STATE_BYTES: u64 = 4 * 1024 * 1024; +const MAX_ARTIFACT_BYTES: usize = 524_288; +const MAX_STATE_FILES: usize = 65_536; + +#[derive(Clone)] +pub(crate) struct DiagnosticJobRuntime { + config: HeartbeatConfig, + signer: TrustedDiagnosticJobSigner, + states: Arc>>, + deliveries: DeliveryRegistry, + store: JobStateStore, +} + +#[derive(Clone, Default)] +struct DeliveryRegistry(Arc>>); + +struct DeliveryLease { + job_id: String, + registry: DeliveryRegistry, +} + +#[derive(Clone, Deserialize, Serialize)] +#[serde(deny_unknown_fields, tag = "state", content = "result", rename_all = "SCREAMING_SNAKE_CASE")] +enum JobState { + Active, + Completed(DiagnosticJobExecution), + Uploaded(UploadedDiagnosticJobResult), + Delivered, +} + +#[derive(Clone)] +struct JobStateStore { + directory: PathBuf, +} + +#[derive(Clone, Deserialize, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +struct UploadedDiagnosticJobResult { + job_id: String, + outcome: String, + reason: String, + artifact_name: Option, + artifact_sha256: Option, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct DiagnosticJobResultRequest<'a> { + protocol_version: &'static str, + request_id: &'a str, + job_id: &'a str, + outcome: &'a str, + reason: &'a str, + artifact_name: Option<&'a str>, + artifact_sha256: Option<&'a str>, +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum DeliveryAttempt { + Accepted, + Retry, + Stop, +} + +impl DiagnosticJobRuntime { + pub(crate) fn from_config(config: &HeartbeatConfig) -> Option { + let state_root = config.state_root()?.to_path_buf(); + Some(Self { + config: config.clone(), + signer: config.diagnostic_job_signer.clone()?, + states: Arc::new(Mutex::new(BTreeMap::new())), + deliveries: DeliveryRegistry::default(), + store: JobStateStore { + directory: state_root.join("diagnostic-jobs"), + }, + }) + } + + pub(crate) fn offer(&self, envelope: DiagnosticJobEnvelope, shutdown: &CancellationToken) { + let Ok((target, identity)) = target_and_identity(&self.config) else { + return; + }; + let Ok(job) = self.signer.verify(&envelope, &target, Utc::now()) else { + return; + }; + let job_id = job.job_id().to_owned(); + let cached = { + let Ok(mut states) = self.states.lock() else { + return; + }; + match states.get(&job_id) { + Some(JobState::Active) => return, + Some(JobState::Completed(result)) => Some(result.clone()), + Some(JobState::Uploaded(result)) => { + let result = result.clone(); + let expire_time = job.expire_time().to_owned(); + drop(states); + let runtime = self.clone(); + let cancel = shutdown.child_token(); + tokio::spawn(async move { + runtime.deliver_uploaded(result, &expire_time, &cancel).await; + }); + return; + } + Some(JobState::Delivered) => return, + None => match self.store.load(&job_id) { + Ok(Some(JobState::Completed(result))) => { + if !valid_execution(&result, &job_id) { + return; + } + states.insert(job_id.clone(), JobState::Completed(result.clone())); + Some(result) + } + Ok(Some(JobState::Delivered)) => { + states.insert(job_id, JobState::Delivered); + return; + } + Ok(Some(JobState::Uploaded(result))) => { + if !valid_uploaded_result(&result, &job_id) { + return; + } + states.insert(job_id.clone(), JobState::Uploaded(result.clone())); + let expire_time = job.expire_time().to_owned(); + drop(states); + let runtime = self.clone(); + let cancel = shutdown.child_token(); + tokio::spawn(async move { + runtime.deliver_uploaded(result, &expire_time, &cancel).await; + }); + return; + } + Ok(Some(JobState::Active)) => { + let result = failed_execution(&job_id, "INTERRUPTED"); + if self.store.save(&job_id, &JobState::Completed(result.clone())).is_err() { + return; + } + states.insert(job_id.clone(), JobState::Completed(result.clone())); + Some(result) + } + Ok(None) => { + if self.store.save(&job_id, &JobState::Active).is_err() { + return; + } + states.insert(job_id.clone(), JobState::Active); + None + } + Err(()) => return, + }, + } + }; + let runtime = self.clone(); + let cancel = shutdown.child_token(); + let expire_time = job.expire_time().to_owned(); + tokio::spawn(async move { + let result = match cached { + Some(result) => result, + None => { + let result = match executable_provenance().await { + Ok(provenance) => execute_diagnostic_job(job, &identity, provenance, &cancel) + .await + .unwrap_or_else(|error| failed_execution(&job_id, error.reason())), + Err(reason) => failed_execution(&job_id, reason), + }; + let terminal = JobState::Completed(result.clone()); + if runtime.store.save(&job_id, &terminal).is_ok() + && let Ok(mut states) = runtime.states.lock() + { + states.insert(job_id.clone(), terminal); + } + result + } + }; + if let Some(uploaded) = prepare_result(&runtime, result, &cancel).await { + runtime.deliver_uploaded(uploaded, &expire_time, &cancel).await; + } + }); + } + + async fn deliver_uploaded(&self, result: UploadedDiagnosticJobResult, expire_time: &str, cancel: &CancellationToken) { + let job_id = result.job_id.clone(); + let Some(_lease) = self.deliveries.acquire(&job_id) else { + return; + }; + if deliver_result(&self.config, &result, expire_time, cancel).await + && self.store.save(&job_id, &JobState::Delivered).is_ok() + && let Ok(mut states) = self.states.lock() + { + states.insert(job_id, JobState::Delivered); + } + } +} + +impl DeliveryRegistry { + fn acquire(&self, job_id: &str) -> Option { + let mut active = self.0.lock().ok()?; + if !active.insert(job_id.to_owned()) { + return None; + } + Some(DeliveryLease { + job_id: job_id.to_owned(), + registry: self.clone(), + }) + } +} + +impl Drop for DeliveryLease { + fn drop(&mut self) { + if let Ok(mut active) = self.registry.0.lock() { + active.remove(&self.job_id); + } + } +} + +impl JobStateStore { + fn path(&self, job_id: &str) -> PathBuf { + self.directory.join(format!("{job_id}.json")) + } + + fn load(&self, job_id: &str) -> Result, ()> { + let path = self.path(job_id); + let initial = match fs::symlink_metadata(&path) { + Ok(metadata) => metadata, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(_) => return Err(()), + }; + if !initial.file_type().is_file() || initial.len() == 0 || initial.len() > MAX_STATE_BYTES { + return Err(()); + } + let mut options = OpenOptions::new(); + options.read(true); + #[cfg(unix)] + options.custom_flags(libc::O_NOFOLLOW | libc::O_CLOEXEC); + let file = options.open(path).map_err(|_| ())?; + let opened = file.metadata().map_err(|_| ())?; + if !secure_state_file(&initial, &opened) { + return Err(()); + } + let mut bytes = Vec::with_capacity(usize::try_from(initial.len()).map_err(|_| ())?); + file.take(MAX_STATE_BYTES + 1).read_to_end(&mut bytes).map_err(|_| ())?; + if u64::try_from(bytes.len()).map_err(|_| ())? > MAX_STATE_BYTES { + return Err(()); + } + serde_json::from_slice(&bytes).map(Some).map_err(|_| ()) + } + + fn save(&self, job_id: &str, state: &JobState) -> Result<(), ()> { + let bytes = serde_json::to_vec(state).map_err(|_| ())?; + if bytes.is_empty() || u64::try_from(bytes.len()).map_err(|_| ())? > MAX_STATE_BYTES { + return Err(()); + } + self.ensure_directory()?; + let destination = self.path(job_id); + if !destination.exists() && self.state_file_count()? >= MAX_STATE_FILES { + return Err(()); + } + let temporary = self.directory.join(format!(".{job_id}.{}.tmp", Uuid::new_v4())); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(0o600); + } + let mut file = options.open(&temporary).map_err(|_| ())?; + let result = file + .write_all(&bytes) + .and_then(|()| file.sync_all()) + .and_then(|()| fs::rename(&temporary, &destination)) + .and_then(|()| sync_directory(&self.directory)); + if result.is_err() { + let _ = fs::remove_file(temporary); + } + result.map_err(|_| ()) + } + + fn ensure_directory(&self) -> Result<(), ()> { + fs::create_dir_all(&self.directory).map_err(|_| ())?; + #[cfg(unix)] + { + let metadata = fs::symlink_metadata(&self.directory).map_err(|_| ())?; + if !metadata.file_type().is_dir() || metadata.uid() != rustix::process::geteuid().as_raw() { + return Err(()); + } + fs::set_permissions(&self.directory, fs::Permissions::from_mode(0o700)).map_err(|_| ())?; + } + Ok(()) + } + + fn state_file_count(&self) -> Result { + let mut count = 0_usize; + for entry in fs::read_dir(&self.directory).map_err(|_| ())? { + let entry = entry.map_err(|_| ())?; + if entry.file_name().to_string_lossy().ends_with(".json") { + count = count.checked_add(1).ok_or(())?; + if count >= MAX_STATE_FILES { + break; + } + } + } + Ok(count) + } +} + +#[cfg(unix)] +fn secure_state_file(initial: &fs::Metadata, opened: &fs::Metadata) -> bool { + opened.is_file() + && opened.uid() == rustix::process::geteuid().as_raw() + && opened.permissions().mode() & 0o077 == 0 + && opened.dev() == initial.dev() + && opened.ino() == initial.ino() +} + +#[cfg(not(unix))] +fn secure_state_file(_initial: &fs::Metadata, _opened: &fs::Metadata) -> bool { + false +} + +fn sync_directory(path: &Path) -> std::io::Result<()> { + File::open(path)?.sync_all() +} + +fn target_and_identity(config: &HeartbeatConfig) -> Result<(DiagnosticJobTarget, crate::connect::DeviceIdentity), ()> { + let credential = config.credential_store.load().map_err(|_| ())?.ok_or(())?; + let parts: Vec<_> = credential.name.split('/').collect(); + if parts.len() != 6 + || parts[0] != "organizations" + || parts[1].is_empty() + || parts[2] != "clusters" + || parts[3].is_empty() + || parts[4] != "clusterDevices" + || parts[5] != credential.uid + { + return Err(()); + } + let identity = config.identity_store.load().map_err(|_| ())?.ok_or(())?; + Ok(( + DiagnosticJobTarget { + organization_name: format!("organizations/{}", parts[1]), + cluster_name: format!("organizations/{}/clusters/{}", parts[1], parts[3]), + device_name: credential.name, + }, + identity, + )) +} + +async fn executable_provenance() -> Result { + let digest = tokio::task::spawn_blocking(hash_current_executable) + .await + .map_err(|_| "PROVENANCE_FAILED")? + .map_err(|_| "PROVENANCE_FAILED")?; + Ok(ProfileProvenance::new( + crate::version::build::COMMIT_HASH, + digest, + env!("CARGO_PKG_VERSION"), + enabled_build_features(), + )) +} + +fn hash_current_executable() -> Result { + let path = std::env::current_exe().map_err(|_| ())?; + let mut file = File::open(path).map_err(|_| ())?; + let metadata = file.metadata().map_err(|_| ())?; + if !metadata.is_file() || metadata.len() == 0 || metadata.len() > MAX_EXECUTABLE_BYTES { + return Err(()); + } + let mut hasher = Sha256::new(); + let mut buffer = [0_u8; 64 * 1024]; + let mut total = 0_u64; + loop { + let read = file.read(&mut buffer).map_err(|_| ())?; + if read == 0 { + break; + } + total = total.checked_add(u64::try_from(read).map_err(|_| ())?).ok_or(())?; + if total > MAX_EXECUTABLE_BYTES { + return Err(()); + } + hasher.update(&buffer[..read]); + } + if total != metadata.len() { + return Err(()); + } + Ok(hex_simd::encode_to_string(hasher.finalize(), hex_simd::AsciiCase::Lower)) +} + +fn enabled_build_features() -> Vec { + let mut features = Vec::new(); + for (enabled, name) in [ + (cfg!(feature = "connect-e2e-short-credentials"), "connect-e2e-short-credentials"), + (cfg!(feature = "dial9"), "dial9"), + (cfg!(feature = "e2e-test-hooks"), "e2e-test-hooks"), + (cfg!(feature = "ftps"), "ftps"), + (cfg!(feature = "full"), "full"), + (cfg!(feature = "gcs"), "gcs"), + (cfg!(feature = "hotpath"), "hotpath"), + (cfg!(feature = "hotpath-alloc"), "hotpath-alloc"), + (cfg!(feature = "hotpath-cpu"), "hotpath-cpu"), + (cfg!(feature = "io-scheduler-debug"), "io-scheduler-debug"), + (cfg!(feature = "license"), "license"), + (cfg!(feature = "metrics-gpu"), "metrics-gpu"), + (cfg!(feature = "offline-enrollment-e2e-root"), "offline-enrollment-e2e-root"), + (cfg!(feature = "pyroscope"), "pyroscope"), + (cfg!(feature = "rio-v2"), "rio-v2"), + (cfg!(feature = "sftp"), "sftp"), + (cfg!(feature = "swift"), "swift"), + (cfg!(feature = "tracing-chunk-debug"), "tracing-chunk-debug"), + (cfg!(feature = "webdav"), "webdav"), + ] { + if enabled { + features.push(name.to_owned()); + } + } + features +} + +fn failed_execution(job_id: &str, reason: &'static str) -> DiagnosticJobExecution { + let reason = if reason == "CANCELLED" { + "CANCELLED" + } else if reason == "LIMIT_EXCEEDED" { + "LIMIT_EXCEEDED" + } else { + "COLLECTION_FAILED" + }; + DiagnosticJobExecution { + job_id: job_id.to_owned(), + outcome: if reason == "CANCELLED" { "CANCELLED" } else { "FAILED" }.to_owned(), + reason: reason.to_owned(), + artifact_uid: None, + artifact_sha256: None, + artifact_bytes: None, + } +} + +async fn prepare_result( + runtime: &DiagnosticJobRuntime, + result: DiagnosticJobExecution, + cancel: &CancellationToken, +) -> Option { + let uploaded = if let (Some(bytes), true) = + (result.artifact_bytes.as_ref(), matches!(result.outcome.as_str(), "SUCCEEDED" | "PARTIAL")) + { + if runtime.store.ensure_directory().is_err() { + return None; + } + let path = runtime + .store + .directory + .join(format!(".{}.{}.artifact", result.job_id, Uuid::new_v4())); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + options.mode(0o600); + let mut file = options.open(&path).ok()?; + if file.write_all(bytes).and_then(|()| file.sync_all()).is_err() { + let _ = fs::remove_file(&path); + return None; + } + drop(file); + let receipt = match ReportUploadClient::new(runtime.config.clone(), Duration::from_secs(15 * 60)) { + Ok(client) => client.upload(&path, cancel).await.ok(), + Err(_) => None, + }; + let _ = fs::remove_file(path); + let receipt = receipt?; + UploadedDiagnosticJobResult { + job_id: result.job_id, + outcome: result.outcome, + reason: result.reason, + artifact_name: Some(receipt.name), + artifact_sha256: Some(receipt.declared_sha256), + } + } else { + UploadedDiagnosticJobResult { + job_id: result.job_id, + outcome: result.outcome, + reason: result.reason, + artifact_name: None, + artifact_sha256: None, + } + }; + let state = JobState::Uploaded(uploaded.clone()); + if runtime.store.save(&uploaded.job_id, &state).is_err() { + return None; + } + if let Ok(mut states) = runtime.states.lock() { + states.insert(uploaded.job_id.clone(), state); + } + Some(uploaded) +} + +async fn deliver_result( + config: &HeartbeatConfig, + result: &UploadedDiagnosticJobResult, + expire_time: &str, + cancel: &CancellationToken, +) -> bool { + let Ok(transport) = TelemetryTransport::new(config.clone()) else { + return false; + }; + let request_id = Uuid::new_v4().to_string(); + let request = DiagnosticJobResultRequest { + protocol_version: PROTOCOL_VERSION, + request_id: &request_id, + job_id: &result.job_id, + outcome: &result.outcome, + reason: &result.reason, + artifact_name: result.artifact_name.as_deref(), + artifact_sha256: result.artifact_sha256.as_deref(), + }; + let Some(deadline) = delivery_deadline(expire_time) else { + return false; + }; + retry_delivery(deadline, cancel, || async { + match transport.post("diagnosticJobResults", &request).await { + Ok(delivery) => classify_delivery_attempt(delivery), + Err(_) => DeliveryAttempt::Retry, + } + }) + .await +} + +fn classify_delivery_attempt(delivery: TelemetryDelivery) -> DeliveryAttempt { + match delivery { + TelemetryDelivery::Accepted { .. } => DeliveryAttempt::Accepted, + TelemetryDelivery::Rejected { status: 409, .. } | TelemetryDelivery::Retry { .. } => DeliveryAttempt::Retry, + TelemetryDelivery::AuthenticationStopped { .. } | TelemetryDelivery::Rejected { .. } => DeliveryAttempt::Stop, + } +} + +fn delivery_deadline(expire_time: &str) -> Option { + let expire = chrono::DateTime::parse_from_rfc3339(expire_time).ok()?.with_timezone(&Utc); + let remaining = (expire - Utc::now()).to_std().ok()?; + Some(tokio::time::Instant::now() + remaining) +} + +async fn retry_delivery(deadline: tokio::time::Instant, cancel: &CancellationToken, mut deliver: F) -> bool +where + F: FnMut() -> Fut, + Fut: std::future::Future, +{ + let mut backoff = INITIAL_DELIVERY_BACKOFF; + loop { + if cancel.is_cancelled() || tokio::time::Instant::now() >= deadline { + return false; + } + match deliver().await { + DeliveryAttempt::Accepted => return true, + DeliveryAttempt::Stop => return false, + DeliveryAttempt::Retry => {} + } + tokio::select! { + () = cancel.cancelled() => return false, + () = tokio::time::sleep_until(deadline.min(tokio::time::Instant::now() + backoff)) => {} + } + backoff = (backoff * 2).min(MAX_DELIVERY_BACKOFF); + } +} + +fn valid_uploaded_result(result: &UploadedDiagnosticJobResult, job_id: &str) -> bool { + result.job_id == job_id + && valid_terminal_fields(&result.outcome, &result.reason) + && match (&result.artifact_name, &result.artifact_sha256) { + (Some(name), Some(digest)) => { + matches!(result.outcome.as_str(), "SUCCEEDED" | "PARTIAL") + && name.len() <= 512 + && name.starts_with("organizations/") + && lower_hex(digest, 64) + } + (None, None) => matches!(result.outcome.as_str(), "FAILED" | "UNSUPPORTED" | "CANCELLED"), + _ => false, + } +} + +fn valid_terminal_fields(outcome: &str, reason: &str) -> bool { + matches!(outcome, "SUCCEEDED" | "PARTIAL" | "FAILED" | "UNSUPPORTED" | "CANCELLED") + && !reason.is_empty() + && reason.len() <= 64 + && reason + .bytes() + .all(|byte| byte.is_ascii_uppercase() || byte.is_ascii_digit() || byte == b'_') +} + +fn lower_hex(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn valid_execution(result: &DiagnosticJobExecution, job_id: &str) -> bool { + let valid_artifact = match (&result.artifact_uid, &result.artifact_sha256, &result.artifact_bytes) { + (Some(uid), Some(digest), Some(bytes)) => { + Uuid::parse_str(uid).is_ok_and(|value| value.get_version_num() == 7 && value.to_string() == *uid) + && !bytes.is_empty() + && bytes.len() <= MAX_ARTIFACT_BYTES + && hex_simd::encode_to_string(Sha256::digest(bytes), hex_simd::AsciiCase::Lower) == *digest + } + (None, None, None) => matches!(result.outcome.as_str(), "FAILED" | "CANCELLED") && result.reason != "COMPLETE", + _ => false, + }; + result.job_id == job_id && valid_terminal_fields(&result.outcome, &result.reason) && valid_artifact +} + +#[cfg(test)] +mod tests { + use super::*; + + fn execution(job_id: &str, artifact: &[u8]) -> DiagnosticJobExecution { + DiagnosticJobExecution { + job_id: job_id.to_owned(), + outcome: "SUCCEEDED".to_owned(), + reason: "COMPLETE".to_owned(), + artifact_uid: Some(Uuid::now_v7().to_string()), + artifact_sha256: Some(hex_simd::encode_to_string(Sha256::digest(artifact), hex_simd::AsciiCase::Lower)), + artifact_bytes: Some(artifact.to_vec()), + } + } + + #[test] + fn result_request_has_no_generic_execution_fields() { + let request = DiagnosticJobResultRequest { + protocol_version: "v1", + request_id: "123e4567-e89b-42d3-a456-426614174001", + job_id: "018cc251-f400-7abc-8def-0123456789ab", + outcome: "FAILED", + reason: "CANCELLED", + artifact_name: None, + artifact_sha256: None, + }; + let value = serde_json::to_value(request).expect("result request"); + assert_eq!(value["jobId"], "018cc251-f400-7abc-8def-0123456789ab"); + for forbidden in [ + "command", + "script", + "path", + "sql", + "arguments", + "artifact", + "artifactUid", + "artifactEncoding", + ] { + assert!(value.get(forbidden).is_none()); + } + } + + #[test] + fn result_redelivery_has_only_one_in_flight_attempt_loop() { + let registry = DeliveryRegistry::default(); + let job_id = Uuid::now_v7().to_string(); + let first = registry.acquire(&job_id).expect("first delivery"); + assert!(registry.acquire(&job_id).is_none()); + drop(first); + assert!(registry.acquire(&job_id).is_some()); + } + + #[tokio::test(start_paused = true)] + async fn result_delivery_retries_a_conflict_until_accepted() { + let cancel = CancellationToken::new(); + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + let mut attempts = 0; + let delivered = retry_delivery(deadline, &cancel, || { + attempts += 1; + std::future::ready(if attempts == 1 { + classify_delivery_attempt(TelemetryDelivery::Rejected { + status: 409, + reason: Some("BUNDLE_NOT_READY".to_owned()), + }) + } else { + DeliveryAttempt::Accepted + }) + }) + .await; + assert!(delivered); + assert_eq!(attempts, 2); + } + + #[tokio::test(start_paused = true)] + async fn result_delivery_stops_at_job_expiry() { + let cancel = CancellationToken::new(); + let mut attempts = 0; + let delivered = retry_delivery(tokio::time::Instant::now() + Duration::from_secs(2), &cancel, || { + attempts += 1; + std::future::ready(DeliveryAttempt::Retry) + }) + .await; + assert!(!delivered); + assert_eq!(attempts, 2); + } + + #[tokio::test(start_paused = true)] + async fn result_delivery_stops_on_shutdown() { + let cancel = CancellationToken::new(); + let attempts = std::cell::Cell::new(0); + let delivery = retry_delivery(tokio::time::Instant::now() + Duration::from_secs(30), &cancel, || { + attempts.set(attempts.get() + 1); + std::future::ready(DeliveryAttempt::Retry) + }); + tokio::pin!(delivery); + tokio::select! { + delivered = &mut delivery => panic!("delivery stopped before shutdown: {delivered}"), + () = tokio::task::yield_now() => {} + } + assert_eq!(attempts.get(), 1); + cancel.cancel(); + let delivered = delivery.await; + assert!(!delivered); + assert_eq!(attempts.get(), 1); + } + + #[test] + fn durable_state_preserves_single_execution_terminal_result() { + let temporary = tempfile::tempdir().expect("temporary directory"); + let store = JobStateStore { + directory: temporary.path().join("diagnostic-jobs"), + }; + let job_id = Uuid::now_v7().to_string(); + store.save(&job_id, &JobState::Active).expect("active state"); + assert!(matches!(store.load(&job_id), Ok(Some(JobState::Active)))); + + let expected = execution(&job_id, b"bounded artifact"); + store + .save(&job_id, &JobState::Completed(expected.clone())) + .expect("completed state"); + let loaded = match store.load(&job_id).expect("load state") { + Some(JobState::Completed(result)) => result, + _ => panic!("completed state expected"), + }; + assert_eq!(loaded, expected); + assert!(valid_execution(&loaded, &job_id)); + + let uploaded = UploadedDiagnosticJobResult { + job_id: job_id.clone(), + outcome: "SUCCEEDED".to_owned(), + reason: "COMPLETE".to_owned(), + artifact_name: Some(format!( + "organizations/{}/clusters/{}/supportBundles/{}", + Uuid::now_v7(), + Uuid::now_v7(), + Uuid::now_v7() + )), + artifact_sha256: Some("a".repeat(64)), + }; + store.save(&job_id, &JobState::Uploaded(uploaded)).expect("uploaded state"); + let loaded = match store.load(&job_id).expect("load state") { + Some(JobState::Uploaded(result)) => result, + _ => panic!("uploaded state expected"), + }; + assert!(valid_uploaded_result(&loaded, &job_id)); + + store.save(&job_id, &JobState::Delivered).expect("delivered state"); + assert!(matches!(store.load(&job_id), Ok(Some(JobState::Delivered)))); + } + + #[test] + fn persisted_result_must_match_job_and_artifact_digest() { + let job_id = Uuid::now_v7().to_string(); + let mut result = execution(&job_id, b"artifact"); + assert!(valid_execution(&result, &job_id)); + assert!(!valid_execution(&result, &Uuid::now_v7().to_string())); + + result.artifact_sha256 = Some("0".repeat(64)); + assert!(!valid_execution(&result, &job_id)); + result.artifact_bytes = Some(vec![0; MAX_ARTIFACT_BYTES + 1]); + assert!(!valid_execution(&result, &job_id)); + } + + #[cfg(unix)] + #[test] + fn state_loader_rejects_group_readable_files() { + let temporary = tempfile::tempdir().expect("temporary directory"); + let store = JobStateStore { + directory: temporary.path().join("diagnostic-jobs"), + }; + let job_id = Uuid::now_v7().to_string(); + store.save(&job_id, &JobState::Active).expect("active state"); + fs::set_permissions(store.path(&job_id), fs::Permissions::from_mode(0o640)).expect("permissions"); + assert!(store.load(&job_id).is_err()); + } +} diff --git a/rustfs/src/connect/diagnostics/logs.rs b/rustfs/src/connect/diagnostics/logs.rs new file mode 100644 index 000000000..b2138d020 --- /dev/null +++ b/rustfs/src/connect/diagnostics/logs.rs @@ -0,0 +1,878 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded capture of allow-listed RustFS structured log events. +//! +//! The collector reads only the active file configured for RustFS local JSON +//! logging. It exports timestamp offsets, severity, and one of three reviewed +//! event IDs. Raw messages, paths, headers, fields, and malformed lines are +//! counted as dropped and never enter the signed L3 artifact. + +use std::fs::{self, File, OpenOptions}; +use std::io::{Cursor, Read as _, Seek as _, SeekFrom, Write as _}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use base64_simd::URL_SAFE_NO_PAD; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use serde::Serialize; +use serde_json::Value; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use crate::connect::DeviceIdentity; + +pub const LOGS_SCHEMA_VERSION: u16 = 1; +pub const LOGS_CAPABILITY: &str = "logs.capture@1"; +pub const MAX_CAPTURE_DURATION: Duration = Duration::from_secs(30); +pub const MAX_EVENTS: usize = 1_024; +pub const MAX_RESULT_BYTES: usize = 262_144; +pub const MAX_SOURCE_BYTES: usize = 1_048_576; +pub const MAX_LINE_BYTES: usize = 4_096; +pub const MAX_BUILD_FEATURES: usize = 64; +pub const MAX_ARCHIVE_BYTES: usize = 524_288; +pub const MAX_DECOMPRESSED_BYTES: usize = 278_528; +pub const MAX_ENVELOPE_BYTES: usize = 16_384; + +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1\0"; +const ENVELOPE_PATH: &str = "envelope.json"; +const SIGNATURE_PATH: &str = "envelope.sig"; +const RESULT_PATH: &str = "result.json"; +const OUTPUT_MODE: u32 = 0o600; +const MAX_VALIDITY_SECONDS: i64 = 2_592_000; +const MAX_FUTURE_SKEW_SECONDS: i64 = 300; +const POLL_INTERVAL: Duration = Duration::from_millis(50); + +static LOG_COLLECTOR_ACTIVE: AtomicBool = AtomicBool::new(false); + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CaptureMode { + Batch, + Live, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LocalLogConsent { + pub consent_uid: String, + pub policy_revision: u64, + pub expires_at_unix: i64, + pub confirmed: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct LogProvenance { + repository: &'static str, + source_commit: String, + executable_sha256: String, + rustfs_version: String, + os_family: OsFamily, + architecture: Architecture, + build_features: Vec, +} + +impl LogProvenance { + pub fn new( + source_commit: impl Into, + executable_sha256: impl Into, + rustfs_version: impl Into, + build_features: Vec, + ) -> Self { + Self { + repository: "rustfs/rustfs", + source_commit: source_commit.into(), + executable_sha256: executable_sha256.into(), + rustfs_version: rustfs_version.into(), + os_family: OsFamily::current(), + architecture: Architecture::current(), + build_features, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum OsFamily { + Linux, + Darwin, + Windows, + Freebsd, + Other, +} + +impl OsFamily { + fn current() -> Self { + match std::env::consts::OS { + "linux" => Self::Linux, + "macos" => Self::Darwin, + "windows" => Self::Windows, + "freebsd" => Self::Freebsd, + _ => Self::Other, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +enum Architecture { + #[serde(rename = "x86_64")] + X86_64, + Aarch64, + Other, +} + +impl Architecture { + fn current() -> Self { + match std::env::consts::ARCH { + "x86_64" => Self::X86_64, + "aarch64" => Self::Aarch64, + _ => Self::Other, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LogCaptureRequest { + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub schema_version: u16, + pub capability: String, + pub consent: LocalLogConsent, + pub produced_at_unix: i64, + pub expires_at_unix: i64, + pub nonce: [u8; 32], + pub mode: CaptureMode, + pub duration: Duration, + pub max_events: usize, + pub provenance: LogProvenance, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum LogOutcome { + Succeeded, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum LogReasonCode { + Complete, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum LogSeverity { + Info, + Warn, + Error, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum LogEventId { + DriveUnavailable, + RequestFailed, + ServiceStarted, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct CapturedLogEvent { + offset_millis: u64, + severity: LogSeverity, + event_id: LogEventId, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +struct LogData { + events: Vec, + dropped_event_count: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +struct Coverage { + requested_units: u32, + completed_units: u32, + unit: &'static str, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +struct LogResult { + schema_version: u16, + run_uid: String, + tool_id: &'static str, + capability: &'static str, + outcome: LogOutcome, + reason_code: LogReasonCode, + duration_millis: u64, + provenance: LogProvenance, + coverage: Coverage, + data: LogData, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SignedLogExport { + pub artifact_uid: String, + pub archive_bytes: Vec, + pub archive_sha256: String, + pub event_count: usize, + pub dropped_event_count: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SavedLogExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +#[derive(Debug, Error)] +pub enum LogCaptureError { + #[error("logs_local_consent_required")] + ConsentRequired, + #[error("logs_local_consent_expired")] + ConsentExpired, + #[error("logs_request_expired")] + Expired, + #[error("logs_invalid_request")] + InvalidRequest, + #[error("logs_unsupported_version")] + UnsupportedVersion, + #[error("logs_unsupported_capability")] + UnsupportedCapability, + #[error("logs_limit_exceeded")] + LimitExceeded, + #[error("logs_collection_cancelled")] + Cancelled, + #[error("logs_collection_already_running")] + Busy, + #[error("logs_file_source_unavailable")] + SourceUnavailable, + #[error("logs_export_signing_failed")] + Signing, + #[error("logs_export_exists")] + AlreadyExists, + #[error("logs_io_failed")] + Io(#[source] std::io::Error), + #[error("logs_export_encoding_failed")] + Encoding, + #[error("logs_export_durability_failed_after_commit")] + DurabilityAfterCommit(#[source] std::io::Error), +} + +#[derive(Clone, Debug)] +pub(crate) struct ConfiguredLogSource { + path: PathBuf, +} + +impl ConfiguredLogSource { + pub(crate) fn discover() -> Result { + let config = rustfs_obs::OtelConfig::new(); + let directory = config.log_directory.ok_or(LogCaptureError::SourceUnavailable)?; + let filename = config.log_filename.ok_or(LogCaptureError::SourceUnavailable)?; + Self::new(directory, filename) + } + + pub(crate) fn new(directory: impl AsRef, filename: impl AsRef) -> Result { + let filename = filename.as_ref(); + if filename.is_absolute() || filename.file_name() != Some(filename.as_os_str()) { + return Err(LogCaptureError::SourceUnavailable); + } + let directory = fs::canonicalize(directory).map_err(|_| LogCaptureError::SourceUnavailable)?; + let path = directory.join(filename); + let metadata = fs::symlink_metadata(&path).map_err(|_| LogCaptureError::SourceUnavailable)?; + if metadata.file_type().is_symlink() || !metadata.is_file() { + return Err(LogCaptureError::SourceUnavailable); + } + let canonical = fs::canonicalize(&path).map_err(|_| LogCaptureError::SourceUnavailable)?; + if canonical.parent() != Some(directory.as_path()) { + return Err(LogCaptureError::SourceUnavailable); + } + Ok(Self { path: canonical }) + } + + fn len(&self) -> Result { + fs::metadata(&self.path) + .map(|metadata| metadata.len()) + .map_err(LogCaptureError::Io) + } + + fn read(&self, start: u64, limit: usize) -> Result { + let mut file = open_read_only(&self.path)?; + let length = file.metadata().map_err(LogCaptureError::Io)?.len(); + let start = start.min(length); + file.seek(SeekFrom::Start(start)).map_err(LogCaptureError::Io)?; + let remaining = usize::try_from(length.saturating_sub(start)).unwrap_or(usize::MAX); + let mut bytes = Vec::with_capacity(limit.min(remaining)); + file.take(limit as u64).read_to_end(&mut bytes).map_err(LogCaptureError::Io)?; + let bytes_read = bytes.len() as u64; + Ok(SourceChunk { + bytes, + next_offset: start.saturating_add(bytes_read), + truncated_prefix: false, + }) + } +} + +struct SourceChunk { + bytes: Vec, + next_offset: u64, + truncated_prefix: bool, +} + +struct CollectorLease; + +impl CollectorLease { + fn acquire() -> Result { + LOG_COLLECTOR_ACTIVE + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map(|_| Self) + .map_err(|_| LogCaptureError::Busy) + } +} + +impl Drop for CollectorLease { + fn drop(&mut self) { + LOG_COLLECTOR_ACTIVE.store(false, Ordering::Release); + } +} + +pub async fn export_logs( + request: &LogCaptureRequest, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + check_cancel(cancel)?; + let source = ConfiguredLogSource::discover()?; + export_logs_from(request, key, cancel, &source).await +} + +pub(crate) async fn export_logs_from( + request: &LogCaptureRequest, + key: &DeviceIdentity, + cancel: &CancellationToken, + source: &ConfiguredLogSource, +) -> Result { + request.validate(unix_now()?)?; + check_cancel(cancel)?; + let _lease = CollectorLease::acquire()?; + let started = Instant::now(); + let capture = match request.mode { + CaptureMode::Batch => capture_batch(source, cancel)?, + CaptureMode::Live => capture_live(source, request.duration, cancel).await?, + }; + check_cancel(cancel)?; + + let (events, dropped_event_count) = parse_events(capture.bytes, capture.truncated_prefix, request); + let result = LogResult { + schema_version: LOGS_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: "logs.capture", + capability: LOGS_CAPABILITY, + outcome: LogOutcome::Succeeded, + reason_code: LogReasonCode::Complete, + duration_millis: u64::try_from(started.elapsed().as_millis()) + .unwrap_or(u64::MAX) + .clamp(1, 30_000), + provenance: request.provenance.clone(), + coverage: Coverage { + requested_units: 1, + completed_units: 1, + unit: "WINDOW", + }, + data: LogData { + events, + dropped_event_count, + }, + }; + encode_signed_export(request, result, key, cancel) +} + +fn capture_batch(source: &ConfiguredLogSource, cancel: &CancellationToken) -> Result { + check_cancel(cancel)?; + let length = source.len()?; + let start = length.saturating_sub(MAX_SOURCE_BYTES as u64); + let mut chunk = source.read(start, MAX_SOURCE_BYTES)?; + chunk.truncated_prefix = start > 0; + Ok(chunk) +} + +async fn capture_live( + source: &ConfiguredLogSource, + duration: Duration, + cancel: &CancellationToken, +) -> Result { + let deadline = Instant::now() + duration; + let mut offset = source.len()?; + let mut bytes = Vec::new(); + let mut truncated_prefix = false; + loop { + check_cancel(cancel)?; + let length = source.len()?; + if length < offset { + offset = 0; + truncated_prefix = true; + } + if length > offset { + let remaining = MAX_SOURCE_BYTES.saturating_sub(bytes.len()); + if remaining == 0 { + truncated_prefix = true; + break; + } + let chunk = source.read(offset, remaining)?; + offset = chunk.next_offset; + bytes.extend_from_slice(&chunk.bytes); + truncated_prefix |= chunk.truncated_prefix; + } + if Instant::now() >= deadline { + break; + } + tokio::select! { + () = cancel.cancelled() => return Err(LogCaptureError::Cancelled), + () = tokio::time::sleep(POLL_INTERVAL.min(deadline.saturating_duration_since(Instant::now()))) => {} + } + } + Ok(SourceChunk { + bytes, + next_offset: offset, + truncated_prefix, + }) +} + +fn parse_events(bytes: Vec, truncated_prefix: bool, request: &LogCaptureRequest) -> (Vec, u64) { + let mut dropped = u64::from(truncated_prefix); + let mut parsed = Vec::new(); + let batch_end_millis = request.produced_at_unix.saturating_mul(1_000); + let batch_start_millis = + batch_end_millis.saturating_sub(i64::try_from(request.duration.as_millis()).unwrap_or(i64::MAX).max(1)); + for (index, line) in bytes.split(|byte| *byte == b'\n').enumerate() { + if line.is_empty() || (truncated_prefix && index == 0) { + continue; + } + if line.len() > MAX_LINE_BYTES { + dropped = dropped.saturating_add(1); + continue; + } + let Ok(value) = serde_json::from_slice::(line) else { + dropped = dropped.saturating_add(1); + continue; + }; + let Some(event) = typed_event(&value) else { + dropped = dropped.saturating_add(1); + continue; + }; + if request.mode == CaptureMode::Batch + && (event.offset_millis < u64::try_from(batch_start_millis).unwrap_or(0) + || event.offset_millis > u64::try_from(batch_end_millis).unwrap_or(0)) + { + dropped = dropped.saturating_add(1); + continue; + } + if parsed.len() == request.max_events { + dropped = dropped.saturating_add(1); + continue; + } + parsed.push(event); + } + let base = parsed.first().map_or(0, |event| event.offset_millis); + for event in &mut parsed { + event.offset_millis = event.offset_millis.saturating_sub(base); + } + (parsed, dropped) +} + +fn typed_event(value: &Value) -> Option { + let fields = value.as_object()?; + let timestamp = fields.get("timestamp")?.as_str()?; + let timestamp = OffsetDateTime::parse(timestamp, &Rfc3339).ok()?; + let timestamp_millis = u64::try_from(timestamp.unix_timestamp_nanos() / 1_000_000).ok()?; + let severity = match fields.get("level")?.as_str()? { + "INFO" => LogSeverity::Info, + "WARN" => LogSeverity::Warn, + "ERROR" => LogSeverity::Error, + _ => return None, + }; + let event_id = match fields.get("event")?.as_str()? { + "drive_unavailable" => LogEventId::DriveUnavailable, + "rpc_request_failed" | "admin_request_failed" | "http_request_failed" => LogEventId::RequestFailed, + "http_startup_endpoints" => LogEventId::ServiceStarted, + _ => return None, + }; + Some(CapturedLogEvent { + offset_millis: timestamp_millis, + severity, + event_id, + }) +} + +fn encode_signed_export( + request: &LogCaptureRequest, + result: LogResult, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + if unix_now()? >= request.expires_at_unix { + return Err(LogCaptureError::Expired); + } + let event_count = result.data.events.len(); + let dropped_event_count = result.data.dropped_event_count; + let result_bytes = serde_json::to_vec(&result).map_err(|_| LogCaptureError::Encoding)?; + if result_bytes.is_empty() || result_bytes.len() > MAX_RESULT_BYTES { + return Err(LogCaptureError::LimitExceeded); + } + let device_key_id = hex_lower(&Sha256::digest(key.public_key_der())); + let envelope = Envelope { + format_version: "rustfs.connect.diagnosticEnvelope/1", + protocol_version: "v1", + organization_name: &request.organization_name, + cluster_name: &request.cluster_name, + device_name: &request.device_name, + run_uid: &request.run_uid, + artifact_uid: &request.artifact_uid, + tool_id: "logs.capture", + schema_version: LOGS_SCHEMA_VERSION, + classification: "L3", + consent_uid: &request.consent.consent_uid, + policy_revision: request.consent.policy_revision, + produced_at: timestamp(request.produced_at_unix)?, + expires_at: timestamp(request.expires_at_unix)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(request.nonce), + device_key_id: &device_key_id, + payload: Payload { + path: RESULT_PATH, + media_type: "application/json", + size_bytes: result_bytes.len() as u64, + sha256: hex_lower(&Sha256::digest(&result_bytes)), + }, + }; + let envelope_bytes = serde_json::to_vec(&envelope).map_err(|_| LogCaptureError::Encoding)?; + if envelope_bytes.is_empty() || envelope_bytes.len() > MAX_ENVELOPE_BYTES { + return Err(LogCaptureError::LimitExceeded); + } + let signature_bytes = signature_document(key, &device_key_id, &envelope_bytes)?; + let decompressed = result_bytes + .len() + .checked_add(envelope_bytes.len()) + .and_then(|size| size.checked_add(signature_bytes.len())) + .ok_or(LogCaptureError::LimitExceeded)?; + if decompressed > MAX_DECOMPRESSED_BYTES { + return Err(LogCaptureError::LimitExceeded); + } + check_cancel(cancel)?; + let archive_bytes = archive(&envelope_bytes, &signature_bytes, &result_bytes)?; + if archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(LogCaptureError::LimitExceeded); + } + Ok(SignedLogExport { + artifact_uid: request.artifact_uid.clone(), + archive_sha256: hex_lower(&Sha256::digest(&archive_bytes)), + archive_bytes, + event_count, + dropped_event_count, + }) +} + +pub fn save_signed_log_export( + output: &Path, + export: &SignedLogExport, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + if !uuid7(&export.artifact_uid) { + return Err(LogCaptureError::InvalidRequest); + } + let parent = output + .parent() + .filter(|path| !path.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let filename = output.file_name().ok_or(LogCaptureError::InvalidRequest)?.to_string_lossy(); + let temporary = parent.join(format!(".{filename}.{}.partial", export.artifact_uid)); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + let mut file = options.open(&temporary).map_err(map_create_error)?; + let result = (|| { + file.write_all(&export.archive_bytes).map_err(LogCaptureError::Io)?; + check_cancel(cancel)?; + file.sync_all().map_err(LogCaptureError::Io)?; + fs::hard_link(&temporary, output).map_err(map_publish_error)?; + if let Err(error) = fs::remove_file(&temporary) { + return Err(LogCaptureError::DurabilityAfterCommit(error)); + } + #[cfg(unix)] + if let Err(error) = File::open(parent).and_then(|directory| directory.sync_all()) { + return Err(LogCaptureError::DurabilityAfterCommit(error)); + } + Ok(SavedLogExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: export.archive_bytes.len() as u64, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if result.is_err() { + let _ = fs::remove_file(&temporary); + } + result +} + +impl LogCaptureRequest { + fn validate(&self, now_unix: i64) -> Result<(), LogCaptureError> { + if self.schema_version != LOGS_SCHEMA_VERSION { + return Err(LogCaptureError::UnsupportedVersion); + } + if self.capability != LOGS_CAPABILITY { + return Err(LogCaptureError::UnsupportedCapability); + } + if !self.consent.confirmed || self.consent.policy_revision == 0 { + return Err(LogCaptureError::ConsentRequired); + } + if self.consent.expires_at_unix <= now_unix || self.expires_at_unix > self.consent.expires_at_unix { + return Err(LogCaptureError::ConsentExpired); + } + let validity = self + .expires_at_unix + .checked_sub(self.produced_at_unix) + .ok_or(LogCaptureError::Expired)?; + if self.produced_at_unix > now_unix.saturating_add(MAX_FUTURE_SKEW_SECONDS) + || validity <= 0 + || self.expires_at_unix <= now_unix + || validity > MAX_VALIDITY_SECONDS + { + return Err(LogCaptureError::Expired); + } + if self.duration.is_zero() || self.duration > MAX_CAPTURE_DURATION || self.max_events == 0 || self.max_events > MAX_EVENTS + { + return Err(LogCaptureError::LimitExceeded); + } + if !uuid7(&self.run_uid) + || !uuid7(&self.artifact_uid) + || !uuid7(&self.consent.consent_uid) + || !resource_names_match(self) + || !lower_hex(&self.provenance.source_commit, 40) + || !lower_hex(&self.provenance.executable_sha256, 64) + || !version(&self.provenance.rustfs_version) + || self.provenance.build_features.len() > MAX_BUILD_FEATURES + || !self.provenance.build_features.iter().all(|feature| build_feature(feature)) + { + return Err(LogCaptureError::InvalidRequest); + } + Ok(()) + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct Envelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: &'static str, + schema_version: u16, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: &'a str, + payload: Payload, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct Payload { + path: &'static str, + media_type: &'static str, + size_bytes: u64, + sha256: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct SignatureDocument<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +fn signature_document(key: &DeviceIdentity, key_id: &str, envelope: &[u8]) -> Result, LogCaptureError> { + let pkcs8 = key.to_pkcs8_der().map_err(|_| LogCaptureError::Signing)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| LogCaptureError::Signing)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + envelope.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.extend_from_slice(envelope); + let signature: Signature = signing_key.sign(&input); + let value = URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()); + serde_json::to_vec(&SignatureDocument { + algorithm: "ES256", + key_id, + value, + }) + .map_err(|_| LogCaptureError::Encoding) +} + +fn archive(envelope: &[u8], signature: &[u8], result: &[u8]) -> Result, LogCaptureError> { + let cursor = Cursor::new(Vec::with_capacity(envelope.len() + signature.len() + result.len() + 512)); + let mut writer = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(OUTPUT_MODE); + for (name, bytes) in [(ENVELOPE_PATH, envelope), (SIGNATURE_PATH, signature), (RESULT_PATH, result)] { + writer.start_file(name, options).map_err(|_| LogCaptureError::Encoding)?; + writer.write_all(bytes).map_err(LogCaptureError::Io)?; + } + writer + .finish() + .map(|cursor| cursor.into_inner()) + .map_err(|_| LogCaptureError::Encoding) +} + +fn open_read_only(path: &Path) -> Result { + let mut options = OpenOptions::new(); + options.read(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.custom_flags(libc::O_NOFOLLOW); + } + options.open(path).map_err(LogCaptureError::Io) +} + +fn resource_names_match(request: &LogCaptureRequest) -> bool { + let Some(organization_uid) = request.organization_name.strip_prefix("organizations/") else { + return false; + }; + if !uuid7(organization_uid) { + return false; + } + let cluster_prefix = format!("{}/clusters/", request.organization_name); + let Some(cluster_uid) = request.cluster_name.strip_prefix(&cluster_prefix) else { + return false; + }; + if !uuid7(cluster_uid) { + return false; + } + let device_prefix = format!("{}/clusterDevices/", request.cluster_name); + request.device_name.strip_prefix(&device_prefix).is_some_and(uuid7) +} + +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn lower_hex(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn version(value: &str) -> bool { + if value.is_empty() + || value.len() > 64 + || !value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'-')) + { + return false; + } + let (core, suffix) = value + .split_once('-') + .map_or((value, None), |(core, suffix)| (core, Some(suffix))); + if suffix.is_some_and(str::is_empty) { + return false; + } + let mut parts = core.split('.'); + parts.clone().count() == 3 && parts.all(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) +} + +fn build_feature(value: &str) -> bool { + !value.is_empty() + && value.len() <= 64 + && value.as_bytes()[0].is_ascii_lowercase() + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'_' | b'-')) +} + +fn timestamp(unix: i64) -> Result { + OffsetDateTime::from_unix_timestamp(unix) + .map_err(|_| LogCaptureError::InvalidRequest)? + .format(&Rfc3339) + .map_err(|_| LogCaptureError::InvalidRequest) +} + +fn unix_now() -> Result { + let duration = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| LogCaptureError::InvalidRequest)?; + i64::try_from(duration.as_secs()).map_err(|_| LogCaptureError::InvalidRequest) +} + +fn check_cancel(cancel: &CancellationToken) -> Result<(), LogCaptureError> { + if cancel.is_cancelled() { + Err(LogCaptureError::Cancelled) + } else { + Ok(()) + } +} + +fn hex_lower(bytes: &[u8]) -> String { + let mut value = String::with_capacity(bytes.len() * 2); + for byte in bytes { + use std::fmt::Write as _; + write!(&mut value, "{byte:02x}").expect("writing hexadecimal to a string cannot fail"); + } + value +} + +fn map_create_error(error: std::io::Error) -> LogCaptureError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + LogCaptureError::AlreadyExists + } else { + LogCaptureError::Io(error) + } +} + +fn map_publish_error(error: std::io::Error) -> LogCaptureError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + LogCaptureError::AlreadyExists + } else { + LogCaptureError::Io(error) + } +} diff --git a/rustfs/src/connect/diagnostics/mod.rs b/rustfs/src/connect/diagnostics/mod.rs new file mode 100644 index 000000000..e4ec5241f --- /dev/null +++ b/rustfs/src/connect/diagnostics/mod.rs @@ -0,0 +1,165 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +mod inspect; +mod job; +pub(crate) mod job_delivery; +mod logs; +mod perf_client; +mod perf_drive; +mod perf_network; +mod perf_object; +mod perf_site_replication; +mod profile_cpu; +mod profile_memory; +mod profile_threads; +mod receipt_delivery; +mod schedule; +mod top_api; +mod top_disk; +mod top_locks; +mod top_net; +mod top_rpc; +mod trace_analysis; +mod trace_otlp; +mod trace_record; +mod trace_replay; +#[cfg(unix)] +mod trace_runtime; + +/// Signed diagnostic producers available through the CLI or authenticated service jobs. +pub const CONNECT_DIAGNOSTIC_CAPABILITIES: &[&str] = &[ + perf_client::CLIENT_CAPABILITY, + perf_drive::DRIVE_CAPABILITY, + perf_network::NETWORK_CAPABILITY, + perf_object::OBJECT_CAPABILITY, + perf_site_replication::SITE_REPLICATION_CAPABILITY, + logs::LOGS_CAPABILITY, + profile_cpu::CPU_PROFILE_CAPABILITY, + profile_cpu::MEMORY_PROFILE_CAPABILITY, + profile_cpu::THREAD_PROFILE_CAPABILITY, + trace_record::TELEMETRY_RECORD_CAPABILITY, + trace_record::TELEMETRY_OTLP_CAPABILITY, + trace_record::TELEMETRY_REPLAY_CAPABILITY, + top_api::TOP_API_CAPABILITY, + top_disk::TOP_DISK_CAPABILITY, + top_locks::TOP_LOCKS_CAPABILITY, + top_net::TOP_NET_CAPABILITY, + top_rpc::TOP_RPC_CAPABILITY, + inspect::INSPECT_CAPABILITY, +]; +#[cfg(not(unix))] +#[path = "trace_runtime_unsupported.rs"] +mod trace_runtime; + +pub use inspect::{ + INSPECT_CAPABILITY, INSPECT_SCHEMA_VERSION, InspectArtifactConsent, InspectDiagnosticResult, InspectError, InspectFinding, + InspectOutcome, InspectProvenance, InspectReason, InspectReasonCode, InspectRequest, InspectRule, InspectRuleOutcome, + InspectRun, Reconstruction, SavedInspectExport, SignedInspectExport, export_inspect_summary, save_signed_inspect_export, +}; +pub use job::{ + DIAGNOSTIC_JOB_SIGNATURE_DOMAIN, DiagnosticJobEnvelope, DiagnosticJobError, DiagnosticJobExecution, DiagnosticJobLimits, + DiagnosticJobParameters, DiagnosticJobTarget, TrustedDiagnosticJobSigner, VerifiedDiagnosticJob, execute_diagnostic_job, +}; +pub use logs::{ + CaptureMode, LOGS_CAPABILITY, LOGS_SCHEMA_VERSION, LocalLogConsent, LogCaptureError, LogCaptureRequest, LogProvenance, + SavedLogExport, SignedLogExport, export_logs, save_signed_log_export, +}; +pub use perf_client::{ + CLIENT_CAPABILITY, CLIENT_SCHEMA_VERSION, ClientDiagnosticResult, ClientMeasurement, ClientOperation, ClientOutcome, + ClientPerformanceData, ClientPerformanceError, ClientPerformanceRequest, ClientProbe, ClientProbeError, ClientProbeFuture, + ClientProbeMeasurement, ClientProvenance, ClientReasonCode, ClientTargetParameters, ClientTargetReasonCode, + ClientTargetResult, ClientTargetUnits, HttpClientProbe, LocalClientConsent, SavedClientExport, SignedClientExport, + measure_client, read_protected_client_credential, save_signed_client_export, sign_client_export, validate_client_limits, +}; +pub use perf_drive::{ + DRIVE_CAPABILITY, DRIVE_SCHEMA_VERSION, DriveDiagnosticResult, DriveMeasurement, DriveOutcome, DrivePerformanceData, + DrivePerformanceError, DrivePerformanceRequest, DriveProvenance, DriveReadMode, DriveReasonCode, DriveTargetParameters, + DriveTargetReasonCode, DriveTargetResult, DriveTargetUnits, LocalDriveConsent, SavedDriveExport, SignedDriveExport, + measure_drive, save_signed_drive_export, sign_drive_export, validate_drive_limits, +}; +pub(crate) use perf_network::runtime_network_peer_aliases; +pub use perf_network::{ + LocalNetworkConsent, MAX_ARCHIVE_BYTES as MAX_NETWORK_ARCHIVE_BYTES, + MAX_BANDWIDTH_BYTES_PER_SECOND as MAX_NETWORK_BANDWIDTH_BYTES_PER_SECOND, + MAX_DECOMPRESSED_BYTES as MAX_NETWORK_DECOMPRESSED_BYTES, MAX_ENVELOPE_BYTES as MAX_NETWORK_ENVELOPE_BYTES, + MAX_NETWORK_DURATION, MAX_OPERATIONS as MAX_NETWORK_OPERATIONS, MAX_PEERS as MAX_NETWORK_PEERS, + MAX_RESULT_BYTES as MAX_NETWORK_RESULT_BYTES, MAX_TRAFFIC_BYTES as MAX_NETWORK_TRAFFIC_BYTES, NETWORK_CAPABILITY, + NETWORK_SCHEMA_VERSION, NETWORK_TOOL_ID, NetworkCoverage, NetworkDiagnosticResult, NetworkMeasurement, NetworkOutcome, + NetworkPeerHarness, NetworkPeerResult, NetworkPerformanceData, NetworkPerformanceError, NetworkPerformanceRequest, + NetworkProvenance, NetworkReasonCode, PeerProbeError, PeerProbeFuture, PeerProbeMeasurement, PeerReasonCode, + SavedNetworkExport, SignedNetworkExport, measure_network, measure_network_with_harness, save_signed_network_export, + sign_network_export, +}; +pub use perf_object::{ + LocalObjectConsent, MAX_OBJECT_BANDWIDTH_BYTES_PER_SECOND, MAX_OBJECT_DURATION, MAX_OBJECT_RESULT_BYTES, + MAX_OBJECT_TRAFFIC_BYTES, OBJECT_CAPABILITY, OBJECT_SCHEMA_VERSION, OBJECT_TOOL_ID, ObjectDiagnosticResult, + ObjectMeasurement, ObjectOperation, ObjectOutcome, ObjectPerformanceData, ObjectPerformanceError, ObjectPerformanceRequest, + ObjectProbe, ObjectProbeError, ObjectProbeFuture, ObjectProbeMeasurement, ObjectProvenance, ObjectReasonCode, + ObjectTargetParameters, ObjectTargetReasonCode, ObjectTargetResult, ObjectTargetUnits, S3ObjectProbe, SavedObjectExport, + SignedObjectExport, measure_object, read_protected_object_credential, save_signed_object_export, sign_object_export, + validate_object_limits, +}; +pub use perf_site_replication::{ + LocalSiteReplicationConsent, MAX_SITE_REPLICATION_DURATION, MAX_SITE_REPLICATION_TRAFFIC_BYTES, S3SiteReplicationProbe, + SITE_REPLICATION_CAPABILITY, SITE_REPLICATION_SCHEMA_VERSION, SITE_REPLICATION_TOOL_ID, SavedSiteReplicationExport, + SignedSiteReplicationExport, SiteReplicationCredentials, SiteReplicationDiagnosticResult, SiteReplicationEndpoint, + SiteReplicationMeasurement, SiteReplicationOutcome, SiteReplicationPerformanceData, SiteReplicationPerformanceError, + SiteReplicationPerformanceRequest, SiteReplicationProbe, SiteReplicationProbeError, SiteReplicationProbeFuture, + SiteReplicationProbeMeasurement, SiteReplicationProvenance, SiteReplicationReasonCode, SiteReplicationTargetReasonCode, + SiteReplicationTargetResult, measure_site_replication, read_protected_site_replication_credential, + save_signed_site_replication_export, sign_site_replication_export, validate_site_replication_limits, +}; +pub use profile_cpu::{ + CPU_PROFILE_CAPABILITY, LocalProfileConsent, MAX_PROFILE_DURATION, MEMORY_PROFILE_CAPABILITY, PROFILE_SCHEMA_VERSION, + ProfileCaptureRequest, ProfileData, ProfileError, ProfileOutcome, ProfileProvenance, ProfileReasonCode, ProfileResult, + ProfileTool, SavedProfileExport, SignedProfileExport, THREAD_PROFILE_CAPABILITY, ThreadProfileData, ThreadProfileScope, + ThreadState, ThreadStateCount, capture_cpu_profile, encode_signed_profile_export, export_cpu_profile, + save_signed_profile_export, +}; +pub use profile_memory::export_memory_profile; +pub use profile_threads::{capture_thread_profile, export_thread_profile}; +pub(crate) use receipt_delivery::{DiagnosticReceiptDelivery, DiagnosticReceiptSender}; +pub use schedule::{ + DiagnosticCollectionPolicy, DiagnosticReceipt, DiagnosticScheduleError, DiagnosticScheduleRuntime, DiagnosticScheduleStatus, + ReceiptOutcome, run_local_environment_once, spawn_environment_schedule, +}; +pub(crate) use top_api::sign_top_export_with_nonce; +pub use top_api::{ + LocalTopConsent, MAX_TOP_DURATION, MAX_TOP_EXPORT_VALIDITY, SavedTopExport, SignedTopExport, TOP_API_CAPABILITY, + TOP_CLASSIFICATION, TOP_SCHEMA_VERSION, TopApiData, TopApiOperation, TopCaptureError, TopCaptureLimits, TopCaptureRequest, + TopCaptureScope, TopCoverage, TopOutcome, TopProvenance, TopReasonCode, TopResult, capture_top_api, save_signed_top_export, + sign_top_export, +}; +pub use top_disk::{DiskCounterSnapshot, TOP_DISK_CAPABILITY, TopDiskData, capture_top_disk, evaluate_disk_window}; +pub use top_locks::{TOP_LOCKS_CAPABILITY, TopLocksData, capture_top_locks, evaluate_lock_snapshot}; +pub use top_net::{NetworkCounterSnapshot, TOP_NET_CAPABILITY, TopNetData, capture_top_net, evaluate_network_window}; +pub use top_rpc::{TOP_RPC_CAPABILITY, TopRpcData, capture_top_rpc}; +pub use trace_analysis::{OperationSummary, TraceAnalysis, TraceAnalysisError, analyze_trace}; +pub use trace_otlp::{ + LocalOtlpHeaders, MAX_OTLP_BODY_BYTES, OtlpBatch, OtlpForwardError, OtlpReceipt, export_trace_otlp, export_trace_otlp_result, +}; +pub use trace_record::{ + LocalTelemetryConsent, MAX_SAFE_INTEGER, MAX_TELEMETRY_DURATION, MAX_TELEMETRY_RESULT_BYTES, MAX_TELEMETRY_SPANS, + ObservedTelemetrySpan, RecordedTrace, SavedTelemetryExport, SignedTelemetryExport, TELEMETRY_OTLP_CAPABILITY, + TELEMETRY_RECORD_CAPABILITY, TELEMETRY_REPLAY_CAPABILITY, TELEMETRY_SCHEMA_VERSION, TelemetryArtifactConsent, + TelemetryArtifactError, TelemetryArtifactRequest, TelemetryCoverage, TelemetryDiagnosticResult, TelemetryOperation, + TelemetryOutcome, TelemetryProducerError, TelemetryProvenance, TelemetryReasonCode, TelemetrySpan, TelemetrySpanStatus, + TelemetryTool, TraceRecordCapture, TraceRecordCompletion, TraceRecordLimits, encode_signed_telemetry_export, + record_diagnostic_result, record_trace, record_trace_bus, save_signed_telemetry_export, +}; +pub use trace_replay::{LocallyReviewedTraceArtifact, ReplayedTrace, TraceReplayError, replay_trace, replay_trace_result}; +pub(crate) use trace_runtime::{ + LocalTraceCaptureError, LocalTraceCaptureRuntime, request_local_trace_capture, spawn_local_trace_capture_runtime, +}; diff --git a/rustfs/src/connect/diagnostics/perf_client.rs b/rustfs/src/connect/diagnostics/perf_client.rs new file mode 100644 index 000000000..00c27c5fc --- /dev/null +++ b/rustfs/src/connect/diagnostics/perf_client.rs @@ -0,0 +1,1255 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Consent-bound client-to-deployment performance measurement. +//! +//! The producer sends generated bytes to, or reads generated bytes from, the +//! bounded RustFS admin speedtest endpoint. It never reads or writes customer +//! objects and keeps deployment credentials local to the invoking process. + +use std::fs::{self, File, OpenOptions}; +use std::future::Future; +use std::io::{Cursor, Read, Write as _}; +use std::path::Path; +use std::pin::Pin; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use base64_simd::URL_SAFE_NO_PAD; +use bytes::Bytes; +use futures::StreamExt as _; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use reqwest::{Client, Method, Response, StatusCode, Url}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zeroize::Zeroizing; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use crate::connect::DeviceIdentity; + +pub const CLIENT_SCHEMA_VERSION: u16 = 1; +pub const CLIENT_TOOL_ID: &str = "performance.client"; +pub const CLIENT_CAPABILITY: &str = "performance.client@1"; +pub const MAX_CLIENT_DURATION: Duration = Duration::from_secs(30); +pub const MAX_CLIENT_TRAFFIC_BYTES: u64 = 1_048_576; +pub const MAX_CLIENT_BANDWIDTH_BYTES_PER_SECOND: u64 = 1_048_576; +pub const MAX_CLIENT_RESULT_BYTES: usize = 262_144; + +const MAX_SAFE_INTEGER: u64 = 9_007_199_254_740_991; +const MAX_BUILD_FEATURES: usize = 64; +const MAX_VALIDITY_SECONDS: i64 = 2_592_000; +const MAX_FUTURE_SKEW_SECONDS: i64 = 300; +const MAX_ENVELOPE_BYTES: usize = 16_384; +const MAX_ARCHIVE_BYTES: usize = 524_288; +const MAX_DECOMPRESSED_BYTES: usize = 278_528; +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1\0"; +const ENVELOPE_PATH: &str = "envelope.json"; +const SIGNATURE_PATH: &str = "envelope.sig"; +const RESULT_PATH: &str = "result.json"; +const CLIENT_SPEEDTEST_PATH: &str = "/rustfs/admin/v3/speedtest/client/devnull"; +const MAX_CLIENT_RESPONSE_BYTES: usize = 16_384; +const OUTPUT_MODE: u32 = 0o600; + +static CLIENT_COLLECTOR_ACTIVE: AtomicBool = AtomicBool::new(false); + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ClientOperation { + GetObject, + PutObject, +} + +impl ClientOperation { + pub const fn as_str(self) -> &'static str { + match self { + Self::GetObject => "GET_OBJECT", + Self::PutObject => "PUT_OBJECT", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ClientOutcome { + Succeeded, + Failed, + Cancelled, +} + +impl ClientOutcome { + pub const fn as_str(self) -> &'static str { + match self { + Self::Succeeded => "SUCCEEDED", + Self::Failed => "FAILED", + Self::Cancelled => "CANCELLED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ClientReasonCode { + Complete, + SourceUnavailable, + PermissionDenied, + Cancelled, + CollectionFailed, +} + +impl ClientReasonCode { + pub const fn as_str(self) -> &'static str { + match self { + Self::Complete => "COMPLETE", + Self::SourceUnavailable => "SOURCE_UNAVAILABLE", + Self::PermissionDenied => "PERMISSION_DENIED", + Self::Cancelled => "CANCELLED", + Self::CollectionFailed => "COLLECTION_FAILED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ClientTargetReasonCode { + Complete, + EndpointUnavailable, + ProxyFailure, + PermissionDenied, + TimedOut, + Cancelled, + ProtocolFailure, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ClientTargetParameters { + pub operation: ClientOperation, + pub requested_bytes: u64, + pub duration_millis: u64, + pub concurrency: u8, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ClientTargetUnits { + pub bytes: &'static str, + pub duration: &'static str, + pub latency: &'static str, + pub operation_count: &'static str, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ClientProvenance { + repository: &'static str, + source_commit: String, + executable_sha256: String, + rustfs_version: String, + os_family: ClientOsFamily, + architecture: ClientArchitecture, + build_features: Vec, +} + +impl ClientProvenance { + pub fn new( + source_commit: impl Into, + executable_sha256: impl Into, + rustfs_version: impl Into, + build_features: Vec, + ) -> Self { + Self { + repository: "rustfs/rustfs", + source_commit: source_commit.into(), + executable_sha256: executable_sha256.into(), + rustfs_version: rustfs_version.into(), + os_family: ClientOsFamily::current(), + architecture: ClientArchitecture::current(), + build_features, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum ClientOsFamily { + Linux, + Darwin, + Windows, + Freebsd, + Other, +} + +impl ClientOsFamily { + fn current() -> Self { + match std::env::consts::OS { + "linux" => Self::Linux, + "macos" => Self::Darwin, + "windows" => Self::Windows, + "freebsd" => Self::Freebsd, + _ => Self::Other, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +enum ClientArchitecture { + #[serde(rename = "x86_64")] + X86_64, + Aarch64, + Other, +} + +impl ClientArchitecture { + fn current() -> Self { + match std::env::consts::ARCH { + "x86_64" => Self::X86_64, + "aarch64" => Self::Aarch64, + _ => Self::Other, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LocalClientConsent { + pub consent_uid: String, + pub policy_revision: u64, + pub expires_at_unix: i64, + pub confirmed: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ClientPerformanceRequest { + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub schema_version: u16, + pub capability: String, + pub consent: LocalClientConsent, + pub produced_at_unix: i64, + pub expires_at_unix: i64, + pub nonce: [u8; 32], + pub duration: Duration, + pub operation: ClientOperation, + pub traffic_bytes: u64, + pub target_alias: String, + pub provenance: ClientProvenance, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ClientPerformanceData { + pub operation: ClientOperation, + pub transferred_bytes: u64, + pub completed_operations: u64, + pub duration_millis: u64, + pub error_count: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +struct ClientCoverage { + requested_units: u32, + completed_units: u32, + unit: &'static str, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ClientDiagnosticResult { + schema_version: u16, + run_uid: String, + tool_id: &'static str, + capability: &'static str, + outcome: ClientOutcome, + reason_code: ClientReasonCode, + duration_millis: u64, + provenance: ClientProvenance, + coverage: ClientCoverage, + data: Option, +} + +impl ClientDiagnosticResult { + pub fn outcome(&self) -> ClientOutcome { + self.outcome + } + + pub fn reason_code(&self) -> ClientReasonCode { + self.reason_code + } + + pub fn data(&self) -> Option<&ClientPerformanceData> { + self.data.as_ref() + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ClientTargetResult { + pub target_alias: String, + pub outcome: ClientOutcome, + pub reason_code: ClientTargetReasonCode, + pub parameters: ClientTargetParameters, + pub units: ClientTargetUnits, + pub transferred_bytes: u64, + pub completed_operations: u64, + pub latency_micros: Option, + pub duration_millis: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ClientMeasurement { + pub result: ClientDiagnosticResult, + pub target: ClientTargetResult, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct ClientProbeMeasurement { + pub transferred_bytes: u64, + pub duration: Duration, + pub latency: Duration, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ClientProbeError { + EndpointUnavailable, + ProxyFailure, + PermissionDenied, + TimedOut, + Cancelled, + ProtocolFailure, +} + +pub type ClientProbeFuture<'a> = Pin> + Send + 'a>>; + +pub trait ClientProbe: Send + Sync { + fn probe<'a>(&'a self, request: &'a ClientPerformanceRequest, cancel: &'a CancellationToken) -> ClientProbeFuture<'a>; +} + +pub struct HttpClientProbe { + endpoint: Url, + client: Client, + access_key: Zeroizing, + secret_key: Zeroizing, + session_token: Zeroizing, + proxy_configured: bool, +} + +impl HttpClientProbe { + pub fn new( + endpoint: &str, + root_ca_pem: Option<&[u8]>, + proxy: Option<&str>, + access_key: Zeroizing, + secret_key: Zeroizing, + session_token: Zeroizing, + timeout: Duration, + ) -> Result { + let endpoint = deployment_endpoint(endpoint)?; + if access_key.is_empty() || secret_key.is_empty() { + return Err(ClientPerformanceError::InvalidCredential); + } + let mut builder = Client::builder() + .no_proxy() + .redirect(reqwest::redirect::Policy::none()) + .timeout(timeout); + if let Some(root_ca_pem) = root_ca_pem { + let certificate = + reqwest::Certificate::from_pem(root_ca_pem).map_err(|_| ClientPerformanceError::InvalidRootCertificate)?; + builder = builder.add_root_certificate(certificate); + } + if let Some(proxy) = proxy { + let proxy_url = proxy_url(proxy)?; + builder = builder.proxy(reqwest::Proxy::all(proxy_url).map_err(|_| ClientPerformanceError::InvalidProxy)?); + } + let client = builder.build().map_err(|_| ClientPerformanceError::TransportConfiguration)?; + Ok(Self { + endpoint, + client, + access_key, + secret_key, + session_token, + proxy_configured: proxy.is_some(), + }) + } + + async fn execute( + &self, + request: &ClientPerformanceRequest, + cancel: &CancellationToken, + ) -> Result { + let started = Instant::now(); + let traffic_bytes = usize::try_from(request.traffic_bytes).map_err(|_| ClientProbeError::ProtocolFailure)?; + let payload = match request.operation { + ClientOperation::PutObject => Bytes::from(vec![0xa5; traffic_bytes]), + ClientOperation::GetObject => Bytes::new(), + }; + let mut url = self + .endpoint + .join(CLIENT_SPEEDTEST_PATH) + .map_err(|_| ClientProbeError::ProtocolFailure)?; + let method = match request.operation { + ClientOperation::PutObject => Method::POST, + ClientOperation::GetObject => { + url.query_pairs_mut().append_pair("bytes", &request.traffic_bytes.to_string()); + Method::GET + } + }; + let payload_hash = hex_lower(&Sha256::digest(&payload)); + let unsigned = http::Request::builder() + .method(method.clone()) + .uri(url.as_str()) + .header("x-amz-content-sha256", payload_hash) + .body(()) + .map_err(|_| ClientProbeError::ProtocolFailure)?; + let signed_headers = rustfs_signer::try_sign_v4_headers( + unsigned.into_parts().0, + i64::try_from(payload.len()).map_err(|_| ClientProbeError::ProtocolFailure)?, + &self.access_key, + &self.secret_key, + &self.session_token, + "us-east-1", + ) + .map_err(|_| ClientProbeError::ProtocolFailure)?; + let send = self.client.request(method, url).headers(signed_headers).body(payload).send(); + let response = tokio::select! { + () = cancel.cancelled() => return Err(ClientProbeError::Cancelled), + response = send => response.map_err(|error| self.transport_error(&error))?, + }; + let status = response.status(); + if status == StatusCode::UNAUTHORIZED || status == StatusCode::FORBIDDEN { + return Err(ClientProbeError::PermissionDenied); + } + if !status.is_success() { + return Err(ClientProbeError::ProtocolFailure); + } + let transferred_bytes = match request.operation { + ClientOperation::PutObject => { + let body = self.read_response_body(response, MAX_CLIENT_RESPONSE_BYTES, cancel).await?; + let response: ClientDevnullResponse = + serde_json::from_slice(&body).map_err(|_| ClientProbeError::ProtocolFailure)?; + if !response.measured || response.kind != "client-devnull" || response.rx_bytes != request.traffic_bytes { + return Err(ClientProbeError::ProtocolFailure); + } + request.traffic_bytes + } + ClientOperation::GetObject => { + let body = self.read_response_body(response, traffic_bytes, cancel).await?; + let body_len = u64::try_from(body.len()).map_err(|_| ClientProbeError::ProtocolFailure)?; + if body_len != request.traffic_bytes || body.iter().any(|byte| *byte != 0) { + return Err(ClientProbeError::ProtocolFailure); + } + body_len + } + }; + let latency = started.elapsed(); + Ok(ClientProbeMeasurement { + transferred_bytes, + duration: latency, + latency, + }) + } + + async fn read_response_body( + &self, + response: Response, + max_bytes: usize, + cancel: &CancellationToken, + ) -> Result, ClientProbeError> { + let max_bytes_u64 = u64::try_from(max_bytes).map_err(|_| ClientProbeError::ProtocolFailure)?; + if response.content_length().is_some_and(|length| length > max_bytes_u64) { + return Err(ClientProbeError::ProtocolFailure); + } + let mut body = Vec::with_capacity(max_bytes.min(MAX_CLIENT_RESPONSE_BYTES)); + let mut stream = response.bytes_stream(); + loop { + let chunk = tokio::select! { + () = cancel.cancelled() => return Err(ClientProbeError::Cancelled), + chunk = stream.next() => chunk, + }; + let Some(chunk) = chunk else { + break; + }; + let chunk = chunk.map_err(|error| self.transport_error(&error))?; + let new_length = body.len().checked_add(chunk.len()).ok_or(ClientProbeError::ProtocolFailure)?; + if new_length > max_bytes { + return Err(ClientProbeError::ProtocolFailure); + } + body.extend_from_slice(&chunk); + } + Ok(body) + } + + fn transport_error(&self, error: &reqwest::Error) -> ClientProbeError { + if error.is_timeout() { + ClientProbeError::TimedOut + } else if self.proxy_configured { + ClientProbeError::ProxyFailure + } else if error.is_connect() { + ClientProbeError::EndpointUnavailable + } else { + ClientProbeError::ProtocolFailure + } + } +} + +impl ClientProbe for HttpClientProbe { + fn probe<'a>(&'a self, request: &'a ClientPerformanceRequest, cancel: &'a CancellationToken) -> ClientProbeFuture<'a> { + Box::pin(self.execute(request, cancel)) + } +} + +#[derive(Deserialize)] +struct ClientDevnullResponse { + kind: String, + measured: bool, + rx_bytes: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SignedClientExport { + pub artifact_uid: String, + pub outcome: ClientOutcome, + pub reason_code: ClientReasonCode, + pub envelope_json: Vec, + pub envelope_signature: Vec, + pub result_json: Vec, + pub archive_bytes: Vec, + pub archive_sha256: String, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SavedClientExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +#[derive(Debug, Error)] +pub enum ClientPerformanceError { + #[error("client_performance_local_consent_required")] + ConsentRequired, + #[error("client_performance_local_consent_expired")] + ConsentExpired, + #[error("client_performance_request_expired")] + Expired, + #[error("client_performance_invalid_request")] + InvalidRequest, + #[error("client_performance_unsupported_version")] + UnsupportedVersion, + #[error("client_performance_unsupported_capability")] + UnsupportedCapability, + #[error("client_performance_limit_exceeded")] + LimitExceeded, + #[error("client_performance_collection_cancelled")] + Cancelled, + #[error("client_performance_busy")] + Busy, + #[error("client_performance_invalid_endpoint")] + InvalidEndpoint, + #[error("client_performance_invalid_proxy")] + InvalidProxy, + #[error("client_performance_invalid_root_certificate")] + InvalidRootCertificate, + #[error("client_performance_invalid_credential")] + InvalidCredential, + #[error("client_performance_transport_configuration")] + TransportConfiguration, + #[error("client_performance_signing_failed")] + Signing, + #[error("client_performance_encoding_failed")] + Encoding, + #[error("client_performance_output_exists")] + AlreadyExists, + #[error("client_performance_io_failed")] + Io(#[source] std::io::Error), + #[error("client_performance_output_durability_failed")] + DurabilityAfterCommit(#[source] std::io::Error), +} + +pub async fn measure_client( + request: &ClientPerformanceRequest, + probe: &impl ClientProbe, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + if CLIENT_COLLECTOR_ACTIVE + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .is_err() + { + return Err(ClientPerformanceError::Busy); + } + let _guard = ActiveGuard; + if cancel.is_cancelled() { + return Ok(terminal_measurement( + request, + ClientOutcome::Cancelled, + ClientReasonCode::Cancelled, + ClientTargetReasonCode::Cancelled, + Duration::ZERO, + )); + } + let started = Instant::now(); + let deadline = tokio::time::sleep(request.duration); + tokio::pin!(deadline); + let probe = probe.probe(request, cancel); + tokio::pin!(probe); + let result = tokio::select! { + () = cancel.cancelled() => Err(ClientProbeError::Cancelled), + () = &mut deadline => Err(ClientProbeError::TimedOut), + result = &mut probe => result, + }; + Ok(match result { + Ok(sample) if sample.transferred_bytes == request.traffic_bytes => success_measurement(request, sample), + Ok(_) => failed_measurement(request, started.elapsed(), ClientTargetReasonCode::ProtocolFailure), + Err(ClientProbeError::Cancelled) => terminal_measurement( + request, + ClientOutcome::Cancelled, + ClientReasonCode::Cancelled, + ClientTargetReasonCode::Cancelled, + started.elapsed(), + ), + Err(error) => failed_measurement(request, started.elapsed(), target_reason(error)), + }) +} + +pub fn sign_client_export( + request: &ClientPerformanceRequest, + measurement: &ClientMeasurement, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + check_cancel(cancel)?; + let result = &measurement.result; + if result.outcome != ClientOutcome::Succeeded + || result.data.is_none() + || result.run_uid != request.run_uid + || result.schema_version != CLIENT_SCHEMA_VERSION + || result.tool_id != CLIENT_TOOL_ID + || result.capability != CLIENT_CAPABILITY + { + return Err(ClientPerformanceError::InvalidRequest); + } + let result_json = serde_json::to_vec(result).map_err(|_| ClientPerformanceError::Encoding)?; + if result_json.is_empty() || result_json.len() > MAX_CLIENT_RESULT_BYTES { + return Err(ClientPerformanceError::LimitExceeded); + } + let device_key_id = hex_lower(&Sha256::digest(key.public_key_der())); + let result_sha256 = hex_lower(&Sha256::digest(&result_json)); + let envelope = ClientEnvelope { + format_version: "rustfs.connect.diagnosticEnvelope/1", + protocol_version: "v1", + organization_name: &request.organization_name, + cluster_name: &request.cluster_name, + device_name: &request.device_name, + run_uid: &request.run_uid, + artifact_uid: &request.artifact_uid, + tool_id: CLIENT_TOOL_ID, + schema_version: CLIENT_SCHEMA_VERSION, + classification: "L1", + consent_uid: &request.consent.consent_uid, + policy_revision: request.consent.policy_revision, + produced_at: timestamp(request.produced_at_unix)?, + expires_at: timestamp(request.expires_at_unix)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(request.nonce), + device_key_id: &device_key_id, + payload: ClientPayload { + path: RESULT_PATH, + media_type: "application/json", + size_bytes: u64::try_from(result_json.len()).map_err(|_| ClientPerformanceError::LimitExceeded)?, + sha256: &result_sha256, + }, + }; + let envelope_json = serde_json::to_vec(&envelope).map_err(|_| ClientPerformanceError::Encoding)?; + if envelope_json.is_empty() || envelope_json.len() > MAX_ENVELOPE_BYTES { + return Err(ClientPerformanceError::LimitExceeded); + } + let envelope_signature = signature_document(key, &device_key_id, &envelope_json)?; + let decompressed = result_json + .len() + .checked_add(envelope_json.len()) + .and_then(|size| size.checked_add(envelope_signature.len())) + .ok_or(ClientPerformanceError::LimitExceeded)?; + if decompressed > MAX_DECOMPRESSED_BYTES { + return Err(ClientPerformanceError::LimitExceeded); + } + check_cancel(cancel)?; + if unix_now()? >= request.expires_at_unix { + return Err(ClientPerformanceError::Expired); + } + let archive_bytes = archive(&envelope_json, &envelope_signature, &result_json)?; + if archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(ClientPerformanceError::LimitExceeded); + } + let archive_sha256 = hex_lower(&Sha256::digest(&archive_bytes)); + Ok(SignedClientExport { + artifact_uid: request.artifact_uid.clone(), + outcome: result.outcome, + reason_code: result.reason_code, + envelope_json, + envelope_signature, + result_json, + archive_bytes, + archive_sha256, + }) +} + +pub fn save_signed_client_export( + output: &Path, + export: &SignedClientExport, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + if !uuid7(&export.artifact_uid) || export.archive_bytes.is_empty() { + return Err(ClientPerformanceError::InvalidRequest); + } + if export.archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(ClientPerformanceError::LimitExceeded); + } + if hex_lower(&Sha256::digest(&export.archive_bytes)) != export.archive_sha256 { + return Err(ClientPerformanceError::InvalidRequest); + } + let parent = output + .parent() + .filter(|path| !path.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let filename = output + .file_name() + .ok_or(ClientPerformanceError::InvalidRequest)? + .to_string_lossy(); + let temporary = parent.join(format!(".{filename}.{}.partial", export.artifact_uid)); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + let mut file = options.open(&temporary).map_err(map_create_error)?; + let saved = (|| { + file.write_all(&export.archive_bytes).map_err(ClientPerformanceError::Io)?; + check_cancel(cancel)?; + file.sync_all().map_err(ClientPerformanceError::Io)?; + check_cancel(cancel)?; + fs::hard_link(&temporary, output).map_err(map_publish_error)?; + fs::remove_file(&temporary).map_err(ClientPerformanceError::DurabilityAfterCommit)?; + #[cfg(unix)] + File::open(parent) + .and_then(|directory| directory.sync_all()) + .map_err(ClientPerformanceError::DurabilityAfterCommit)?; + Ok(SavedClientExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: u64::try_from(export.archive_bytes.len()).map_err(|_| ClientPerformanceError::LimitExceeded)?, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if saved.is_err() { + let _ = fs::remove_file(&temporary); + } + saved +} + +pub fn validate_client_limits(duration: Duration, traffic_bytes: u64) -> Result<(), ClientPerformanceError> { + if duration.is_zero() + || duration.as_millis() == 0 + || duration > MAX_CLIENT_DURATION + || traffic_bytes == 0 + || traffic_bytes > MAX_CLIENT_TRAFFIC_BYTES + { + return Err(ClientPerformanceError::LimitExceeded); + } + let bandwidth_budget = u64::try_from( + u128::from(MAX_CLIENT_BANDWIDTH_BYTES_PER_SECOND) + .checked_mul(duration.as_millis()) + .ok_or(ClientPerformanceError::LimitExceeded)? + / 1_000, + ) + .map_err(|_| ClientPerformanceError::LimitExceeded)?; + if traffic_bytes > bandwidth_budget.max(1) { + return Err(ClientPerformanceError::LimitExceeded); + } + Ok(()) +} + +pub fn read_protected_client_credential(path: &Path) -> Result, ClientPerformanceError> { + let metadata = fs::symlink_metadata(path).map_err(ClientPerformanceError::Io)?; + if metadata.file_type().is_symlink() || !metadata.is_file() { + return Err(ClientPerformanceError::InvalidCredential); + } + #[cfg(unix)] + { + use std::os::unix::fs::{MetadataExt as _, OpenOptionsExt as _, PermissionsExt as _}; + if metadata.uid() != process_uid() || metadata.permissions().mode() & 0o077 != 0 { + return Err(ClientPerformanceError::InvalidCredential); + } + let mut options = OpenOptions::new(); + options.read(true).custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW); + let mut file = options.open(path).map_err(ClientPerformanceError::Io)?; + let opened = file.metadata().map_err(ClientPerformanceError::Io)?; + if opened.dev() != metadata.dev() || opened.ino() != metadata.ino() { + return Err(ClientPerformanceError::InvalidCredential); + } + read_credential_value(&mut file) + } + #[cfg(not(unix))] + { + let mut file = File::open(path).map_err(ClientPerformanceError::Io)?; + read_credential_value(&mut file) + } +} + +fn read_credential_value(reader: &mut impl Read) -> Result, ClientPerformanceError> { + let mut bytes = Vec::with_capacity(256); + reader + .take(4_097) + .read_to_end(&mut bytes) + .map_err(ClientPerformanceError::Io)?; + if bytes.is_empty() || bytes.len() > 4_096 || bytes.contains(&0) { + return Err(ClientPerformanceError::InvalidCredential); + } + while matches!(bytes.last(), Some(b'\n' | b'\r')) { + bytes.pop(); + } + let value = String::from_utf8(bytes).map_err(|_| ClientPerformanceError::InvalidCredential)?; + if value.is_empty() || value.len() > 4_096 || value.trim() != value { + return Err(ClientPerformanceError::InvalidCredential); + } + Ok(Zeroizing::new(value)) +} + +#[cfg(unix)] +#[allow(unsafe_code)] +fn process_uid() -> u32 { + // SAFETY: geteuid has no pointer arguments or caller preconditions. + unsafe { libc::geteuid() } +} + +impl ClientPerformanceRequest { + fn validate(&self, now_unix: i64) -> Result<(), ClientPerformanceError> { + if self.schema_version != CLIENT_SCHEMA_VERSION { + return Err(ClientPerformanceError::UnsupportedVersion); + } + if self.capability != CLIENT_CAPABILITY { + return Err(ClientPerformanceError::UnsupportedCapability); + } + if !self.consent.confirmed || self.consent.policy_revision == 0 { + return Err(ClientPerformanceError::ConsentRequired); + } + if self.consent.expires_at_unix <= now_unix || self.expires_at_unix > self.consent.expires_at_unix { + return Err(ClientPerformanceError::ConsentExpired); + } + let validity = self + .expires_at_unix + .checked_sub(self.produced_at_unix) + .ok_or(ClientPerformanceError::Expired)?; + if self.produced_at_unix > now_unix.saturating_add(MAX_FUTURE_SKEW_SECONDS) + || validity <= 0 + || validity > MAX_VALIDITY_SECONDS + || self.expires_at_unix <= now_unix + { + return Err(ClientPerformanceError::Expired); + } + validate_client_limits(self.duration, self.traffic_bytes)?; + if !uuid7(&self.run_uid) + || !uuid7(&self.artifact_uid) + || !uuid7(&self.consent.consent_uid) + || !resource_names_match(self) + || self.target_alias.is_empty() + || self.target_alias.len() > 128 + || !self + .target_alias + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || b"._:-".contains(&byte)) + || !lower_hex_string(&self.provenance.source_commit, 40) + || !lower_hex_string(&self.provenance.executable_sha256, 64) + || !version(&self.provenance.rustfs_version) + || self.provenance.build_features.len() > MAX_BUILD_FEATURES + || !self.provenance.build_features.iter().all(|value| build_feature(value)) + { + return Err(ClientPerformanceError::InvalidRequest); + } + Ok(()) + } +} + +struct ActiveGuard; + +impl Drop for ActiveGuard { + fn drop(&mut self) { + CLIENT_COLLECTOR_ACTIVE.store(false, Ordering::Release); + } +} + +fn success_measurement(request: &ClientPerformanceRequest, sample: ClientProbeMeasurement) -> ClientMeasurement { + let duration_millis = elapsed_millis(sample.duration); + let data = ClientPerformanceData { + operation: request.operation, + transferred_bytes: sample.transferred_bytes, + completed_operations: 1, + duration_millis, + error_count: 0, + }; + ClientMeasurement { + result: result( + request, + ClientOutcome::Succeeded, + ClientReasonCode::Complete, + duration_millis, + 1, + Some(data), + ), + target: ClientTargetResult { + target_alias: request.target_alias.clone(), + outcome: ClientOutcome::Succeeded, + reason_code: ClientTargetReasonCode::Complete, + parameters: target_parameters(request), + units: target_units(), + transferred_bytes: sample.transferred_bytes, + completed_operations: 1, + latency_micros: Some(elapsed_micros(sample.latency)), + duration_millis, + }, + } +} + +fn failed_measurement( + request: &ClientPerformanceRequest, + elapsed: Duration, + target_reason: ClientTargetReasonCode, +) -> ClientMeasurement { + let reason = match target_reason { + ClientTargetReasonCode::EndpointUnavailable => ClientReasonCode::SourceUnavailable, + ClientTargetReasonCode::PermissionDenied => ClientReasonCode::PermissionDenied, + _ => ClientReasonCode::CollectionFailed, + }; + terminal_measurement(request, ClientOutcome::Failed, reason, target_reason, elapsed) +} + +fn terminal_measurement( + request: &ClientPerformanceRequest, + outcome: ClientOutcome, + reason_code: ClientReasonCode, + target_reason: ClientTargetReasonCode, + elapsed: Duration, +) -> ClientMeasurement { + let duration_millis = elapsed_millis_allow_zero(elapsed); + ClientMeasurement { + result: result(request, outcome, reason_code, duration_millis, 0, None), + target: ClientTargetResult { + target_alias: request.target_alias.clone(), + outcome, + reason_code: target_reason, + parameters: target_parameters(request), + units: target_units(), + transferred_bytes: 0, + completed_operations: 0, + latency_micros: None, + duration_millis, + }, + } +} + +fn target_reason(error: ClientProbeError) -> ClientTargetReasonCode { + match error { + ClientProbeError::EndpointUnavailable => ClientTargetReasonCode::EndpointUnavailable, + ClientProbeError::ProxyFailure => ClientTargetReasonCode::ProxyFailure, + ClientProbeError::PermissionDenied => ClientTargetReasonCode::PermissionDenied, + ClientProbeError::TimedOut => ClientTargetReasonCode::TimedOut, + ClientProbeError::Cancelled => ClientTargetReasonCode::Cancelled, + ClientProbeError::ProtocolFailure => ClientTargetReasonCode::ProtocolFailure, + } +} + +fn target_parameters(request: &ClientPerformanceRequest) -> ClientTargetParameters { + ClientTargetParameters { + operation: request.operation, + requested_bytes: request.traffic_bytes, + duration_millis: elapsed_millis_allow_zero(request.duration), + concurrency: 1, + } +} + +fn target_units() -> ClientTargetUnits { + ClientTargetUnits { + bytes: "BYTE", + duration: "MILLISECOND", + latency: "MICROSECOND", + operation_count: "OPERATION", + } +} + +fn result( + request: &ClientPerformanceRequest, + outcome: ClientOutcome, + reason_code: ClientReasonCode, + duration_millis: u64, + completed_units: u32, + data: Option, +) -> ClientDiagnosticResult { + ClientDiagnosticResult { + schema_version: CLIENT_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: CLIENT_TOOL_ID, + capability: CLIENT_CAPABILITY, + outcome, + reason_code, + duration_millis: duration_millis.min(30_000), + provenance: request.provenance.clone(), + coverage: ClientCoverage { + requested_units: 1, + completed_units, + unit: "WINDOW", + }, + data, + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ClientEnvelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: &'static str, + schema_version: u16, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: &'a str, + payload: ClientPayload<'a>, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ClientPayload<'a> { + path: &'static str, + media_type: &'static str, + size_bytes: u64, + sha256: &'a str, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ClientSignature<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +fn signature_document(key: &DeviceIdentity, key_id: &str, envelope: &[u8]) -> Result, ClientPerformanceError> { + let pkcs8 = key.to_pkcs8_der().map_err(|_| ClientPerformanceError::Signing)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| ClientPerformanceError::Signing)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + envelope.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.extend_from_slice(envelope); + let signature: Signature = signing_key.sign(&input); + serde_json::to_vec(&ClientSignature { + algorithm: "ES256", + key_id, + value: URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()), + }) + .map_err(|_| ClientPerformanceError::Encoding) +} + +fn archive(envelope: &[u8], signature: &[u8], result: &[u8]) -> Result, ClientPerformanceError> { + let cursor = Cursor::new(Vec::with_capacity(envelope.len() + signature.len() + result.len() + 512)); + let mut writer = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(OUTPUT_MODE); + for (name, bytes) in [(ENVELOPE_PATH, envelope), (SIGNATURE_PATH, signature), (RESULT_PATH, result)] { + writer + .start_file(name, options) + .map_err(|_| ClientPerformanceError::Encoding)?; + writer.write_all(bytes).map_err(ClientPerformanceError::Io)?; + } + writer + .finish() + .map(|cursor| cursor.into_inner()) + .map_err(|_| ClientPerformanceError::Encoding) +} + +fn deployment_endpoint(value: &str) -> Result { + let mut url = Url::parse(value).map_err(|_| ClientPerformanceError::InvalidEndpoint)?; + let local_http = url.scheme() == "http" + && url + .host_str() + .is_some_and(|host| host == "localhost" || host.parse::().is_ok_and(|ip| ip.is_loopback())); + if (url.scheme() != "https" && !local_http) + || url.cannot_be_a_base() + || !url.username().is_empty() + || url.password().is_some() + || url.query().is_some() + || url.fragment().is_some() + { + return Err(ClientPerformanceError::InvalidEndpoint); + } + url.set_query(None); + url.set_fragment(None); + if !url.path().ends_with('/') { + url.set_path(&format!("{}/", url.path())); + } + Ok(url) +} + +fn proxy_url(value: &str) -> Result { + let url = Url::parse(value).map_err(|_| ClientPerformanceError::InvalidProxy)?; + if !matches!(url.scheme(), "http" | "https") + || url.cannot_be_a_base() + || !url.username().is_empty() + || url.password().is_some() + || url.query().is_some() + || url.fragment().is_some() + { + return Err(ClientPerformanceError::InvalidProxy); + } + Ok(url) +} + +fn resource_names_match(request: &ClientPerformanceRequest) -> bool { + let Some(organization_uid) = request.organization_name.strip_prefix("organizations/") else { + return false; + }; + if !uuid7(organization_uid) { + return false; + } + let cluster_prefix = format!("{}/clusters/", request.organization_name); + let Some(cluster_uid) = request.cluster_name.strip_prefix(&cluster_prefix) else { + return false; + }; + if !uuid7(cluster_uid) { + return false; + } + let device_prefix = format!("{}/clusterDevices/", request.cluster_name); + request.device_name.strip_prefix(&device_prefix).is_some_and(uuid7) +} + +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn lower_hex_string(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_hexdigit() && !byte.is_ascii_uppercase()) +} + +fn version(value: &str) -> bool { + if value.is_empty() || value.len() > 64 { + return false; + } + let (core, prerelease) = value + .split_once('-') + .map_or((value, None), |(core, prerelease)| (core, Some(prerelease))); + let mut parts = core.split('.'); + let valid_core = (0..3).all(|_| { + parts + .next() + .is_some_and(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) + }) && parts.next().is_none(); + valid_core + && prerelease.is_none_or(|part| { + !part.is_empty() + && part + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'-')) + }) +} + +fn build_feature(value: &str) -> bool { + value.len() <= 64 + && value.as_bytes().split_first().is_some_and(|(first, rest)| { + first.is_ascii_lowercase() + && rest + .iter() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'-' | b'_')) + }) +} + +fn elapsed_millis(duration: Duration) -> u64 { + elapsed_millis_allow_zero(duration).max(1) +} + +fn elapsed_millis_allow_zero(duration: Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).min(MAX_SAFE_INTEGER) +} + +fn elapsed_micros(duration: Duration) -> u64 { + u64::try_from(duration.as_micros()).unwrap_or(u64::MAX).min(MAX_SAFE_INTEGER) +} + +fn unix_now() -> Result { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| ClientPerformanceError::InvalidRequest) + .and_then(|duration| i64::try_from(duration.as_secs()).map_err(|_| ClientPerformanceError::InvalidRequest)) +} + +fn timestamp(unix: i64) -> Result { + OffsetDateTime::from_unix_timestamp(unix) + .map_err(|_| ClientPerformanceError::InvalidRequest)? + .format(&Rfc3339) + .map_err(|_| ClientPerformanceError::Encoding) +} + +fn check_cancel(cancel: &CancellationToken) -> Result<(), ClientPerformanceError> { + if cancel.is_cancelled() { + Err(ClientPerformanceError::Cancelled) + } else { + Ok(()) + } +} + +fn hex_lower(bytes: &[u8]) -> String { + let mut encoded = String::with_capacity(bytes.len() * 2); + for byte in bytes { + use std::fmt::Write as _; + let _ = write!(encoded, "{byte:02x}"); + } + encoded +} + +fn map_create_error(error: std::io::Error) -> ClientPerformanceError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + ClientPerformanceError::AlreadyExists + } else { + ClientPerformanceError::Io(error) + } +} + +fn map_publish_error(error: std::io::Error) -> ClientPerformanceError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + ClientPerformanceError::AlreadyExists + } else { + ClientPerformanceError::Io(error) + } +} diff --git a/rustfs/src/connect/diagnostics/perf_drive.rs b/rustfs/src/connect/diagnostics/perf_drive.rs new file mode 100644 index 000000000..441a28095 --- /dev/null +++ b/rustfs/src/connect/diagnostics/perf_drive.rs @@ -0,0 +1,1145 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Consent-bound local drive performance measurement and offline export. +//! +//! The benchmark writes generated bytes to a task-owned scratch directory, +//! reads and verifies them, and removes the scratch state before returning. +//! The read follows the write and therefore measures the filesystem's warm +//! page-cache-visible path. It is not reported as cold physical-media I/O; +//! RustFS does not expose a portable cache-bypass adapter for this producer. +//! Existing StorageInfo counters are deliberately not used because passive +//! observations are not evidence of an active read/write benchmark. + +use std::fs::{self, File, OpenOptions}; +use std::io::{Cursor, Write as _}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use base64_simd::URL_SAFE_NO_PAD; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use serde::Serialize; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio::io::{AsyncReadExt as _, AsyncSeekExt as _, AsyncWriteExt as _}; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use crate::connect::DeviceIdentity; + +pub const DRIVE_SCHEMA_VERSION: u16 = 1; +pub const DRIVE_TOOL_ID: &str = "performance.drive"; +pub const DRIVE_CAPABILITY: &str = "performance.drive@1"; +pub const MAX_DRIVE_DURATION: Duration = Duration::from_secs(30); +pub const MAX_TEMPORARY_BYTES: u64 = 2_097_152; +pub const MAX_IO_BYTES: u64 = 1_048_576; +pub const MAX_BANDWIDTH_BYTES_PER_SECOND: u64 = 1_048_576; +pub const MAX_OPERATIONS: u64 = 1_024; +pub const MAX_BLOCK_BYTES: u64 = 65_536; +pub const MAX_RESULT_BYTES: usize = 262_144; +pub const MAX_ENVELOPE_BYTES: usize = 16_384; +pub const MAX_ARCHIVE_BYTES: usize = 524_288; +pub const MAX_DECOMPRESSED_BYTES: usize = 278_528; + +const MAX_SAFE_INTEGER: u64 = 9_007_199_254_740_991; +const MAX_BUILD_FEATURES: usize = 64; +const MAX_VALIDITY_SECONDS: i64 = 2_592_000; +const MAX_FUTURE_SKEW_SECONDS: i64 = 300; +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1\0"; +const ENVELOPE_PATH: &str = "envelope.json"; +const SIGNATURE_PATH: &str = "envelope.sig"; +const RESULT_PATH: &str = "result.json"; +const OUTPUT_MODE: u32 = 0o600; +const SCRATCH_MODE: u32 = 0o700; +const SCRATCH_PREFIX: &str = ".rustfs-connect-drive-"; + +static DRIVE_COLLECTOR_ACTIVE: AtomicBool = AtomicBool::new(false); + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum DriveOutcome { + Succeeded, + Failed, + Cancelled, +} + +impl DriveOutcome { + pub const fn as_str(self) -> &'static str { + match self { + Self::Succeeded => "SUCCEEDED", + Self::Failed => "FAILED", + Self::Cancelled => "CANCELLED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum DriveReasonCode { + Complete, + LimitExceeded, + SourceUnavailable, + PermissionDenied, + Cancelled, + CollectionFailed, +} + +impl DriveReasonCode { + pub const fn as_str(self) -> &'static str { + match self { + Self::Complete => "COMPLETE", + Self::LimitExceeded => "LIMIT_EXCEEDED", + Self::SourceUnavailable => "SOURCE_UNAVAILABLE", + Self::PermissionDenied => "PERMISSION_DENIED", + Self::Cancelled => "CANCELLED", + Self::CollectionFailed => "COLLECTION_FAILED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum DriveTargetReasonCode { + Complete, + LimitExceeded, + SourceUnavailable, + PermissionDenied, + TimedOut, + Cancelled, + IoFailure, + VerificationFailed, + CleanupFailed, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum DriveReadMode { + WarmPageCache, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct DriveTargetParameters { + pub scratch_bytes: u64, + pub block_bytes: u64, + pub duration_millis: u64, + pub concurrency: u8, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct DriveTargetUnits { + pub bytes: &'static str, + pub duration: &'static str, + pub latency: &'static str, + pub io_count: &'static str, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct DriveProvenance { + repository: &'static str, + source_commit: String, + executable_sha256: String, + rustfs_version: String, + os_family: DriveOsFamily, + architecture: DriveArchitecture, + build_features: Vec, +} + +impl DriveProvenance { + pub fn new( + source_commit: impl Into, + executable_sha256: impl Into, + rustfs_version: impl Into, + build_features: Vec, + ) -> Self { + Self { + repository: "rustfs/rustfs", + source_commit: source_commit.into(), + executable_sha256: executable_sha256.into(), + rustfs_version: rustfs_version.into(), + os_family: DriveOsFamily::current(), + architecture: DriveArchitecture::current(), + build_features, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum DriveOsFamily { + Linux, + Darwin, + Windows, + Freebsd, + Other, +} + +impl DriveOsFamily { + fn current() -> Self { + match std::env::consts::OS { + "linux" => Self::Linux, + "macos" => Self::Darwin, + "windows" => Self::Windows, + "freebsd" => Self::Freebsd, + _ => Self::Other, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +enum DriveArchitecture { + #[serde(rename = "x86_64")] + X86_64, + Aarch64, + Other, +} + +impl DriveArchitecture { + fn current() -> Self { + match std::env::consts::ARCH { + "x86_64" => Self::X86_64, + "aarch64" => Self::Aarch64, + _ => Self::Other, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LocalDriveConsent { + pub consent_uid: String, + pub policy_revision: u64, + pub expires_at_unix: i64, + pub confirmed: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct DrivePerformanceRequest { + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub schema_version: u16, + pub capability: String, + pub consent: LocalDriveConsent, + pub produced_at_unix: i64, + pub expires_at_unix: i64, + pub nonce: [u8; 32], + pub duration: Duration, + pub target_alias: String, + pub scratch_root: PathBuf, + pub scratch_bytes: u64, + pub block_bytes: u64, + pub provenance: DriveProvenance, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct DrivePerformanceData { + pub read_bytes: u64, + pub write_bytes: u64, + pub io_count: u64, + pub duration_millis: u64, + pub error_count: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct DriveCoverage { + requested_units: u32, + completed_units: u32, + unit: &'static str, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct DriveDiagnosticResult { + schema_version: u16, + run_uid: String, + tool_id: &'static str, + capability: &'static str, + outcome: DriveOutcome, + reason_code: DriveReasonCode, + duration_millis: u64, + provenance: DriveProvenance, + coverage: DriveCoverage, + data: Option, +} + +impl DriveDiagnosticResult { + pub fn outcome(&self) -> DriveOutcome { + self.outcome + } + + pub fn reason_code(&self) -> DriveReasonCode { + self.reason_code + } + + pub fn data(&self) -> Option<&DrivePerformanceData> { + self.data.as_ref() + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct DriveTargetResult { + pub target_alias: String, + pub outcome: DriveOutcome, + pub reason_code: DriveTargetReasonCode, + pub parameters: DriveTargetParameters, + pub units: DriveTargetUnits, + pub read_mode: Option, + pub read_bytes: u64, + pub write_bytes: u64, + pub io_count: u64, + pub cached_read_latency_micros: Option, + pub write_latency_micros: Option, + pub duration_millis: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct DriveMeasurement { + pub result: DriveDiagnosticResult, + pub target: DriveTargetResult, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SignedDriveExport { + pub artifact_uid: String, + pub outcome: DriveOutcome, + pub reason_code: DriveReasonCode, + pub envelope_json: Vec, + pub envelope_signature: Vec, + pub result_json: Vec, + pub archive_bytes: Vec, + pub archive_sha256: String, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SavedDriveExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +#[derive(Debug, Error)] +pub enum DrivePerformanceError { + #[error("drive_performance_local_consent_required")] + ConsentRequired, + #[error("drive_performance_local_consent_expired")] + ConsentExpired, + #[error("drive_performance_request_expired")] + Expired, + #[error("drive_performance_invalid_request")] + InvalidRequest, + #[error("drive_performance_unsupported_version")] + UnsupportedVersion, + #[error("drive_performance_unsupported_capability")] + UnsupportedCapability, + #[error("drive_performance_limit_exceeded")] + LimitExceeded, + #[error("drive_performance_collection_cancelled")] + Cancelled, + #[error("drive_performance_collection_already_running")] + Busy, + #[error("drive_performance_export_signing_failed")] + Signing, + #[error("drive_performance_export_exists")] + AlreadyExists, + #[error("drive_performance_export_io_failed")] + Io(#[source] std::io::Error), + #[error("drive_performance_export_encoding_failed")] + Encoding, + #[error("drive_performance_export_durability_failed_after_commit")] + DurabilityAfterCommit(#[source] std::io::Error), +} + +pub async fn measure_drive( + request: &DrivePerformanceRequest, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + if cancel.is_cancelled() { + return Ok(terminal_measurement( + request, + DriveOutcome::Cancelled, + DriveReasonCode::Cancelled, + DriveTargetReasonCode::Cancelled, + 0, + )); + } + let _lease = CollectorLease::acquire()?; + let started = Instant::now(); + let deadline = tokio::time::Instant::now() + request.duration; + let mut scratch = match ScratchGuard::create(request) { + Ok(scratch) => scratch, + Err(error) => return Ok(failed_measurement(request, started.elapsed(), classify_io(&error), 0, 0, 0)), + }; + + let measured = run_benchmark(request, &scratch, cancel, deadline).await; + let cleanup = scratch.cleanup(); + let elapsed = started.elapsed().min(request.duration); + if cleanup.is_err() { + let (read_bytes, write_bytes, io_count) = match &measured { + Ok(sample) => (sample.read_bytes, sample.write_bytes, sample.io_count), + Err(_) => (0, 0, 0), + }; + return Ok(failed_measurement( + request, + elapsed, + DriveTargetReasonCode::CleanupFailed, + read_bytes, + write_bytes, + io_count, + )); + } + match measured { + Ok(sample) => Ok(success_measurement(request, elapsed, sample)), + Err(BenchmarkFailure::Cancelled) => Ok(terminal_measurement( + request, + DriveOutcome::Cancelled, + DriveReasonCode::Cancelled, + DriveTargetReasonCode::Cancelled, + elapsed_millis_allow_zero(elapsed), + )), + Err(BenchmarkFailure::TimedOut) => Ok(failed_measurement(request, elapsed, DriveTargetReasonCode::TimedOut, 0, 0, 0)), + Err(BenchmarkFailure::Verification) => { + Ok(failed_measurement(request, elapsed, DriveTargetReasonCode::VerificationFailed, 0, 0, 0)) + } + Err(BenchmarkFailure::Io(error)) => Ok(failed_measurement(request, elapsed, classify_io(&error), 0, 0, 0)), + } +} + +pub fn sign_drive_export( + request: &DrivePerformanceRequest, + measurement: &DriveMeasurement, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + check_cancel(cancel)?; + let result = &measurement.result; + if result.outcome != DriveOutcome::Succeeded + || result.data.is_none() + || result.run_uid != request.run_uid + || result.schema_version != DRIVE_SCHEMA_VERSION + || result.tool_id != DRIVE_TOOL_ID + || result.capability != DRIVE_CAPABILITY + { + return Err(DrivePerformanceError::InvalidRequest); + } + let result_json = serde_json::to_vec(result).map_err(|_| DrivePerformanceError::Encoding)?; + if result_json.is_empty() || result_json.len() > MAX_RESULT_BYTES { + return Err(DrivePerformanceError::LimitExceeded); + } + let device_key_id = hex_lower(&Sha256::digest(key.public_key_der())); + let result_sha256 = hex_lower(&Sha256::digest(&result_json)); + let envelope = DriveEnvelope { + format_version: "rustfs.connect.diagnosticEnvelope/1", + protocol_version: "v1", + organization_name: &request.organization_name, + cluster_name: &request.cluster_name, + device_name: &request.device_name, + run_uid: &request.run_uid, + artifact_uid: &request.artifact_uid, + tool_id: DRIVE_TOOL_ID, + schema_version: DRIVE_SCHEMA_VERSION, + classification: "L1", + consent_uid: &request.consent.consent_uid, + policy_revision: request.consent.policy_revision, + produced_at: timestamp(request.produced_at_unix)?, + expires_at: timestamp(request.expires_at_unix)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(request.nonce), + device_key_id: &device_key_id, + payload: DrivePayload { + path: RESULT_PATH, + media_type: "application/json", + size_bytes: result_json.len() as u64, + sha256: &result_sha256, + }, + }; + let envelope_json = serde_json::to_vec(&envelope).map_err(|_| DrivePerformanceError::Encoding)?; + if envelope_json.is_empty() || envelope_json.len() > MAX_ENVELOPE_BYTES { + return Err(DrivePerformanceError::LimitExceeded); + } + let envelope_signature = signature_document(key, &device_key_id, &envelope_json)?; + let decompressed = result_json + .len() + .checked_add(envelope_json.len()) + .and_then(|size| size.checked_add(envelope_signature.len())) + .ok_or(DrivePerformanceError::LimitExceeded)?; + if decompressed > MAX_DECOMPRESSED_BYTES { + return Err(DrivePerformanceError::LimitExceeded); + } + check_cancel(cancel)?; + if unix_now()? >= request.expires_at_unix { + return Err(DrivePerformanceError::Expired); + } + let archive_bytes = archive(&envelope_json, &envelope_signature, &result_json)?; + if archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(DrivePerformanceError::LimitExceeded); + } + let archive_sha256 = hex_lower(&Sha256::digest(&archive_bytes)); + Ok(SignedDriveExport { + artifact_uid: request.artifact_uid.clone(), + outcome: result.outcome, + reason_code: result.reason_code, + envelope_json, + envelope_signature, + result_json, + archive_bytes, + archive_sha256, + }) +} + +pub fn save_signed_drive_export( + output: &Path, + export: &SignedDriveExport, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + if !uuid7(&export.artifact_uid) { + return Err(DrivePerformanceError::InvalidRequest); + } + if export.archive_bytes.is_empty() { + return Err(DrivePerformanceError::InvalidRequest); + } + if export.archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(DrivePerformanceError::LimitExceeded); + } + if hex_lower(&Sha256::digest(&export.archive_bytes)) != export.archive_sha256 { + return Err(DrivePerformanceError::InvalidRequest); + } + let parent = output + .parent() + .filter(|path| !path.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let filename = output + .file_name() + .ok_or(DrivePerformanceError::InvalidRequest)? + .to_string_lossy(); + let temporary = parent.join(format!(".{filename}.{}.partial", export.artifact_uid)); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + let mut file = options.open(&temporary).map_err(map_create_error)?; + let saved = (|| { + file.write_all(&export.archive_bytes).map_err(DrivePerformanceError::Io)?; + check_cancel(cancel)?; + file.sync_all().map_err(DrivePerformanceError::Io)?; + check_cancel(cancel)?; + fs::hard_link(&temporary, output).map_err(map_publish_error)?; + fs::remove_file(&temporary).map_err(DrivePerformanceError::DurabilityAfterCommit)?; + #[cfg(unix)] + File::open(parent) + .and_then(|directory| directory.sync_all()) + .map_err(DrivePerformanceError::DurabilityAfterCommit)?; + Ok(SavedDriveExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: export.archive_bytes.len() as u64, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if saved.is_err() { + let _ = fs::remove_file(&temporary); + } + saved +} + +impl DrivePerformanceRequest { + fn validate(&self, now_unix: i64) -> Result<(), DrivePerformanceError> { + if self.schema_version != DRIVE_SCHEMA_VERSION { + return Err(DrivePerformanceError::UnsupportedVersion); + } + if self.capability != DRIVE_CAPABILITY { + return Err(DrivePerformanceError::UnsupportedCapability); + } + if !self.consent.confirmed || self.consent.policy_revision == 0 { + return Err(DrivePerformanceError::ConsentRequired); + } + if self.consent.expires_at_unix <= now_unix || self.expires_at_unix > self.consent.expires_at_unix { + return Err(DrivePerformanceError::ConsentExpired); + } + let validity = self + .expires_at_unix + .checked_sub(self.produced_at_unix) + .ok_or(DrivePerformanceError::Expired)?; + if self.produced_at_unix > now_unix.saturating_add(MAX_FUTURE_SKEW_SECONDS) + || validity <= 0 + || validity > MAX_VALIDITY_SECONDS + || self.expires_at_unix <= now_unix + { + return Err(DrivePerformanceError::Expired); + } + validate_drive_limits(self.duration, self.scratch_bytes, self.block_bytes)?; + if !uuid7(&self.run_uid) + || !uuid7(&self.artifact_uid) + || !uuid7(&self.consent.consent_uid) + || !resource_names_match(self) + || self.target_alias != "drive-1" + || self.scratch_root.as_os_str().is_empty() + || !lower_hex(&self.provenance.source_commit, 40) + || !lower_hex(&self.provenance.executable_sha256, 64) + || !version(&self.provenance.rustfs_version) + || self.provenance.build_features.len() > MAX_BUILD_FEATURES + || !self.provenance.build_features.iter().all(|value| build_feature(value)) + { + return Err(DrivePerformanceError::InvalidRequest); + } + Ok(()) + } +} + +pub fn validate_drive_limits(duration: Duration, scratch_bytes: u64, block_bytes: u64) -> Result<(), DrivePerformanceError> { + if duration.is_zero() + || duration > MAX_DRIVE_DURATION + || scratch_bytes == 0 + || scratch_bytes > MAX_TEMPORARY_BYTES + || block_bytes == 0 + || block_bytes > MAX_BLOCK_BYTES + || block_bytes > scratch_bytes + { + return Err(DrivePerformanceError::LimitExceeded); + } + let io_bytes = scratch_bytes.checked_mul(2).ok_or(DrivePerformanceError::LimitExceeded)?; + let blocks = scratch_bytes.div_ceil(block_bytes); + let operations = blocks.checked_mul(2).ok_or(DrivePerformanceError::LimitExceeded)?; + let bandwidth_budget = u64::try_from( + u128::from(MAX_BANDWIDTH_BYTES_PER_SECOND) + .checked_mul(duration.as_millis()) + .ok_or(DrivePerformanceError::LimitExceeded)? + / 1_000, + ) + .map_err(|_| DrivePerformanceError::LimitExceeded)?; + if io_bytes > MAX_IO_BYTES || io_bytes > bandwidth_budget.max(1) || operations > MAX_OPERATIONS { + return Err(DrivePerformanceError::LimitExceeded); + } + Ok(()) +} + +#[derive(Clone, Copy)] +struct BenchmarkSample { + read_bytes: u64, + write_bytes: u64, + io_count: u64, + read_latency: Duration, + write_latency: Duration, +} + +enum BenchmarkFailure { + Cancelled, + TimedOut, + Verification, + Io(std::io::Error), +} + +async fn run_benchmark( + request: &DrivePerformanceRequest, + scratch: &ScratchGuard, + cancel: &CancellationToken, + deadline: tokio::time::Instant, +) -> Result { + let std_file = scratch.open_file().map_err(BenchmarkFailure::Io)?; + let mut file = tokio::fs::File::from_std(std_file); + let block_size = usize::try_from(request.block_bytes).map_err(|_| BenchmarkFailure::Verification)?; + let write_buffer = vec![0xa5; block_size]; + let mut remaining = request.scratch_bytes; + let mut write_bytes = 0_u64; + let mut io_count = 0_u64; + let write_started = Instant::now(); + while remaining > 0 { + let length = usize::try_from(remaining.min(request.block_bytes)).map_err(|_| BenchmarkFailure::Verification)?; + cancellable(file.write_all(&write_buffer[..length]), cancel, deadline).await?; + write_bytes = write_bytes.checked_add(length as u64).ok_or(BenchmarkFailure::Verification)?; + io_count = io_count.checked_add(1).ok_or(BenchmarkFailure::Verification)?; + remaining -= length as u64; + tokio::task::yield_now().await; + } + cancellable(file.sync_all(), cancel, deadline).await?; + let write_latency = write_started.elapsed(); + cancellable(file.seek(std::io::SeekFrom::Start(0)), cancel, deadline).await?; + + let mut read_buffer = vec![0_u8; block_size]; + remaining = request.scratch_bytes; + let mut read_bytes = 0_u64; + let read_started = Instant::now(); + while remaining > 0 { + let length = usize::try_from(remaining.min(request.block_bytes)).map_err(|_| BenchmarkFailure::Verification)?; + cancellable(file.read_exact(&mut read_buffer[..length]), cancel, deadline).await?; + if read_buffer[..length] != write_buffer[..length] { + return Err(BenchmarkFailure::Verification); + } + read_bytes = read_bytes.checked_add(length as u64).ok_or(BenchmarkFailure::Verification)?; + io_count = io_count.checked_add(1).ok_or(BenchmarkFailure::Verification)?; + remaining -= length as u64; + tokio::task::yield_now().await; + } + let read_latency = read_started.elapsed(); + drop(file); + if read_bytes != request.scratch_bytes || write_bytes != request.scratch_bytes || io_count > MAX_OPERATIONS { + return Err(BenchmarkFailure::Verification); + } + Ok(BenchmarkSample { + read_bytes, + write_bytes, + io_count, + read_latency, + write_latency, + }) +} + +async fn cancellable( + operation: F, + cancel: &CancellationToken, + deadline: tokio::time::Instant, +) -> Result +where + F: std::future::Future>, +{ + tokio::select! { + () = cancel.cancelled() => Err(BenchmarkFailure::Cancelled), + () = tokio::time::sleep_until(deadline) => Err(BenchmarkFailure::TimedOut), + result = operation => result.map_err(BenchmarkFailure::Io), + } +} + +struct ScratchGuard { + directory: PathBuf, + file: PathBuf, + active: bool, +} + +impl ScratchGuard { + fn create(request: &DrivePerformanceRequest) -> std::io::Result { + let metadata = fs::symlink_metadata(&request.scratch_root)?; + if metadata.file_type().is_symlink() || !metadata.is_dir() { + return Err(std::io::Error::new( + std::io::ErrorKind::NotADirectory, + "drive benchmark scratch root must be a real directory", + )); + } + let directory = request.scratch_root.join(format!("{SCRATCH_PREFIX}{}", request.artifact_uid)); + let mut builder = fs::DirBuilder::new(); + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt as _; + builder.mode(SCRATCH_MODE); + } + builder.create(&directory)?; + Ok(Self { + file: directory.join("benchmark.bin"), + directory, + active: true, + }) + } + + fn open_file(&self) -> std::io::Result { + let mut options = OpenOptions::new(); + options.read(true).write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + options.open(&self.file) + } + + fn cleanup(&mut self) -> std::io::Result<()> { + match fs::remove_file(&self.file) { + Ok(()) => {} + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => return Err(error), + } + fs::remove_dir(&self.directory)?; + self.active = false; + Ok(()) + } +} + +impl Drop for ScratchGuard { + fn drop(&mut self) { + if self.active { + let _ = fs::remove_file(&self.file); + let _ = fs::remove_dir(&self.directory); + } + } +} + +pub(crate) struct CollectorLease; + +impl CollectorLease { + pub(crate) fn acquire() -> Result { + DRIVE_COLLECTOR_ACTIVE + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map(|_| Self) + .map_err(|_| DrivePerformanceError::Busy) + } +} + +impl Drop for CollectorLease { + fn drop(&mut self) { + DRIVE_COLLECTOR_ACTIVE.store(false, Ordering::Release); + } +} + +fn success_measurement(request: &DrivePerformanceRequest, elapsed: Duration, sample: BenchmarkSample) -> DriveMeasurement { + let duration_millis = elapsed_millis(elapsed); + let data = DrivePerformanceData { + read_bytes: sample.read_bytes, + write_bytes: sample.write_bytes, + io_count: sample.io_count, + duration_millis, + error_count: 0, + }; + DriveMeasurement { + result: result( + request, + DriveOutcome::Succeeded, + DriveReasonCode::Complete, + duration_millis, + 1, + Some(data), + ), + target: DriveTargetResult { + target_alias: request.target_alias.clone(), + outcome: DriveOutcome::Succeeded, + reason_code: DriveTargetReasonCode::Complete, + parameters: target_parameters(request), + units: target_units(), + read_mode: Some(DriveReadMode::WarmPageCache), + read_bytes: sample.read_bytes, + write_bytes: sample.write_bytes, + io_count: sample.io_count, + cached_read_latency_micros: Some(elapsed_micros(sample.read_latency)), + write_latency_micros: Some(elapsed_micros(sample.write_latency)), + duration_millis, + }, + } +} + +pub(crate) fn failed_measurement( + request: &DrivePerformanceRequest, + elapsed: Duration, + target_reason: DriveTargetReasonCode, + read_bytes: u64, + write_bytes: u64, + io_count: u64, +) -> DriveMeasurement { + let reason = match target_reason { + DriveTargetReasonCode::PermissionDenied => DriveReasonCode::PermissionDenied, + DriveTargetReasonCode::SourceUnavailable => DriveReasonCode::SourceUnavailable, + _ => DriveReasonCode::CollectionFailed, + }; + let duration_millis = elapsed_millis_allow_zero(elapsed); + DriveMeasurement { + result: result(request, DriveOutcome::Failed, reason, duration_millis, 0, None), + target: DriveTargetResult { + target_alias: request.target_alias.clone(), + outcome: DriveOutcome::Failed, + reason_code: target_reason, + parameters: target_parameters(request), + units: target_units(), + read_mode: None, + read_bytes, + write_bytes, + io_count, + cached_read_latency_micros: None, + write_latency_micros: None, + duration_millis, + }, + } +} + +fn terminal_measurement( + request: &DrivePerformanceRequest, + outcome: DriveOutcome, + reason_code: DriveReasonCode, + target_reason: DriveTargetReasonCode, + duration_millis: u64, +) -> DriveMeasurement { + DriveMeasurement { + result: result(request, outcome, reason_code, duration_millis, 0, None), + target: DriveTargetResult { + target_alias: request.target_alias.clone(), + outcome, + reason_code: target_reason, + parameters: target_parameters(request), + units: target_units(), + read_mode: None, + read_bytes: 0, + write_bytes: 0, + io_count: 0, + cached_read_latency_micros: None, + write_latency_micros: None, + duration_millis, + }, + } +} + +fn target_parameters(request: &DrivePerformanceRequest) -> DriveTargetParameters { + DriveTargetParameters { + scratch_bytes: request.scratch_bytes, + block_bytes: request.block_bytes, + duration_millis: elapsed_millis_allow_zero(request.duration), + concurrency: 1, + } +} + +fn target_units() -> DriveTargetUnits { + DriveTargetUnits { + bytes: "BYTE", + duration: "MILLISECOND", + latency: "MICROSECOND", + io_count: "OPERATION", + } +} + +fn result( + request: &DrivePerformanceRequest, + outcome: DriveOutcome, + reason_code: DriveReasonCode, + duration_millis: u64, + completed_units: u32, + data: Option, +) -> DriveDiagnosticResult { + DriveDiagnosticResult { + schema_version: DRIVE_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: DRIVE_TOOL_ID, + capability: DRIVE_CAPABILITY, + outcome, + reason_code, + duration_millis: duration_millis.min(30_000), + provenance: request.provenance.clone(), + coverage: DriveCoverage { + requested_units: 1, + completed_units, + unit: "WINDOW", + }, + data, + } +} + +pub(crate) fn classify_io(error: &std::io::Error) -> DriveTargetReasonCode { + match error.kind() { + std::io::ErrorKind::PermissionDenied | std::io::ErrorKind::ReadOnlyFilesystem => DriveTargetReasonCode::PermissionDenied, + std::io::ErrorKind::StorageFull | std::io::ErrorKind::FileTooLarge | std::io::ErrorKind::QuotaExceeded => { + DriveTargetReasonCode::LimitExceeded + } + std::io::ErrorKind::NotFound | std::io::ErrorKind::NotADirectory => DriveTargetReasonCode::SourceUnavailable, + _ => DriveTargetReasonCode::IoFailure, + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct DriveEnvelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: &'static str, + schema_version: u16, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: &'a str, + payload: DrivePayload<'a>, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct DrivePayload<'a> { + path: &'static str, + media_type: &'static str, + size_bytes: u64, + sha256: &'a str, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct DriveSignature<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +fn signature_document(key: &DeviceIdentity, key_id: &str, envelope: &[u8]) -> Result, DrivePerformanceError> { + let pkcs8 = key.to_pkcs8_der().map_err(|_| DrivePerformanceError::Signing)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| DrivePerformanceError::Signing)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + envelope.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.extend_from_slice(envelope); + let signature: Signature = signing_key.sign(&input); + serde_json::to_vec(&DriveSignature { + algorithm: "ES256", + key_id, + value: URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()), + }) + .map_err(|_| DrivePerformanceError::Encoding) +} + +fn archive(envelope: &[u8], signature: &[u8], result: &[u8]) -> Result, DrivePerformanceError> { + let cursor = Cursor::new(Vec::with_capacity(envelope.len() + signature.len() + result.len() + 512)); + let mut writer = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(OUTPUT_MODE); + for (name, bytes) in [(ENVELOPE_PATH, envelope), (SIGNATURE_PATH, signature), (RESULT_PATH, result)] { + writer + .start_file(name, options) + .map_err(|_| DrivePerformanceError::Encoding)?; + writer.write_all(bytes).map_err(DrivePerformanceError::Io)?; + } + writer + .finish() + .map(|cursor| cursor.into_inner()) + .map_err(|_| DrivePerformanceError::Encoding) +} + +fn resource_names_match(request: &DrivePerformanceRequest) -> bool { + let Some(organization_uid) = request.organization_name.strip_prefix("organizations/") else { + return false; + }; + if !uuid7(organization_uid) { + return false; + } + let cluster_prefix = format!("{}/clusters/", request.organization_name); + let Some(cluster_uid) = request.cluster_name.strip_prefix(&cluster_prefix) else { + return false; + }; + if !uuid7(cluster_uid) { + return false; + } + let device_prefix = format!("{}/clusterDevices/", request.cluster_name); + request.device_name.strip_prefix(&device_prefix).is_some_and(uuid7) +} + +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn lower_hex(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn build_feature(value: &str) -> bool { + value.len() <= 64 + && value.as_bytes().first().is_some_and(u8::is_ascii_lowercase) + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'_' | b'-')) +} + +fn version(value: &str) -> bool { + if value.is_empty() + || value.len() > 64 + || !value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'-')) + { + return false; + } + let (core, suffix) = value + .split_once('-') + .map_or((value, None), |(core, suffix)| (core, Some(suffix))); + if suffix.is_some_and(str::is_empty) { + return false; + } + let mut parts = core.split('.'); + parts.clone().count() == 3 && parts.all(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) +} + +fn timestamp(unix: i64) -> Result { + OffsetDateTime::from_unix_timestamp(unix) + .map_err(|_| DrivePerformanceError::InvalidRequest)? + .format(&Rfc3339) + .map_err(|_| DrivePerformanceError::InvalidRequest) +} + +fn unix_now() -> Result { + let duration = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| DrivePerformanceError::InvalidRequest)?; + i64::try_from(duration.as_secs()).map_err(|_| DrivePerformanceError::InvalidRequest) +} + +fn check_cancel(cancel: &CancellationToken) -> Result<(), DrivePerformanceError> { + if cancel.is_cancelled() { + Err(DrivePerformanceError::Cancelled) + } else { + Ok(()) + } +} + +fn elapsed_millis(duration: Duration) -> u64 { + elapsed_millis_allow_zero(duration).max(1) +} + +fn elapsed_millis_allow_zero(duration: Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).min(30_000) +} + +fn elapsed_micros(duration: Duration) -> u64 { + u64::try_from(duration.as_micros()) + .unwrap_or(MAX_SAFE_INTEGER) + .min(MAX_SAFE_INTEGER) +} + +fn hex_lower(bytes: &[u8]) -> String { + let mut value = String::with_capacity(bytes.len() * 2); + for byte in bytes { + use std::fmt::Write as _; + write!(&mut value, "{byte:02x}").expect("writing hexadecimal to a string cannot fail"); + } + value +} + +fn map_create_error(error: std::io::Error) -> DrivePerformanceError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + DrivePerformanceError::AlreadyExists + } else { + DrivePerformanceError::Io(error) + } +} + +fn map_publish_error(error: std::io::Error) -> DrivePerformanceError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + DrivePerformanceError::AlreadyExists + } else { + DrivePerformanceError::Io(error) + } +} diff --git a/rustfs/src/connect/diagnostics/perf_network.rs b/rustfs/src/connect/diagnostics/perf_network.rs new file mode 100644 index 000000000..04e184379 --- /dev/null +++ b/rustfs/src/connect/diagnostics/perf_network.rs @@ -0,0 +1,1018 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Consent-bound inter-node network performance results. + +use std::fs::{self, File, OpenOptions}; +use std::future::Future; +use std::io::{Cursor, Write as _}; +use std::path::Path; +use std::pin::Pin; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use base64_simd::URL_SAFE_NO_PAD; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use serde::Serialize; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use crate::connect::DeviceIdentity; +use crate::storage_api::cluster::network_probe::{NetworkPeerProbeClient, NetworkPeerProbeError as NativePeerProbeError}; + +pub const NETWORK_SCHEMA_VERSION: u16 = 1; +pub const NETWORK_TOOL_ID: &str = "performance.network"; +pub const NETWORK_CAPABILITY: &str = "performance.network@1"; +pub const MAX_NETWORK_DURATION: Duration = Duration::from_secs(30); +pub const MAX_TRAFFIC_BYTES: u64 = 1_048_576; +pub const MAX_BANDWIDTH_BYTES_PER_SECOND: u64 = 1_048_576; +pub const MAX_PEERS: usize = 256; +pub const MAX_OPERATIONS: usize = 1_024; +pub const MAX_RESULT_BYTES: usize = 262_144; +pub const MAX_ENVELOPE_BYTES: usize = 16_384; +pub const MAX_ARCHIVE_BYTES: usize = 524_288; +pub const MAX_DECOMPRESSED_BYTES: usize = 278_528; + +const MAX_SAFE_INTEGER: u64 = 9_007_199_254_740_991; +const MAX_BUILD_FEATURES: usize = 64; +const MAX_VALIDITY_SECONDS: i64 = 2_592_000; +const MAX_FUTURE_SKEW_SECONDS: i64 = 300; +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1\0"; +const ENVELOPE_PATH: &str = "envelope.json"; +const SIGNATURE_PATH: &str = "envelope.sig"; +const RESULT_PATH: &str = "result.json"; +const OUTPUT_MODE: u32 = 0o600; + +static NETWORK_COLLECTOR_ACTIVE: AtomicBool = AtomicBool::new(false); + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum NetworkOutcome { + Succeeded, + Partial, + Failed, + Unsupported, + Cancelled, +} + +impl NetworkOutcome { + pub const fn as_str(self) -> &'static str { + match self { + Self::Succeeded => "SUCCEEDED", + Self::Partial => "PARTIAL", + Self::Failed => "FAILED", + Self::Unsupported => "UNSUPPORTED", + Self::Cancelled => "CANCELLED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum NetworkReasonCode { + Complete, + LimitExceeded, + SourceUnavailable, + PermissionDenied, + UnsupportedTool, + UnsupportedVersion, + UnsupportedPlatform, + Cancelled, + InvalidInput, + CollectionFailed, +} + +impl NetworkReasonCode { + pub const fn as_str(self) -> &'static str { + match self { + Self::Complete => "COMPLETE", + Self::LimitExceeded => "LIMIT_EXCEEDED", + Self::SourceUnavailable => "SOURCE_UNAVAILABLE", + Self::PermissionDenied => "PERMISSION_DENIED", + Self::UnsupportedTool => "UNSUPPORTED_TOOL", + Self::UnsupportedVersion => "UNSUPPORTED_VERSION", + Self::UnsupportedPlatform => "UNSUPPORTED_PLATFORM", + Self::Cancelled => "CANCELLED", + Self::InvalidInput => "INVALID_INPUT", + Self::CollectionFailed => "COLLECTION_FAILED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum PeerReasonCode { + Complete, + Unreachable, + TimedOut, + ProtocolFailure, + LimitExceeded, + Cancelled, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct NetworkProvenance { + repository: &'static str, + source_commit: String, + executable_sha256: String, + rustfs_version: String, + os_family: NetworkOsFamily, + architecture: NetworkArchitecture, + build_features: Vec, +} + +impl NetworkProvenance { + pub fn new( + source_commit: impl Into, + executable_sha256: impl Into, + rustfs_version: impl Into, + build_features: Vec, + ) -> Self { + Self { + repository: "rustfs/rustfs", + source_commit: source_commit.into(), + executable_sha256: executable_sha256.into(), + rustfs_version: rustfs_version.into(), + os_family: NetworkOsFamily::current(), + architecture: NetworkArchitecture::current(), + build_features, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum NetworkOsFamily { + Linux, + Darwin, + Windows, + Freebsd, + Other, +} + +impl NetworkOsFamily { + fn current() -> Self { + match std::env::consts::OS { + "linux" => Self::Linux, + "macos" => Self::Darwin, + "windows" => Self::Windows, + "freebsd" => Self::Freebsd, + _ => Self::Other, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +enum NetworkArchitecture { + #[serde(rename = "x86_64")] + X86_64, + Aarch64, + Other, +} + +impl NetworkArchitecture { + fn current() -> Self { + match std::env::consts::ARCH { + "x86_64" => Self::X86_64, + "aarch64" => Self::Aarch64, + _ => Self::Other, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LocalNetworkConsent { + pub consent_uid: String, + pub policy_revision: u64, + pub expires_at_unix: i64, + pub confirmed: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct NetworkPerformanceRequest { + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub schema_version: u16, + pub capability: String, + pub consent: LocalNetworkConsent, + pub produced_at_unix: i64, + pub expires_at_unix: i64, + pub nonce: [u8; 32], + pub duration: Duration, + pub peer_aliases: Vec, + pub traffic_bytes_per_peer: u64, + pub provenance: NetworkProvenance, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct NetworkPerformanceData { + pub transferred_bytes: u64, + pub duration_millis: u64, + pub error_count: u64, + pub peer_count: u16, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct NetworkCoverage { + requested_units: u32, + completed_units: u32, + unit: &'static str, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct NetworkDiagnosticResult { + schema_version: u16, + run_uid: String, + tool_id: &'static str, + capability: &'static str, + outcome: NetworkOutcome, + reason_code: NetworkReasonCode, + duration_millis: u64, + provenance: NetworkProvenance, + coverage: NetworkCoverage, + data: Option, +} + +impl NetworkDiagnosticResult { + pub fn outcome(&self) -> NetworkOutcome { + self.outcome + } + + pub fn reason_code(&self) -> NetworkReasonCode { + self.reason_code + } + + pub fn data(&self) -> Option<&NetworkPerformanceData> { + self.data.as_ref() + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct NetworkPeerResult { + pub peer_alias: String, + pub outcome: NetworkOutcome, + pub reason_code: PeerReasonCode, + pub transferred_bytes: u64, + pub duration_millis: u64, + pub latency_micros: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct NetworkMeasurement { + pub result: NetworkDiagnosticResult, + pub peers: Vec, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct PeerProbeMeasurement { + pub transferred_bytes: u64, + pub duration: Duration, + pub latency: Duration, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum PeerProbeError { + Unreachable, + TimedOut, + ProtocolFailure, + Cancelled, +} + +pub type PeerProbeFuture<'a> = Pin> + Send + 'a>>; + +pub trait NetworkPeerHarness: Send + Sync { + fn probe<'a>(&'a self, peer_alias: &'a str, traffic_bytes: u64, cancel: &'a CancellationToken) -> PeerProbeFuture<'a>; +} + +struct RuntimeNetworkPeerHarness { + client: NetworkPeerProbeClient, +} + +impl NetworkPeerHarness for RuntimeNetworkPeerHarness { + fn probe<'a>(&'a self, peer_alias: &'a str, traffic_bytes: u64, cancel: &'a CancellationToken) -> PeerProbeFuture<'a> { + Box::pin(async move { + self.client + .probe(peer_alias, traffic_bytes, MAX_NETWORK_DURATION, cancel) + .await + .map(|measurement| PeerProbeMeasurement { + transferred_bytes: measurement.transferred_bytes, + duration: measurement.duration, + latency: measurement.latency, + }) + .map_err(map_native_probe_error) + }) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SignedNetworkExport { + pub artifact_uid: String, + pub outcome: NetworkOutcome, + pub reason_code: NetworkReasonCode, + pub envelope_json: Vec, + pub envelope_signature: Vec, + pub result_json: Vec, + pub archive_bytes: Vec, + pub archive_sha256: String, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SavedNetworkExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +#[derive(Debug, Error)] +pub enum NetworkPerformanceError { + #[error("network_performance_local_consent_required")] + ConsentRequired, + #[error("network_performance_local_consent_expired")] + ConsentExpired, + #[error("network_performance_request_expired")] + Expired, + #[error("network_performance_invalid_request")] + InvalidRequest, + #[error("network_performance_unsupported_version")] + UnsupportedVersion, + #[error("network_performance_unsupported_capability")] + UnsupportedCapability, + #[error("network_performance_limit_exceeded")] + LimitExceeded, + #[error("network_performance_collection_cancelled")] + Cancelled, + #[error("network_performance_collection_already_running")] + Busy, + #[error("network_performance_export_signing_failed")] + Signing, + #[error("network_performance_export_exists")] + AlreadyExists, + #[error("network_performance_export_io_failed")] + Io(#[source] std::io::Error), + #[error("network_performance_export_encoding_failed")] + Encoding, + #[error("network_performance_export_durability_failed_after_commit")] + DurabilityAfterCommit(#[source] std::io::Error), +} + +/// Return the current native capability state without generating test traffic. +pub async fn measure_network( + request: &NetworkPerformanceRequest, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + if cancel.is_cancelled() { + return Ok(terminal_measurement(request, NetworkOutcome::Cancelled, NetworkReasonCode::Cancelled)); + } + + let Some(endpoint_pools) = crate::runtime_sources::current_endpoints_handle() else { + return Ok(terminal_measurement( + request, + NetworkOutcome::Unsupported, + NetworkReasonCode::SourceUnavailable, + )); + }; + let harness = RuntimeNetworkPeerHarness { + client: NetworkPeerProbeClient::from_endpoint_pools(&endpoint_pools), + }; + let aliases = harness + .client + .targets() + .into_iter() + .map(|target| target.alias) + .collect::>(); + if aliases.is_empty() { + return Ok(terminal_measurement( + request, + NetworkOutcome::Unsupported, + NetworkReasonCode::SourceUnavailable, + )); + } + if aliases != request.peer_aliases { + return Err(NetworkPerformanceError::InvalidRequest); + } + measure_network_with_harness(request, &harness, cancel).await +} + +pub(crate) fn runtime_network_peer_aliases() -> Option> { + let endpoint_pools = crate::runtime_sources::current_endpoints_handle()?; + let aliases = NetworkPeerProbeClient::from_endpoint_pools(&endpoint_pools) + .targets() + .into_iter() + .map(|target| target.alias) + .collect::>(); + (!aliases.is_empty()).then_some(aliases) +} + +pub async fn measure_network_with_harness( + request: &NetworkPerformanceRequest, + harness: &dyn NetworkPeerHarness, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + if cancel.is_cancelled() { + return Ok(terminal_measurement(request, NetworkOutcome::Cancelled, NetworkReasonCode::Cancelled)); + } + let _lease = CollectorLease::acquire()?; + let started = Instant::now(); + let deadline = tokio::time::Instant::now() + request.duration; + let mut peers = Vec::with_capacity(request.peer_aliases.len()); + let mut transferred_bytes = 0_u64; + let mut completed_units = 0_u32; + + for alias in &request.peer_aliases { + let peer_started = Instant::now(); + let outcome = tokio::select! { + () = cancel.cancelled() => { + return Ok(cancelled_measurement(request, started.elapsed(), completed_units, peers)); + } + () = tokio::time::sleep_until(deadline) => Err(PeerProbeError::TimedOut), + result = harness.probe(alias, request.traffic_bytes_per_peer, cancel) => result, + }; + completed_units = completed_units.saturating_add(1); + match outcome { + Ok(value) => { + let bounded = value.transferred_bytes <= MAX_TRAFFIC_BYTES + && value.duration > Duration::ZERO + && value.duration <= request.duration + && value.latency <= value.duration + && value.duration.as_millis() <= u128::from(u64::MAX) + && value.latency.as_micros() <= u128::from(MAX_SAFE_INTEGER); + if bounded && value.transferred_bytes == request.traffic_bytes_per_peer { + transferred_bytes = transferred_bytes + .checked_add(value.transferred_bytes) + .ok_or(NetworkPerformanceError::LimitExceeded)?; + peers.push(NetworkPeerResult { + peer_alias: alias.clone(), + outcome: NetworkOutcome::Succeeded, + reason_code: PeerReasonCode::Complete, + transferred_bytes: value.transferred_bytes, + duration_millis: elapsed_millis(value.duration), + latency_micros: Some( + u64::try_from(value.latency.as_micros()).map_err(|_| NetworkPerformanceError::LimitExceeded)?, + ), + }); + } else if !bounded || value.transferred_bytes > request.traffic_bytes_per_peer { + peers.push(failed_peer( + alias, + PeerReasonCode::LimitExceeded, + elapsed_millis_allow_zero(peer_started.elapsed()), + )); + } else { + peers.push(failed_peer( + alias, + PeerReasonCode::ProtocolFailure, + elapsed_millis_allow_zero(peer_started.elapsed()), + )); + } + } + Err(PeerProbeError::Cancelled) => { + return Ok(cancelled_measurement(request, started.elapsed(), completed_units, peers)); + } + Err(error) => peers.push(failed_peer(alias, peer_reason(error), elapsed_millis_allow_zero(peer_started.elapsed()))), + } + } + + let elapsed = started.elapsed().min(request.duration); + let error_count = peers.iter().filter(|peer| peer.outcome != NetworkOutcome::Succeeded).count() as u64; + let outcome = match (error_count, peers.len()) { + (0, _) => NetworkOutcome::Succeeded, + (errors, count) if errors == count as u64 => NetworkOutcome::Failed, + _ => NetworkOutcome::Partial, + }; + let reason_code = if error_count == 0 { + NetworkReasonCode::Complete + } else { + NetworkReasonCode::CollectionFailed + }; + let data = if outcome == NetworkOutcome::Failed { + None + } else { + Some(NetworkPerformanceData { + transferred_bytes, + duration_millis: elapsed_millis(elapsed), + error_count, + peer_count: u16::try_from(peers.len()).map_err(|_| NetworkPerformanceError::LimitExceeded)?, + }) + }; + let result = result(request, outcome, reason_code, elapsed_millis_allow_zero(elapsed), completed_units, data); + Ok(NetworkMeasurement { result, peers }) +} + +pub fn sign_network_export( + request: &NetworkPerformanceRequest, + measurement: &NetworkMeasurement, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + check_cancel(cancel)?; + let result = &measurement.result; + if !matches!(result.outcome, NetworkOutcome::Succeeded | NetworkOutcome::Partial) + || result.data.is_none() + || result.run_uid != request.run_uid + || result.schema_version != NETWORK_SCHEMA_VERSION + || result.tool_id != NETWORK_TOOL_ID + || result.capability != NETWORK_CAPABILITY + { + return Err(NetworkPerformanceError::InvalidRequest); + } + let result_json = serde_json::to_vec(result).map_err(|_| NetworkPerformanceError::Encoding)?; + if result_json.is_empty() || result_json.len() > MAX_RESULT_BYTES { + return Err(NetworkPerformanceError::LimitExceeded); + } + let device_key_id = hex_lower(&Sha256::digest(key.public_key_der())); + let result_sha256 = hex_lower(&Sha256::digest(&result_json)); + let envelope = NetworkEnvelope { + format_version: "rustfs.connect.diagnosticEnvelope/1", + protocol_version: "v1", + organization_name: &request.organization_name, + cluster_name: &request.cluster_name, + device_name: &request.device_name, + run_uid: &request.run_uid, + artifact_uid: &request.artifact_uid, + tool_id: NETWORK_TOOL_ID, + schema_version: NETWORK_SCHEMA_VERSION, + classification: "L1", + consent_uid: &request.consent.consent_uid, + policy_revision: request.consent.policy_revision, + produced_at: timestamp(request.produced_at_unix)?, + expires_at: timestamp(request.expires_at_unix)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(request.nonce), + device_key_id: &device_key_id, + payload: NetworkPayload { + path: RESULT_PATH, + media_type: "application/json", + size_bytes: result_json.len() as u64, + sha256: &result_sha256, + }, + }; + let envelope_json = serde_json::to_vec(&envelope).map_err(|_| NetworkPerformanceError::Encoding)?; + if envelope_json.is_empty() || envelope_json.len() > MAX_ENVELOPE_BYTES { + return Err(NetworkPerformanceError::LimitExceeded); + } + let envelope_signature = signature_document(key, &device_key_id, &envelope_json)?; + let decompressed = result_json + .len() + .checked_add(envelope_json.len()) + .and_then(|size| size.checked_add(envelope_signature.len())) + .ok_or(NetworkPerformanceError::LimitExceeded)?; + if decompressed > MAX_DECOMPRESSED_BYTES { + return Err(NetworkPerformanceError::LimitExceeded); + } + check_cancel(cancel)?; + if unix_now()? >= request.expires_at_unix { + return Err(NetworkPerformanceError::Expired); + } + let archive_bytes = archive(&envelope_json, &envelope_signature, &result_json)?; + if archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(NetworkPerformanceError::LimitExceeded); + } + let archive_sha256 = hex_lower(&Sha256::digest(&archive_bytes)); + Ok(SignedNetworkExport { + artifact_uid: request.artifact_uid.clone(), + outcome: result.outcome, + reason_code: result.reason_code, + envelope_json, + envelope_signature, + result_json, + archive_bytes, + archive_sha256, + }) +} + +pub fn save_signed_network_export( + output: &Path, + export: &SignedNetworkExport, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + if !uuid7(&export.artifact_uid) { + return Err(NetworkPerformanceError::InvalidRequest); + } + let parent = output + .parent() + .filter(|path| !path.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let filename = output + .file_name() + .ok_or(NetworkPerformanceError::InvalidRequest)? + .to_string_lossy(); + let temporary = parent.join(format!(".{filename}.{}.partial", export.artifact_uid)); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + let mut file = options.open(&temporary).map_err(map_create_error)?; + let saved = (|| { + file.write_all(&export.archive_bytes).map_err(NetworkPerformanceError::Io)?; + check_cancel(cancel)?; + file.sync_all().map_err(NetworkPerformanceError::Io)?; + check_cancel(cancel)?; + fs::hard_link(&temporary, output).map_err(map_publish_error)?; + fs::remove_file(&temporary).map_err(NetworkPerformanceError::DurabilityAfterCommit)?; + #[cfg(unix)] + File::open(parent) + .and_then(|directory| directory.sync_all()) + .map_err(NetworkPerformanceError::DurabilityAfterCommit)?; + Ok(SavedNetworkExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: export.archive_bytes.len() as u64, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if saved.is_err() { + let _ = fs::remove_file(&temporary); + } + saved +} + +impl NetworkPerformanceRequest { + fn validate(&self, now_unix: i64) -> Result<(), NetworkPerformanceError> { + if self.schema_version != NETWORK_SCHEMA_VERSION { + return Err(NetworkPerformanceError::UnsupportedVersion); + } + if self.capability != NETWORK_CAPABILITY { + return Err(NetworkPerformanceError::UnsupportedCapability); + } + if !self.consent.confirmed || self.consent.policy_revision == 0 { + return Err(NetworkPerformanceError::ConsentRequired); + } + if self.consent.expires_at_unix <= now_unix || self.expires_at_unix > self.consent.expires_at_unix { + return Err(NetworkPerformanceError::ConsentExpired); + } + let validity = self + .expires_at_unix + .checked_sub(self.produced_at_unix) + .ok_or(NetworkPerformanceError::Expired)?; + if self.produced_at_unix > now_unix.saturating_add(MAX_FUTURE_SKEW_SECONDS) + || validity <= 0 + || validity > MAX_VALIDITY_SECONDS + || self.expires_at_unix <= now_unix + { + return Err(NetworkPerformanceError::Expired); + } + if self.duration.is_zero() + || self.duration.as_millis() == 0 + || self.duration > MAX_NETWORK_DURATION + || self.peer_aliases.is_empty() + || self.peer_aliases.len() > MAX_PEERS + || self.peer_aliases.len() > MAX_OPERATIONS + || self.traffic_bytes_per_peer == 0 + { + return Err(NetworkPerformanceError::LimitExceeded); + } + let traffic = self + .traffic_bytes_per_peer + .checked_mul(self.peer_aliases.len() as u64) + .ok_or(NetworkPerformanceError::LimitExceeded)?; + let bandwidth_budget = u64::try_from( + u128::from(MAX_BANDWIDTH_BYTES_PER_SECOND) + .checked_mul(self.duration.as_millis()) + .ok_or(NetworkPerformanceError::LimitExceeded)? + / 1_000, + ) + .map_err(|_| NetworkPerformanceError::LimitExceeded)?; + if traffic > MAX_TRAFFIC_BYTES || traffic > bandwidth_budget.max(1) { + return Err(NetworkPerformanceError::LimitExceeded); + } + if !uuid7(&self.run_uid) + || !uuid7(&self.artifact_uid) + || !uuid7(&self.consent.consent_uid) + || !resource_names_match(self) + || !lower_hex(&self.provenance.source_commit, 40) + || !lower_hex(&self.provenance.executable_sha256, 64) + || !version(&self.provenance.rustfs_version) + || self.provenance.build_features.len() > MAX_BUILD_FEATURES + || !self.provenance.build_features.iter().all(|value| build_feature(value)) + || !valid_peer_aliases(&self.peer_aliases) + { + return Err(NetworkPerformanceError::InvalidRequest); + } + Ok(()) + } +} + +struct CollectorLease; + +impl CollectorLease { + fn acquire() -> Result { + NETWORK_COLLECTOR_ACTIVE + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map(|_| Self) + .map_err(|_| NetworkPerformanceError::Busy) + } +} + +impl Drop for CollectorLease { + fn drop(&mut self) { + NETWORK_COLLECTOR_ACTIVE.store(false, Ordering::Release); + } +} + +fn terminal_measurement( + request: &NetworkPerformanceRequest, + outcome: NetworkOutcome, + reason_code: NetworkReasonCode, +) -> NetworkMeasurement { + NetworkMeasurement { + result: result(request, outcome, reason_code, 0, 0, None), + peers: Vec::new(), + } +} + +fn cancelled_measurement( + request: &NetworkPerformanceRequest, + elapsed: Duration, + completed_units: u32, + peers: Vec, +) -> NetworkMeasurement { + NetworkMeasurement { + result: result( + request, + NetworkOutcome::Cancelled, + NetworkReasonCode::Cancelled, + elapsed_millis_allow_zero(elapsed), + completed_units, + None, + ), + peers, + } +} + +fn result( + request: &NetworkPerformanceRequest, + outcome: NetworkOutcome, + reason_code: NetworkReasonCode, + duration_millis: u64, + completed_units: u32, + data: Option, +) -> NetworkDiagnosticResult { + NetworkDiagnosticResult { + schema_version: NETWORK_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: NETWORK_TOOL_ID, + capability: NETWORK_CAPABILITY, + outcome, + reason_code, + duration_millis: duration_millis.min(30_000), + provenance: request.provenance.clone(), + coverage: NetworkCoverage { + requested_units: request.peer_aliases.len() as u32, + completed_units, + unit: "OPERATION", + }, + data, + } +} + +fn failed_peer(alias: &str, reason_code: PeerReasonCode, duration_millis: u64) -> NetworkPeerResult { + NetworkPeerResult { + peer_alias: alias.to_owned(), + outcome: NetworkOutcome::Failed, + reason_code, + transferred_bytes: 0, + duration_millis, + latency_micros: None, + } +} + +fn peer_reason(error: PeerProbeError) -> PeerReasonCode { + match error { + PeerProbeError::Unreachable => PeerReasonCode::Unreachable, + PeerProbeError::TimedOut => PeerReasonCode::TimedOut, + PeerProbeError::ProtocolFailure => PeerReasonCode::ProtocolFailure, + PeerProbeError::Cancelled => PeerReasonCode::Cancelled, + } +} + +fn map_native_probe_error(error: NativePeerProbeError) -> PeerProbeError { + match error { + NativePeerProbeError::Cancelled => PeerProbeError::Cancelled, + NativePeerProbeError::Unreachable => PeerProbeError::Unreachable, + NativePeerProbeError::TimedOut => PeerProbeError::TimedOut, + NativePeerProbeError::UnknownPeer | NativePeerProbeError::LimitExceeded | NativePeerProbeError::ProtocolFailure => { + PeerProbeError::ProtocolFailure + } + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct NetworkEnvelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: &'static str, + schema_version: u16, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: &'a str, + payload: NetworkPayload<'a>, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct NetworkPayload<'a> { + path: &'static str, + media_type: &'static str, + size_bytes: u64, + sha256: &'a str, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct NetworkSignature<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +fn signature_document(key: &DeviceIdentity, key_id: &str, envelope: &[u8]) -> Result, NetworkPerformanceError> { + let pkcs8 = key.to_pkcs8_der().map_err(|_| NetworkPerformanceError::Signing)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| NetworkPerformanceError::Signing)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + envelope.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.extend_from_slice(envelope); + let signature: Signature = signing_key.sign(&input); + serde_json::to_vec(&NetworkSignature { + algorithm: "ES256", + key_id, + value: URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()), + }) + .map_err(|_| NetworkPerformanceError::Encoding) +} + +fn archive(envelope: &[u8], signature: &[u8], result: &[u8]) -> Result, NetworkPerformanceError> { + let cursor = Cursor::new(Vec::with_capacity(envelope.len() + signature.len() + result.len() + 512)); + let mut writer = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(OUTPUT_MODE); + for (name, bytes) in [(ENVELOPE_PATH, envelope), (SIGNATURE_PATH, signature), (RESULT_PATH, result)] { + writer + .start_file(name, options) + .map_err(|_| NetworkPerformanceError::Encoding)?; + writer.write_all(bytes).map_err(NetworkPerformanceError::Io)?; + } + writer + .finish() + .map(|cursor| cursor.into_inner()) + .map_err(|_| NetworkPerformanceError::Encoding) +} + +fn resource_names_match(request: &NetworkPerformanceRequest) -> bool { + let Some(organization_uid) = request.organization_name.strip_prefix("organizations/") else { + return false; + }; + if !uuid7(organization_uid) { + return false; + } + let cluster_prefix = format!("{}/clusters/", request.organization_name); + let Some(cluster_uid) = request.cluster_name.strip_prefix(&cluster_prefix) else { + return false; + }; + if !uuid7(cluster_uid) { + return false; + } + let device_prefix = format!("{}/clusterDevices/", request.cluster_name); + request.device_name.strip_prefix(&device_prefix).is_some_and(uuid7) +} + +fn valid_peer_aliases(aliases: &[String]) -> bool { + aliases + .iter() + .enumerate() + .all(|(index, alias)| alias == &format!("peer-{}", index + 1)) +} + +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn lower_hex(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn build_feature(value: &str) -> bool { + value.len() <= 64 + && value.as_bytes().first().is_some_and(u8::is_ascii_lowercase) + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'_' | b'-')) +} + +fn version(value: &str) -> bool { + if value.is_empty() + || value.len() > 64 + || !value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'-')) + { + return false; + } + let (core, suffix) = value + .split_once('-') + .map_or((value, None), |(core, suffix)| (core, Some(suffix))); + if suffix.is_some_and(str::is_empty) { + return false; + } + let mut parts = core.split('.'); + parts.clone().count() == 3 && parts.all(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) +} + +fn timestamp(unix: i64) -> Result { + OffsetDateTime::from_unix_timestamp(unix) + .map_err(|_| NetworkPerformanceError::InvalidRequest)? + .format(&Rfc3339) + .map_err(|_| NetworkPerformanceError::InvalidRequest) +} + +fn unix_now() -> Result { + let duration = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| NetworkPerformanceError::InvalidRequest)?; + i64::try_from(duration.as_secs()).map_err(|_| NetworkPerformanceError::InvalidRequest) +} + +fn check_cancel(cancel: &CancellationToken) -> Result<(), NetworkPerformanceError> { + if cancel.is_cancelled() { + Err(NetworkPerformanceError::Cancelled) + } else { + Ok(()) + } +} + +fn elapsed_millis(duration: Duration) -> u64 { + elapsed_millis_allow_zero(duration).max(1) +} + +fn elapsed_millis_allow_zero(duration: Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).min(30_000) +} + +fn hex_lower(bytes: &[u8]) -> String { + let mut value = String::with_capacity(bytes.len() * 2); + for byte in bytes { + use std::fmt::Write as _; + write!(&mut value, "{byte:02x}").expect("writing hexadecimal to a string cannot fail"); + } + value +} + +fn map_create_error(error: std::io::Error) -> NetworkPerformanceError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + NetworkPerformanceError::AlreadyExists + } else { + NetworkPerformanceError::Io(error) + } +} + +fn map_publish_error(error: std::io::Error) -> NetworkPerformanceError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + NetworkPerformanceError::AlreadyExists + } else { + NetworkPerformanceError::Io(error) + } +} diff --git a/rustfs/src/connect/diagnostics/perf_object.rs b/rustfs/src/connect/diagnostics/perf_object.rs new file mode 100644 index 000000000..dda98f8ea --- /dev/null +++ b/rustfs/src/connect/diagnostics/perf_object.rs @@ -0,0 +1,1345 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Consent-bound object-to-deployment performance measurement. +//! +//! The producer creates a dedicated temporary bucket, writes or reads one +//! generated object, and removes the namespace before returning. It never +//! accepts customer bucket or object names and keeps credentials local. + +use std::fs::{self, File, OpenOptions}; +use std::future::Future; +use std::io::{Cursor, Read, Write as _}; +use std::path::Path; +use std::pin::Pin; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use base64_simd::URL_SAFE_NO_PAD; +use bytes::Bytes; +use futures::StreamExt as _; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use reqwest::{Client, Method, Response, StatusCode, Url}; +use serde::Serialize; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zeroize::Zeroizing; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use crate::connect::DeviceIdentity; + +pub const OBJECT_SCHEMA_VERSION: u16 = 1; +pub const OBJECT_TOOL_ID: &str = "performance.object"; +pub const OBJECT_CAPABILITY: &str = "performance.object@1"; +pub const MAX_OBJECT_DURATION: Duration = Duration::from_secs(30); +pub const MAX_OBJECT_TRAFFIC_BYTES: u64 = 1_048_576; +pub const MAX_OBJECT_BANDWIDTH_BYTES_PER_SECOND: u64 = 1_048_576; +pub const MAX_OBJECT_RESULT_BYTES: usize = 262_144; + +const MAX_SAFE_INTEGER: u64 = 9_007_199_254_740_991; +const MAX_BUILD_FEATURES: usize = 64; +const MAX_VALIDITY_SECONDS: i64 = 2_592_000; +const MAX_FUTURE_SKEW_SECONDS: i64 = 300; +const MAX_ENVELOPE_BYTES: usize = 16_384; +const MAX_ARCHIVE_BYTES: usize = 524_288; +const MAX_DECOMPRESSED_BYTES: usize = 278_528; +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1\0"; +const ENVELOPE_PATH: &str = "envelope.json"; +const SIGNATURE_PATH: &str = "envelope.sig"; +const RESULT_PATH: &str = "result.json"; +const MAX_OBJECT_RESPONSE_BYTES: usize = 16_384; +const CLEANUP_RESERVE_MAX: Duration = Duration::from_millis(250); +const OUTPUT_MODE: u32 = 0o600; + +static OBJECT_COLLECTOR_ACTIVE: AtomicBool = AtomicBool::new(false); + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ObjectOperation { + GetObject, + PutObject, +} + +impl ObjectOperation { + pub const fn as_str(self) -> &'static str { + match self { + Self::GetObject => "GET_OBJECT", + Self::PutObject => "PUT_OBJECT", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ObjectOutcome { + Succeeded, + Failed, + Cancelled, +} + +impl ObjectOutcome { + pub const fn as_str(self) -> &'static str { + match self { + Self::Succeeded => "SUCCEEDED", + Self::Failed => "FAILED", + Self::Cancelled => "CANCELLED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ObjectReasonCode { + Complete, + SourceUnavailable, + PermissionDenied, + Cancelled, + CollectionFailed, +} + +impl ObjectReasonCode { + pub const fn as_str(self) -> &'static str { + match self { + Self::Complete => "COMPLETE", + Self::SourceUnavailable => "SOURCE_UNAVAILABLE", + Self::PermissionDenied => "PERMISSION_DENIED", + Self::Cancelled => "CANCELLED", + Self::CollectionFailed => "COLLECTION_FAILED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ObjectTargetReasonCode { + Complete, + EndpointUnavailable, + ProxyFailure, + PermissionDenied, + TimedOut, + Cancelled, + NamespaceConflict, + CleanupFailed, + ProtocolFailure, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ObjectTargetParameters { + pub operation: ObjectOperation, + pub requested_bytes: u64, + pub duration_millis: u64, + pub concurrency: u8, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ObjectTargetUnits { + pub bytes: &'static str, + pub duration: &'static str, + pub latency: &'static str, + pub operation_count: &'static str, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ObjectProvenance { + repository: &'static str, + source_commit: String, + executable_sha256: String, + rustfs_version: String, + os_family: ObjectOsFamily, + architecture: ObjectArchitecture, + build_features: Vec, +} + +impl ObjectProvenance { + pub fn new( + source_commit: impl Into, + executable_sha256: impl Into, + rustfs_version: impl Into, + build_features: Vec, + ) -> Self { + Self { + repository: "rustfs/rustfs", + source_commit: source_commit.into(), + executable_sha256: executable_sha256.into(), + rustfs_version: rustfs_version.into(), + os_family: ObjectOsFamily::current(), + architecture: ObjectArchitecture::current(), + build_features, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum ObjectOsFamily { + Linux, + Darwin, + Windows, + Freebsd, + Other, +} + +impl ObjectOsFamily { + fn current() -> Self { + match std::env::consts::OS { + "linux" => Self::Linux, + "macos" => Self::Darwin, + "windows" => Self::Windows, + "freebsd" => Self::Freebsd, + _ => Self::Other, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +enum ObjectArchitecture { + #[serde(rename = "x86_64")] + X86_64, + Aarch64, + Other, +} + +impl ObjectArchitecture { + fn current() -> Self { + match std::env::consts::ARCH { + "x86_64" => Self::X86_64, + "aarch64" => Self::Aarch64, + _ => Self::Other, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LocalObjectConsent { + pub consent_uid: String, + pub policy_revision: u64, + pub expires_at_unix: i64, + pub confirmed: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ObjectPerformanceRequest { + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub schema_version: u16, + pub capability: String, + pub consent: LocalObjectConsent, + pub produced_at_unix: i64, + pub expires_at_unix: i64, + pub nonce: [u8; 32], + pub duration: Duration, + pub operation: ObjectOperation, + pub traffic_bytes: u64, + pub target_alias: String, + pub provenance: ObjectProvenance, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ObjectPerformanceData { + pub operation: ObjectOperation, + pub transferred_bytes: u64, + pub completed_operations: u64, + pub duration_millis: u64, + pub error_count: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +struct ObjectCoverage { + requested_units: u32, + completed_units: u32, + unit: &'static str, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ObjectDiagnosticResult { + schema_version: u16, + run_uid: String, + tool_id: &'static str, + capability: &'static str, + outcome: ObjectOutcome, + reason_code: ObjectReasonCode, + duration_millis: u64, + provenance: ObjectProvenance, + coverage: ObjectCoverage, + data: Option, +} + +impl ObjectDiagnosticResult { + pub fn outcome(&self) -> ObjectOutcome { + self.outcome + } + + pub fn reason_code(&self) -> ObjectReasonCode { + self.reason_code + } + + pub fn data(&self) -> Option<&ObjectPerformanceData> { + self.data.as_ref() + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ObjectTargetResult { + pub target_alias: String, + pub outcome: ObjectOutcome, + pub reason_code: ObjectTargetReasonCode, + pub parameters: ObjectTargetParameters, + pub units: ObjectTargetUnits, + pub transferred_bytes: u64, + pub completed_operations: u64, + pub latency_micros: Option, + pub duration_millis: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ObjectMeasurement { + pub result: ObjectDiagnosticResult, + pub target: ObjectTargetResult, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct ObjectProbeMeasurement { + pub transferred_bytes: u64, + pub duration: Duration, + pub latency: Duration, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ObjectProbeError { + EndpointUnavailable, + ProxyFailure, + PermissionDenied, + TimedOut, + Cancelled, + NamespaceConflict, + CleanupFailed, + ProtocolFailure, +} + +pub type ObjectProbeFuture<'a> = Pin> + Send + 'a>>; + +pub trait ObjectProbe: Send + Sync { + fn probe<'a>(&'a self, request: &'a ObjectPerformanceRequest, cancel: &'a CancellationToken) -> ObjectProbeFuture<'a>; +} + +pub struct S3ObjectProbe { + endpoint: Url, + client: Client, + access_key: Zeroizing, + secret_key: Zeroizing, + session_token: Zeroizing, + proxy_configured: bool, +} + +impl S3ObjectProbe { + pub fn new( + endpoint: &str, + root_ca_pem: Option<&[u8]>, + proxy: Option<&str>, + access_key: Zeroizing, + secret_key: Zeroizing, + session_token: Zeroizing, + timeout: Duration, + ) -> Result { + let endpoint = deployment_endpoint(endpoint)?; + if access_key.is_empty() || secret_key.is_empty() { + return Err(ObjectPerformanceError::InvalidCredential); + } + let mut builder = Client::builder() + .no_proxy() + .redirect(reqwest::redirect::Policy::none()) + .timeout(timeout); + if let Some(root_ca_pem) = root_ca_pem { + let certificate = + reqwest::Certificate::from_pem(root_ca_pem).map_err(|_| ObjectPerformanceError::InvalidRootCertificate)?; + builder = builder.add_root_certificate(certificate); + } + if let Some(proxy) = proxy { + let proxy_url = proxy_url(proxy)?; + builder = builder.proxy(reqwest::Proxy::all(proxy_url).map_err(|_| ObjectPerformanceError::InvalidProxy)?); + } + let client = builder.build().map_err(|_| ObjectPerformanceError::TransportConfiguration)?; + Ok(Self { + endpoint, + client, + access_key, + secret_key, + session_token, + proxy_configured: proxy.is_some(), + }) + } + + async fn execute( + &self, + request: &ObjectPerformanceRequest, + cancel: &CancellationToken, + ) -> Result { + let started = Instant::now(); + let reserve = CLEANUP_RESERVE_MAX.min(request.duration / 10); + let operation_deadline = started + request.duration.saturating_sub(reserve); + let cleanup_deadline = started + request.duration; + let bucket = format!("rustfs-connect-perf-{}", request.artifact_uid.replace('-', "")); + let object = "synthetic-object"; + let bucket_url = self.object_url(&bucket, None)?; + let object_url = self.object_url(&bucket, Some(object))?; + let payload_len = usize::try_from(request.traffic_bytes).map_err(|_| ObjectProbeError::ProtocolFailure)?; + let payload = Bytes::from(vec![0xa5; payload_len]); + + let create = self + .send(Method::PUT, bucket_url.clone(), Bytes::new(), operation_deadline, Some(cancel)) + .await?; + if create.status() == StatusCode::CONFLICT { + return Err(ObjectProbeError::NamespaceConflict); + } + if !create.status().is_success() { + return self.status_error(create.status()); + } + let measurement = match self.require_success(create, operation_deadline, Some(cancel)).await { + Ok(()) => { + self.measure_in_namespace(request, object_url.clone(), payload, operation_deadline, cancel) + .await + } + Err(error) => Err(error), + }; + let cleanup = self.cleanup(object_url, bucket_url, cleanup_deadline).await; + if cleanup.is_err() { + return Err(ObjectProbeError::CleanupFailed); + } + measurement.map(|(transferred_bytes, latency)| ObjectProbeMeasurement { + transferred_bytes, + duration: latency, + latency, + }) + } + + async fn measure_in_namespace( + &self, + request: &ObjectPerformanceRequest, + object_url: Url, + payload: Bytes, + deadline: Instant, + cancel: &CancellationToken, + ) -> Result<(u64, Duration), ObjectProbeError> { + if request.operation == ObjectOperation::GetObject { + let preload = self + .send(Method::PUT, object_url.clone(), payload.clone(), deadline, Some(cancel)) + .await?; + self.require_success(preload, deadline, Some(cancel)).await?; + } + + let operation_started = Instant::now(); + match request.operation { + ObjectOperation::PutObject => { + let response = self.send(Method::PUT, object_url, payload, deadline, Some(cancel)).await?; + self.require_success(response, deadline, Some(cancel)).await?; + } + ObjectOperation::GetObject => { + let response = self + .send(Method::GET, object_url, Bytes::new(), deadline, Some(cancel)) + .await?; + if !response.status().is_success() { + return self.status_error(response.status()); + } + let body = self + .read_response_body(response, request.traffic_bytes, deadline, Some(cancel)) + .await?; + if body.len() != usize::try_from(request.traffic_bytes).map_err(|_| ObjectProbeError::ProtocolFailure)? + || body.iter().any(|byte| *byte != 0xa5) + { + return Err(ObjectProbeError::ProtocolFailure); + } + } + } + Ok((request.traffic_bytes, operation_started.elapsed())) + } + + async fn cleanup(&self, object_url: Url, bucket_url: Url, deadline: Instant) -> Result<(), ObjectProbeError> { + let object = self.send(Method::DELETE, object_url, Bytes::new(), deadline, None).await?; + if !object.status().is_success() && object.status() != StatusCode::NOT_FOUND { + return Err(ObjectProbeError::CleanupFailed); + } + let bucket = self.send(Method::DELETE, bucket_url, Bytes::new(), deadline, None).await?; + if !bucket.status().is_success() { + return Err(ObjectProbeError::CleanupFailed); + } + Ok(()) + } + + async fn send( + &self, + method: Method, + url: Url, + payload: Bytes, + deadline: Instant, + cancel: Option<&CancellationToken>, + ) -> Result { + let payload_hash = hex_lower(&Sha256::digest(&payload)); + let unsigned = http::Request::builder() + .method(method.clone()) + .uri(url.as_str()) + .header("x-amz-content-sha256", payload_hash) + .body(()) + .map_err(|_| ObjectProbeError::ProtocolFailure)?; + let signed_headers = rustfs_signer::try_sign_v4_headers( + unsigned.into_parts().0, + i64::try_from(payload.len()).map_err(|_| ObjectProbeError::ProtocolFailure)?, + &self.access_key, + &self.secret_key, + &self.session_token, + "us-east-1", + ) + .map_err(|_| ObjectProbeError::ProtocolFailure)?; + let send = self.client.request(method, url).headers(signed_headers).body(payload).send(); + let timed = tokio::time::timeout_at(tokio::time::Instant::from_std(deadline), send); + let response = if let Some(cancel) = cancel { + tokio::select! { + () = cancel.cancelled() => return Err(ObjectProbeError::Cancelled), + response = timed => response, + } + } else { + timed.await + }; + response + .map_err(|_| ObjectProbeError::TimedOut)? + .map_err(|error| self.transport_error(&error)) + } + + async fn require_success( + &self, + response: Response, + deadline: Instant, + cancel: Option<&CancellationToken>, + ) -> Result<(), ObjectProbeError> { + if !response.status().is_success() { + return self.status_error(response.status()); + } + self.read_response_body(response, MAX_OBJECT_RESPONSE_BYTES as u64, deadline, cancel) + .await + .map(|_| ()) + } + + async fn read_response_body( + &self, + response: Response, + max_bytes: u64, + deadline: Instant, + cancel: Option<&CancellationToken>, + ) -> Result, ObjectProbeError> { + if response.content_length().is_some_and(|length| length > max_bytes) { + return Err(ObjectProbeError::ProtocolFailure); + } + let max_bytes = usize::try_from(max_bytes).map_err(|_| ObjectProbeError::ProtocolFailure)?; + let mut body = Vec::with_capacity(max_bytes.min(MAX_OBJECT_RESPONSE_BYTES)); + let mut stream = response.bytes_stream(); + loop { + let next = tokio::time::timeout_at(tokio::time::Instant::from_std(deadline), stream.next()); + let chunk = if let Some(cancel) = cancel { + tokio::select! { + () = cancel.cancelled() => return Err(ObjectProbeError::Cancelled), + chunk = next => chunk, + } + } else { + next.await + } + .map_err(|_| ObjectProbeError::TimedOut)?; + let Some(chunk) = chunk else { + break; + }; + let chunk = chunk.map_err(|error| self.transport_error(&error))?; + let length = body.len().checked_add(chunk.len()).ok_or(ObjectProbeError::ProtocolFailure)?; + if length > max_bytes { + return Err(ObjectProbeError::ProtocolFailure); + } + body.extend_from_slice(&chunk); + } + Ok(body) + } + + fn object_url(&self, bucket: &str, object: Option<&str>) -> Result { + let path = object.map_or_else(|| format!("{bucket}/"), |object| format!("{bucket}/{object}")); + self.endpoint.join(&path).map_err(|_| ObjectProbeError::ProtocolFailure) + } + + fn status_error(&self, status: StatusCode) -> Result { + if status == StatusCode::UNAUTHORIZED || status == StatusCode::FORBIDDEN { + Err(ObjectProbeError::PermissionDenied) + } else { + Err(ObjectProbeError::ProtocolFailure) + } + } + + fn transport_error(&self, error: &reqwest::Error) -> ObjectProbeError { + if error.is_timeout() { + ObjectProbeError::TimedOut + } else if self.proxy_configured { + ObjectProbeError::ProxyFailure + } else if error.is_connect() { + ObjectProbeError::EndpointUnavailable + } else { + ObjectProbeError::ProtocolFailure + } + } +} + +impl ObjectProbe for S3ObjectProbe { + fn probe<'a>(&'a self, request: &'a ObjectPerformanceRequest, cancel: &'a CancellationToken) -> ObjectProbeFuture<'a> { + Box::pin(self.execute(request, cancel)) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SignedObjectExport { + pub artifact_uid: String, + pub outcome: ObjectOutcome, + pub reason_code: ObjectReasonCode, + pub envelope_json: Vec, + pub envelope_signature: Vec, + pub result_json: Vec, + pub archive_bytes: Vec, + pub archive_sha256: String, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SavedObjectExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +#[derive(Debug, Error)] +pub enum ObjectPerformanceError { + #[error("object_performance_local_consent_required")] + ConsentRequired, + #[error("object_performance_local_consent_expired")] + ConsentExpired, + #[error("object_performance_request_expired")] + Expired, + #[error("object_performance_invalid_request")] + InvalidRequest, + #[error("object_performance_unsupported_version")] + UnsupportedVersion, + #[error("object_performance_unsupported_capability")] + UnsupportedCapability, + #[error("object_performance_limit_exceeded")] + LimitExceeded, + #[error("object_performance_collection_cancelled")] + Cancelled, + #[error("object_performance_busy")] + Busy, + #[error("object_performance_invalid_endpoint")] + InvalidEndpoint, + #[error("object_performance_invalid_proxy")] + InvalidProxy, + #[error("object_performance_invalid_root_certificate")] + InvalidRootCertificate, + #[error("object_performance_invalid_credential")] + InvalidCredential, + #[error("object_performance_transport_configuration")] + TransportConfiguration, + #[error("object_performance_signing_failed")] + Signing, + #[error("object_performance_encoding_failed")] + Encoding, + #[error("object_performance_output_exists")] + AlreadyExists, + #[error("object_performance_io_failed")] + Io(#[source] std::io::Error), + #[error("object_performance_output_durability_failed")] + DurabilityAfterCommit(#[source] std::io::Error), +} + +pub async fn measure_object( + request: &ObjectPerformanceRequest, + probe: &impl ObjectProbe, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + if OBJECT_COLLECTOR_ACTIVE + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .is_err() + { + return Err(ObjectPerformanceError::Busy); + } + let _guard = ActiveGuard; + if cancel.is_cancelled() { + return Ok(terminal_measurement( + request, + ObjectOutcome::Cancelled, + ObjectReasonCode::Cancelled, + ObjectTargetReasonCode::Cancelled, + Duration::ZERO, + )); + } + let started = Instant::now(); + // The concrete probe owns the task namespace and must observe cancellation + // before returning so its async cleanup cannot be dropped halfway through. + let result = probe.probe(request, cancel).await; + Ok(match result { + Ok(sample) if sample.transferred_bytes == request.traffic_bytes => success_measurement(request, sample), + Ok(_) => failed_measurement(request, started.elapsed(), ObjectTargetReasonCode::ProtocolFailure), + Err(ObjectProbeError::Cancelled) => terminal_measurement( + request, + ObjectOutcome::Cancelled, + ObjectReasonCode::Cancelled, + ObjectTargetReasonCode::Cancelled, + started.elapsed(), + ), + Err(error) => failed_measurement(request, started.elapsed(), target_reason(error)), + }) +} + +pub fn sign_object_export( + request: &ObjectPerformanceRequest, + measurement: &ObjectMeasurement, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + check_cancel(cancel)?; + let result = &measurement.result; + if result.outcome != ObjectOutcome::Succeeded + || result.data.is_none() + || result.run_uid != request.run_uid + || result.schema_version != OBJECT_SCHEMA_VERSION + || result.tool_id != OBJECT_TOOL_ID + || result.capability != OBJECT_CAPABILITY + { + return Err(ObjectPerformanceError::InvalidRequest); + } + let result_json = serde_json::to_vec(result).map_err(|_| ObjectPerformanceError::Encoding)?; + if result_json.is_empty() || result_json.len() > MAX_OBJECT_RESULT_BYTES { + return Err(ObjectPerformanceError::LimitExceeded); + } + let device_key_id = hex_lower(&Sha256::digest(key.public_key_der())); + let result_sha256 = hex_lower(&Sha256::digest(&result_json)); + let envelope = ObjectEnvelope { + format_version: "rustfs.connect.diagnosticEnvelope/1", + protocol_version: "v1", + organization_name: &request.organization_name, + cluster_name: &request.cluster_name, + device_name: &request.device_name, + run_uid: &request.run_uid, + artifact_uid: &request.artifact_uid, + tool_id: OBJECT_TOOL_ID, + schema_version: OBJECT_SCHEMA_VERSION, + classification: "L1", + consent_uid: &request.consent.consent_uid, + policy_revision: request.consent.policy_revision, + produced_at: timestamp(request.produced_at_unix)?, + expires_at: timestamp(request.expires_at_unix)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(request.nonce), + device_key_id: &device_key_id, + payload: ObjectPayload { + path: RESULT_PATH, + media_type: "application/json", + size_bytes: u64::try_from(result_json.len()).map_err(|_| ObjectPerformanceError::LimitExceeded)?, + sha256: &result_sha256, + }, + }; + let envelope_json = serde_json::to_vec(&envelope).map_err(|_| ObjectPerformanceError::Encoding)?; + if envelope_json.is_empty() || envelope_json.len() > MAX_ENVELOPE_BYTES { + return Err(ObjectPerformanceError::LimitExceeded); + } + let envelope_signature = signature_document(key, &device_key_id, &envelope_json)?; + let decompressed = result_json + .len() + .checked_add(envelope_json.len()) + .and_then(|size| size.checked_add(envelope_signature.len())) + .ok_or(ObjectPerformanceError::LimitExceeded)?; + if decompressed > MAX_DECOMPRESSED_BYTES { + return Err(ObjectPerformanceError::LimitExceeded); + } + check_cancel(cancel)?; + if unix_now()? >= request.expires_at_unix { + return Err(ObjectPerformanceError::Expired); + } + let archive_bytes = archive(&envelope_json, &envelope_signature, &result_json)?; + if archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(ObjectPerformanceError::LimitExceeded); + } + let archive_sha256 = hex_lower(&Sha256::digest(&archive_bytes)); + Ok(SignedObjectExport { + artifact_uid: request.artifact_uid.clone(), + outcome: result.outcome, + reason_code: result.reason_code, + envelope_json, + envelope_signature, + result_json, + archive_bytes, + archive_sha256, + }) +} + +pub fn save_signed_object_export( + output: &Path, + export: &SignedObjectExport, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + if !uuid7(&export.artifact_uid) || export.archive_bytes.is_empty() { + return Err(ObjectPerformanceError::InvalidRequest); + } + if export.archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(ObjectPerformanceError::LimitExceeded); + } + if hex_lower(&Sha256::digest(&export.archive_bytes)) != export.archive_sha256 { + return Err(ObjectPerformanceError::InvalidRequest); + } + let parent = output + .parent() + .filter(|path| !path.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let filename = output + .file_name() + .ok_or(ObjectPerformanceError::InvalidRequest)? + .to_string_lossy(); + let temporary = parent.join(format!(".{filename}.{}.partial", export.artifact_uid)); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + let mut file = options.open(&temporary).map_err(map_create_error)?; + let saved = (|| { + file.write_all(&export.archive_bytes).map_err(ObjectPerformanceError::Io)?; + check_cancel(cancel)?; + file.sync_all().map_err(ObjectPerformanceError::Io)?; + check_cancel(cancel)?; + fs::hard_link(&temporary, output).map_err(map_publish_error)?; + fs::remove_file(&temporary).map_err(ObjectPerformanceError::DurabilityAfterCommit)?; + #[cfg(unix)] + File::open(parent) + .and_then(|directory| directory.sync_all()) + .map_err(ObjectPerformanceError::DurabilityAfterCommit)?; + Ok(SavedObjectExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: u64::try_from(export.archive_bytes.len()).map_err(|_| ObjectPerformanceError::LimitExceeded)?, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if saved.is_err() { + let _ = fs::remove_file(&temporary); + } + saved +} + +pub fn validate_object_limits( + duration: Duration, + _operation: ObjectOperation, + traffic_bytes: u64, +) -> Result<(), ObjectPerformanceError> { + if duration.is_zero() + || duration.as_millis() == 0 + || duration > MAX_OBJECT_DURATION + || traffic_bytes == 0 + || traffic_bytes > MAX_OBJECT_TRAFFIC_BYTES + { + return Err(ObjectPerformanceError::LimitExceeded); + } + let bandwidth_budget = u64::try_from( + u128::from(MAX_OBJECT_BANDWIDTH_BYTES_PER_SECOND) + .checked_mul(duration.as_millis()) + .ok_or(ObjectPerformanceError::LimitExceeded)? + / 1_000, + ) + .map_err(|_| ObjectPerformanceError::LimitExceeded)?; + if traffic_bytes > bandwidth_budget.max(1) { + return Err(ObjectPerformanceError::LimitExceeded); + } + Ok(()) +} + +pub fn read_protected_object_credential(path: &Path) -> Result, ObjectPerformanceError> { + let metadata = fs::symlink_metadata(path).map_err(ObjectPerformanceError::Io)?; + if metadata.file_type().is_symlink() || !metadata.is_file() { + return Err(ObjectPerformanceError::InvalidCredential); + } + #[cfg(unix)] + { + use std::os::unix::fs::{MetadataExt as _, OpenOptionsExt as _, PermissionsExt as _}; + if metadata.uid() != process_uid() || metadata.permissions().mode() & 0o077 != 0 { + return Err(ObjectPerformanceError::InvalidCredential); + } + let mut options = OpenOptions::new(); + options.read(true).custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW); + let mut file = options.open(path).map_err(ObjectPerformanceError::Io)?; + let opened = file.metadata().map_err(ObjectPerformanceError::Io)?; + if opened.dev() != metadata.dev() || opened.ino() != metadata.ino() { + return Err(ObjectPerformanceError::InvalidCredential); + } + read_credential_value(&mut file) + } + #[cfg(not(unix))] + { + let mut file = File::open(path).map_err(ObjectPerformanceError::Io)?; + read_credential_value(&mut file) + } +} + +fn read_credential_value(reader: &mut impl Read) -> Result, ObjectPerformanceError> { + let mut bytes = Vec::with_capacity(256); + reader + .take(4_097) + .read_to_end(&mut bytes) + .map_err(ObjectPerformanceError::Io)?; + if bytes.is_empty() || bytes.len() > 4_096 || bytes.contains(&0) { + return Err(ObjectPerformanceError::InvalidCredential); + } + while matches!(bytes.last(), Some(b'\n' | b'\r')) { + bytes.pop(); + } + let value = String::from_utf8(bytes).map_err(|_| ObjectPerformanceError::InvalidCredential)?; + if value.is_empty() || value.len() > 4_096 || value.trim() != value { + return Err(ObjectPerformanceError::InvalidCredential); + } + Ok(Zeroizing::new(value)) +} + +#[cfg(unix)] +#[allow(unsafe_code)] +fn process_uid() -> u32 { + // SAFETY: geteuid has no pointer arguments or caller preconditions. + unsafe { libc::geteuid() } +} + +impl ObjectPerformanceRequest { + fn validate(&self, now_unix: i64) -> Result<(), ObjectPerformanceError> { + if self.schema_version != OBJECT_SCHEMA_VERSION { + return Err(ObjectPerformanceError::UnsupportedVersion); + } + if self.capability != OBJECT_CAPABILITY { + return Err(ObjectPerformanceError::UnsupportedCapability); + } + if !self.consent.confirmed || self.consent.policy_revision == 0 { + return Err(ObjectPerformanceError::ConsentRequired); + } + if self.consent.expires_at_unix <= now_unix || self.expires_at_unix > self.consent.expires_at_unix { + return Err(ObjectPerformanceError::ConsentExpired); + } + let validity = self + .expires_at_unix + .checked_sub(self.produced_at_unix) + .ok_or(ObjectPerformanceError::Expired)?; + if self.produced_at_unix > now_unix.saturating_add(MAX_FUTURE_SKEW_SECONDS) + || validity <= 0 + || validity > MAX_VALIDITY_SECONDS + || self.expires_at_unix <= now_unix + { + return Err(ObjectPerformanceError::Expired); + } + validate_object_limits(self.duration, self.operation, self.traffic_bytes)?; + if !uuid7(&self.run_uid) + || !uuid7(&self.artifact_uid) + || !uuid7(&self.consent.consent_uid) + || !resource_names_match(self) + || self.target_alias.is_empty() + || self.target_alias.len() > 128 + || !self + .target_alias + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || b"._:-".contains(&byte)) + || !lower_hex_string(&self.provenance.source_commit, 40) + || !lower_hex_string(&self.provenance.executable_sha256, 64) + || !version(&self.provenance.rustfs_version) + || self.provenance.build_features.len() > MAX_BUILD_FEATURES + || !self.provenance.build_features.iter().all(|value| build_feature(value)) + { + return Err(ObjectPerformanceError::InvalidRequest); + } + Ok(()) + } +} + +struct ActiveGuard; + +impl Drop for ActiveGuard { + fn drop(&mut self) { + OBJECT_COLLECTOR_ACTIVE.store(false, Ordering::Release); + } +} + +fn success_measurement(request: &ObjectPerformanceRequest, sample: ObjectProbeMeasurement) -> ObjectMeasurement { + let duration_millis = elapsed_millis(sample.duration); + let data = ObjectPerformanceData { + operation: request.operation, + transferred_bytes: sample.transferred_bytes, + completed_operations: 1, + duration_millis, + error_count: 0, + }; + ObjectMeasurement { + result: result( + request, + ObjectOutcome::Succeeded, + ObjectReasonCode::Complete, + duration_millis, + 1, + Some(data), + ), + target: ObjectTargetResult { + target_alias: request.target_alias.clone(), + outcome: ObjectOutcome::Succeeded, + reason_code: ObjectTargetReasonCode::Complete, + parameters: target_parameters(request), + units: target_units(), + transferred_bytes: sample.transferred_bytes, + completed_operations: 1, + latency_micros: Some(elapsed_micros(sample.latency)), + duration_millis, + }, + } +} + +fn failed_measurement( + request: &ObjectPerformanceRequest, + elapsed: Duration, + target_reason: ObjectTargetReasonCode, +) -> ObjectMeasurement { + let reason = match target_reason { + ObjectTargetReasonCode::EndpointUnavailable => ObjectReasonCode::SourceUnavailable, + ObjectTargetReasonCode::PermissionDenied => ObjectReasonCode::PermissionDenied, + _ => ObjectReasonCode::CollectionFailed, + }; + terminal_measurement(request, ObjectOutcome::Failed, reason, target_reason, elapsed) +} + +fn terminal_measurement( + request: &ObjectPerformanceRequest, + outcome: ObjectOutcome, + reason_code: ObjectReasonCode, + target_reason: ObjectTargetReasonCode, + elapsed: Duration, +) -> ObjectMeasurement { + let duration_millis = elapsed_millis_allow_zero(elapsed); + ObjectMeasurement { + result: result(request, outcome, reason_code, duration_millis, 0, None), + target: ObjectTargetResult { + target_alias: request.target_alias.clone(), + outcome, + reason_code: target_reason, + parameters: target_parameters(request), + units: target_units(), + transferred_bytes: 0, + completed_operations: 0, + latency_micros: None, + duration_millis, + }, + } +} + +fn target_reason(error: ObjectProbeError) -> ObjectTargetReasonCode { + match error { + ObjectProbeError::EndpointUnavailable => ObjectTargetReasonCode::EndpointUnavailable, + ObjectProbeError::ProxyFailure => ObjectTargetReasonCode::ProxyFailure, + ObjectProbeError::PermissionDenied => ObjectTargetReasonCode::PermissionDenied, + ObjectProbeError::TimedOut => ObjectTargetReasonCode::TimedOut, + ObjectProbeError::Cancelled => ObjectTargetReasonCode::Cancelled, + ObjectProbeError::NamespaceConflict => ObjectTargetReasonCode::NamespaceConflict, + ObjectProbeError::CleanupFailed => ObjectTargetReasonCode::CleanupFailed, + ObjectProbeError::ProtocolFailure => ObjectTargetReasonCode::ProtocolFailure, + } +} + +fn target_parameters(request: &ObjectPerformanceRequest) -> ObjectTargetParameters { + ObjectTargetParameters { + operation: request.operation, + requested_bytes: request.traffic_bytes, + duration_millis: elapsed_millis_allow_zero(request.duration), + concurrency: 1, + } +} + +fn target_units() -> ObjectTargetUnits { + ObjectTargetUnits { + bytes: "BYTE", + duration: "MILLISECOND", + latency: "MICROSECOND", + operation_count: "OPERATION", + } +} + +fn result( + request: &ObjectPerformanceRequest, + outcome: ObjectOutcome, + reason_code: ObjectReasonCode, + duration_millis: u64, + completed_units: u32, + data: Option, +) -> ObjectDiagnosticResult { + ObjectDiagnosticResult { + schema_version: OBJECT_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: OBJECT_TOOL_ID, + capability: OBJECT_CAPABILITY, + outcome, + reason_code, + duration_millis: duration_millis.min(30_000), + provenance: request.provenance.clone(), + coverage: ObjectCoverage { + requested_units: 1, + completed_units, + unit: "WINDOW", + }, + data, + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ObjectEnvelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: &'static str, + schema_version: u16, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: &'a str, + payload: ObjectPayload<'a>, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ObjectPayload<'a> { + path: &'static str, + media_type: &'static str, + size_bytes: u64, + sha256: &'a str, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ObjectSignature<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +fn signature_document(key: &DeviceIdentity, key_id: &str, envelope: &[u8]) -> Result, ObjectPerformanceError> { + let pkcs8 = key.to_pkcs8_der().map_err(|_| ObjectPerformanceError::Signing)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| ObjectPerformanceError::Signing)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + envelope.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.extend_from_slice(envelope); + let signature: Signature = signing_key.sign(&input); + serde_json::to_vec(&ObjectSignature { + algorithm: "ES256", + key_id, + value: URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()), + }) + .map_err(|_| ObjectPerformanceError::Encoding) +} + +fn archive(envelope: &[u8], signature: &[u8], result: &[u8]) -> Result, ObjectPerformanceError> { + let cursor = Cursor::new(Vec::with_capacity(envelope.len() + signature.len() + result.len() + 512)); + let mut writer = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(OUTPUT_MODE); + for (name, bytes) in [(ENVELOPE_PATH, envelope), (SIGNATURE_PATH, signature), (RESULT_PATH, result)] { + writer + .start_file(name, options) + .map_err(|_| ObjectPerformanceError::Encoding)?; + writer.write_all(bytes).map_err(ObjectPerformanceError::Io)?; + } + writer + .finish() + .map(|cursor| cursor.into_inner()) + .map_err(|_| ObjectPerformanceError::Encoding) +} + +fn deployment_endpoint(value: &str) -> Result { + let mut url = Url::parse(value).map_err(|_| ObjectPerformanceError::InvalidEndpoint)?; + let local_http = url.scheme() == "http" + && url + .host_str() + .is_some_and(|host| host == "localhost" || host.parse::().is_ok_and(|ip| ip.is_loopback())); + if (url.scheme() != "https" && !local_http) + || url.cannot_be_a_base() + || !url.username().is_empty() + || url.password().is_some() + || url.path() != "/" + || url.query().is_some() + || url.fragment().is_some() + { + return Err(ObjectPerformanceError::InvalidEndpoint); + } + url.set_query(None); + url.set_fragment(None); + if !url.path().ends_with('/') { + url.set_path(&format!("{}/", url.path())); + } + Ok(url) +} + +fn proxy_url(value: &str) -> Result { + let url = Url::parse(value).map_err(|_| ObjectPerformanceError::InvalidProxy)?; + if !matches!(url.scheme(), "http" | "https") + || url.cannot_be_a_base() + || !url.username().is_empty() + || url.password().is_some() + || url.query().is_some() + || url.fragment().is_some() + { + return Err(ObjectPerformanceError::InvalidProxy); + } + Ok(url) +} + +fn resource_names_match(request: &ObjectPerformanceRequest) -> bool { + let Some(organization_uid) = request.organization_name.strip_prefix("organizations/") else { + return false; + }; + if !uuid7(organization_uid) { + return false; + } + let cluster_prefix = format!("{}/clusters/", request.organization_name); + let Some(cluster_uid) = request.cluster_name.strip_prefix(&cluster_prefix) else { + return false; + }; + if !uuid7(cluster_uid) { + return false; + } + let device_prefix = format!("{}/clusterDevices/", request.cluster_name); + request.device_name.strip_prefix(&device_prefix).is_some_and(uuid7) +} + +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn lower_hex_string(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_hexdigit() && !byte.is_ascii_uppercase()) +} + +fn version(value: &str) -> bool { + if value.is_empty() || value.len() > 64 { + return false; + } + let (core, prerelease) = value + .split_once('-') + .map_or((value, None), |(core, prerelease)| (core, Some(prerelease))); + let mut parts = core.split('.'); + let valid_core = (0..3).all(|_| { + parts + .next() + .is_some_and(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) + }) && parts.next().is_none(); + valid_core + && prerelease.is_none_or(|part| { + !part.is_empty() + && part + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'-')) + }) +} + +fn build_feature(value: &str) -> bool { + value.len() <= 64 + && value.as_bytes().split_first().is_some_and(|(first, rest)| { + first.is_ascii_lowercase() + && rest + .iter() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'-' | b'_')) + }) +} + +fn elapsed_millis(duration: Duration) -> u64 { + elapsed_millis_allow_zero(duration).max(1) +} + +fn elapsed_millis_allow_zero(duration: Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).min(MAX_SAFE_INTEGER) +} + +fn elapsed_micros(duration: Duration) -> u64 { + u64::try_from(duration.as_micros()).unwrap_or(u64::MAX).min(MAX_SAFE_INTEGER) +} + +fn unix_now() -> Result { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| ObjectPerformanceError::InvalidRequest) + .and_then(|duration| i64::try_from(duration.as_secs()).map_err(|_| ObjectPerformanceError::InvalidRequest)) +} + +fn timestamp(unix: i64) -> Result { + OffsetDateTime::from_unix_timestamp(unix) + .map_err(|_| ObjectPerformanceError::InvalidRequest)? + .format(&Rfc3339) + .map_err(|_| ObjectPerformanceError::Encoding) +} + +fn check_cancel(cancel: &CancellationToken) -> Result<(), ObjectPerformanceError> { + if cancel.is_cancelled() { + Err(ObjectPerformanceError::Cancelled) + } else { + Ok(()) + } +} + +fn hex_lower(bytes: &[u8]) -> String { + let mut encoded = String::with_capacity(bytes.len() * 2); + for byte in bytes { + use std::fmt::Write as _; + let _ = write!(encoded, "{byte:02x}"); + } + encoded +} + +fn map_create_error(error: std::io::Error) -> ObjectPerformanceError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + ObjectPerformanceError::AlreadyExists + } else { + ObjectPerformanceError::Io(error) + } +} + +fn map_publish_error(error: std::io::Error) -> ObjectPerformanceError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + ObjectPerformanceError::AlreadyExists + } else { + ObjectPerformanceError::Io(error) + } +} diff --git a/rustfs/src/connect/diagnostics/perf_site_replication.rs b/rustfs/src/connect/diagnostics/perf_site_replication.rs new file mode 100644 index 000000000..66fd64d58 --- /dev/null +++ b/rustfs/src/connect/diagnostics/perf_site_replication.rs @@ -0,0 +1,1672 @@ +// Copyright 2026 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Consent-bound, destination-confirmed site-replication measurement. +//! +//! The producer uses a pre-provisioned versioned scratch bucket. It never +//! creates or removes buckets. A sample succeeds only after the destination +//! returns the exact source version and generated payload. Cleanup lists the +//! unique task key on both sites and deletes every version, including versions +//! that arrive after cancellation or timeout during a bounded grace window. + +use std::fs::{self, File, OpenOptions}; +use std::future::Future; +use std::io::{Cursor, Read, Write as _}; +use std::path::Path; +use std::pin::Pin; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use base64_simd::URL_SAFE_NO_PAD; +use bytes::Bytes; +use futures::StreamExt as _; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use percent_encoding::{NON_ALPHANUMERIC, utf8_percent_encode}; +use reqwest::{Client, Method, Response, StatusCode, Url}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zeroize::Zeroizing; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use crate::connect::DeviceIdentity; + +pub const SITE_REPLICATION_SCHEMA_VERSION: u16 = 1; +pub const SITE_REPLICATION_TOOL_ID: &str = "performance.siteReplication"; +pub const SITE_REPLICATION_CAPABILITY: &str = "performance.siteReplication@1"; +pub const MAX_SITE_REPLICATION_DURATION: Duration = Duration::from_secs(30); +pub const MAX_SITE_REPLICATION_TRAFFIC_BYTES: u64 = 1_048_576; +const MAX_LATE_ARRIVAL_CLEANUP: Duration = Duration::from_secs(5); +const MAX_RESPONSE_BYTES: u64 = 1_064_960; +const MAX_BUILD_FEATURES: usize = 64; +const MAX_SAFE_INTEGER: u64 = 9_007_199_254_740_991; +const MAX_FUTURE_SKEW_SECONDS: i64 = 300; +const MAX_VALIDITY_SECONDS: i64 = 2_592_000; +const MAX_RESULT_BYTES: usize = 262_144; +const MAX_ENVELOPE_BYTES: usize = 16_384; +const MAX_ARCHIVE_BYTES: usize = 524_288; +const MAX_DECOMPRESSED_BYTES: usize = 278_528; +const OUTPUT_MODE: u32 = 0o600; +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1\0"; +const ENVELOPE_PATH: &str = "envelope.json"; +const SIGNATURE_PATH: &str = "envelope.sig"; +const RESULT_PATH: &str = "result.json"; + +static SITE_REPLICATION_COLLECTOR_ACTIVE: AtomicBool = AtomicBool::new(false); + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum SiteReplicationOutcome { + Succeeded, + Failed, + Cancelled, + Unsupported, +} + +impl SiteReplicationOutcome { + pub const fn as_str(self) -> &'static str { + match self { + Self::Succeeded => "SUCCEEDED", + Self::Failed => "FAILED", + Self::Cancelled => "CANCELLED", + Self::Unsupported => "UNSUPPORTED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum SiteReplicationReasonCode { + Complete, + CollectionFailed, + PermissionDenied, + Cancelled, + UnsupportedTool, +} + +impl SiteReplicationReasonCode { + pub const fn as_str(self) -> &'static str { + match self { + Self::Complete => "COMPLETE", + Self::CollectionFailed => "COLLECTION_FAILED", + Self::PermissionDenied => "PERMISSION_DENIED", + Self::Cancelled => "CANCELLED", + Self::UnsupportedTool => "UNSUPPORTED_TOOL", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum SiteReplicationTargetReasonCode { + Complete, + SiteReplicationUnavailable, + EndpointUnavailable, + PermissionDenied, + TimedOut, + Cancelled, + VersioningRequired, + DestinationUnconfirmed, + CleanupFailed, + ProtocolFailure, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct SiteReplicationProvenance { + repository: &'static str, + source_commit: String, + executable_sha256: String, + rustfs_version: String, + os_family: OsFamily, + architecture: Architecture, + build_features: Vec, +} + +impl SiteReplicationProvenance { + pub fn new( + source_commit: impl Into, + executable_sha256: impl Into, + rustfs_version: impl Into, + build_features: Vec, + ) -> Self { + Self { + repository: "rustfs/rustfs", + source_commit: source_commit.into(), + executable_sha256: executable_sha256.into(), + rustfs_version: rustfs_version.into(), + os_family: OsFamily::current(), + architecture: Architecture::current(), + build_features, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum OsFamily { + Linux, + Darwin, + Windows, + Freebsd, + Other, +} + +impl OsFamily { + fn current() -> Self { + match std::env::consts::OS { + "linux" => Self::Linux, + "macos" => Self::Darwin, + "windows" => Self::Windows, + "freebsd" => Self::Freebsd, + _ => Self::Other, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +enum Architecture { + #[serde(rename = "x86_64")] + X86_64, + Aarch64, + Other, +} + +impl Architecture { + fn current() -> Self { + match std::env::consts::ARCH { + "x86_64" => Self::X86_64, + "aarch64" => Self::Aarch64, + _ => Self::Other, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LocalSiteReplicationConsent { + pub consent_uid: String, + pub policy_revision: u64, + pub expires_at_unix: i64, + pub nonce: [u8; 32], + pub confirmed: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SiteReplicationPerformanceRequest { + pub organization_name: String, + pub cluster_name: String, + pub destination_cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub schema_version: u16, + pub capability: String, + pub consent: LocalSiteReplicationConsent, + pub produced_at_unix: i64, + pub expires_at_unix: i64, + pub duration: Duration, + pub traffic_bytes: u64, + pub source_alias: String, + pub source_deployment_id: String, + pub destination_alias: String, + pub destination_deployment_id: String, + pub scratch_bucket: String, + pub late_arrival_cleanup: Duration, + pub provenance: SiteReplicationProvenance, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct SiteReplicationPerformanceData { + pub replicated_bytes: u64, + pub confirmed_objects: u64, + pub duration_millis: u64, + pub max_observed_lag_millis: u64, + pub error_count: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +struct Coverage { + requested_units: u32, + completed_units: u32, + unit: &'static str, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct SiteReplicationDiagnosticResult { + schema_version: u16, + run_uid: String, + tool_id: &'static str, + capability: &'static str, + outcome: SiteReplicationOutcome, + reason_code: SiteReplicationReasonCode, + duration_millis: u64, + provenance: SiteReplicationProvenance, + coverage: Coverage, + data: Option, +} + +impl SiteReplicationDiagnosticResult { + pub fn outcome(&self) -> SiteReplicationOutcome { + self.outcome + } + + pub fn reason_code(&self) -> SiteReplicationReasonCode { + self.reason_code + } + + pub fn data(&self) -> Option<&SiteReplicationPerformanceData> { + self.data.as_ref() + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct SiteReplicationTargetResult { + pub source_alias: String, + pub destination_alias: String, + pub outcome: SiteReplicationOutcome, + pub reason_code: SiteReplicationTargetReasonCode, + pub requested_bytes: u64, + pub duration_millis: u64, + pub concurrency: u8, + pub bytes_unit: &'static str, + pub duration_unit: &'static str, + pub operation_count_unit: &'static str, + pub replicated_bytes: u64, + pub confirmed_objects: u64, + pub max_observed_lag_millis: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SiteReplicationMeasurement { + pub result: SiteReplicationDiagnosticResult, + pub target: SiteReplicationTargetResult, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct SiteReplicationProbeMeasurement { + pub replicated_bytes: u64, + pub confirmed_objects: u64, + pub duration: Duration, + pub max_observed_lag: Duration, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum SiteReplicationProbeError { + SiteReplicationUnavailable, + EndpointUnavailable, + PermissionDenied, + TimedOut, + Cancelled, + VersioningRequired, + DestinationUnconfirmed, + CleanupFailed, + ProtocolFailure, +} + +pub type SiteReplicationProbeFuture<'a> = + Pin> + Send + 'a>>; + +pub trait SiteReplicationProbe: Send + Sync { + fn probe<'a>( + &'a self, + request: &'a SiteReplicationPerformanceRequest, + cancel: &'a CancellationToken, + ) -> SiteReplicationProbeFuture<'a>; +} + +/// Static credential set for one site-replication endpoint connection. +pub struct SiteReplicationCredentials { + pub access_key: Zeroizing, + pub secret_key: Zeroizing, + pub session_token: Zeroizing, +} + +pub struct SiteReplicationEndpoint { + pub alias: String, + pub deployment_id: String, + endpoint: Url, + client: Client, + access_key: Zeroizing, + secret_key: Zeroizing, + session_token: Zeroizing, +} + +impl SiteReplicationEndpoint { + pub fn new( + alias: impl Into, + deployment_id: impl Into, + endpoint: &str, + root_ca_pem: Option<&[u8]>, + credentials: SiteReplicationCredentials, + timeout: Duration, + ) -> Result { + let endpoint = deployment_endpoint(endpoint)?; + if credentials.access_key.is_empty() || credentials.secret_key.is_empty() { + return Err(SiteReplicationPerformanceError::InvalidCredential); + } + let mut builder = Client::builder() + .no_proxy() + .redirect(reqwest::redirect::Policy::none()) + .timeout(timeout); + if let Some(root_ca_pem) = root_ca_pem { + let certificate = reqwest::Certificate::from_pem(root_ca_pem) + .map_err(|_| SiteReplicationPerformanceError::InvalidRootCertificate)?; + builder = builder.add_root_certificate(certificate); + } + let client = builder + .build() + .map_err(|_| SiteReplicationPerformanceError::TransportConfiguration)?; + Ok(Self { + alias: alias.into(), + deployment_id: deployment_id.into(), + endpoint, + client, + access_key: credentials.access_key, + secret_key: credentials.secret_key, + session_token: credentials.session_token, + }) + } +} + +pub struct S3SiteReplicationProbe { + source: SiteReplicationEndpoint, + destination: SiteReplicationEndpoint, +} + +impl S3SiteReplicationProbe { + pub fn new(source: SiteReplicationEndpoint, destination: SiteReplicationEndpoint) -> Self { + Self { source, destination } + } + + async fn execute( + &self, + request: &SiteReplicationPerformanceRequest, + cancel: &CancellationToken, + ) -> Result { + self.validate_endpoint_bindings(request)?; + let started = Instant::now(); + let cleanup_window = request.late_arrival_cleanup.min(MAX_LATE_ARRIVAL_CLEANUP); + let operation_deadline = started + request.duration.saturating_sub(cleanup_window); + self.confirm_topology(request, operation_deadline, cancel).await?; + let key = format!("rustfs-connect/site-replication/{}", request.artifact_uid); + let payload_len = usize::try_from(request.traffic_bytes).map_err(|_| SiteReplicationProbeError::ProtocolFailure)?; + let payload = Bytes::from(vec![0xa5; payload_len]); + + let measured = self.measure(request, &key, payload, operation_deadline, cancel).await; + let cleanup_deadline = (started + request.duration).min(Instant::now() + cleanup_window); + let cleanup = self.cleanup_late_arrivals(request, &key, cleanup_deadline).await; + if cleanup.is_err() { + return Err(SiteReplicationProbeError::CleanupFailed); + } + measured + } + + fn validate_endpoint_bindings(&self, request: &SiteReplicationPerformanceRequest) -> Result<(), SiteReplicationProbeError> { + if self.source.alias != request.source_alias + || self.source.deployment_id != request.source_deployment_id + || self.destination.alias != request.destination_alias + || self.destination.deployment_id != request.destination_deployment_id + { + return Err(SiteReplicationProbeError::ProtocolFailure); + } + Ok(()) + } + + async fn measure( + &self, + request: &SiteReplicationPerformanceRequest, + key: &str, + payload: Bytes, + deadline: Instant, + cancel: &CancellationToken, + ) -> Result { + let source_url = object_url(&self.source.endpoint, &request.scratch_bucket, key, None)?; + let replication_started = Instant::now(); + // Do not cancel an in-flight PUT: a cancelled request may still commit at + // the server. Waiting for its bounded response preserves the version ID; + // the fallback version listing in cleanup also covers a lost response. + let response = self + .source + .send(Method::PUT, source_url, payload.clone(), deadline, None) + .await?; + if !response.status().is_success() { + return status_error(response.status()); + } + let version_id = response + .headers() + .get("x-amz-version-id") + .and_then(|value| value.to_str().ok()) + .filter(|value| !value.is_empty() && *value != "null") + .ok_or(SiteReplicationProbeError::VersioningRequired)? + .to_owned(); + drain_response(response, 16_384, deadline, None).await?; + + loop { + if cancel.is_cancelled() { + return Err(SiteReplicationProbeError::Cancelled); + } + if Instant::now() >= deadline { + return Err(SiteReplicationProbeError::DestinationUnconfirmed); + } + let destination_url = object_url(&self.destination.endpoint, &request.scratch_bucket, key, Some(&version_id))?; + let response = self + .destination + .send(Method::GET, destination_url, Bytes::new(), deadline, Some(cancel)) + .await?; + if response.status() == StatusCode::NOT_FOUND { + tokio::time::sleep(Duration::from_millis(50)).await; + continue; + } + if !response.status().is_success() { + return status_error(response.status()); + } + let returned_version = response + .headers() + .get("x-amz-version-id") + .and_then(|value| value.to_str().ok()); + if returned_version != Some(version_id.as_str()) { + return Err(SiteReplicationProbeError::DestinationUnconfirmed); + } + let body = drain_response(response, request.traffic_bytes, deadline, Some(cancel)).await?; + if body.as_slice() != payload.as_ref() { + return Err(SiteReplicationProbeError::DestinationUnconfirmed); + } + let elapsed = replication_started.elapsed(); + return Ok(SiteReplicationProbeMeasurement { + replicated_bytes: request.traffic_bytes, + confirmed_objects: 1, + duration: elapsed, + max_observed_lag: elapsed, + }); + } + } + + async fn confirm_topology( + &self, + request: &SiteReplicationPerformanceRequest, + deadline: Instant, + cancel: &CancellationToken, + ) -> Result<(), SiteReplicationProbeError> { + let url = self + .source + .endpoint + .join("rustfs/admin/v3/site-replication/info") + .map_err(|_| SiteReplicationProbeError::ProtocolFailure)?; + let response = self + .source + .send(Method::GET, url, Bytes::new(), deadline, Some(cancel)) + .await?; + if !response.status().is_success() { + return status_error(response.status()); + } + let body = drain_response(response, 262_144, deadline, Some(cancel)).await?; + let info: SiteReplicationInfo = serde_json::from_slice(&body).map_err(|_| SiteReplicationProbeError::ProtocolFailure)?; + let expected = [&request.source_deployment_id, &request.destination_deployment_id]; + if !info.enabled + || !expected + .iter() + .all(|id| info.sites.iter().any(|site| &site.deployment_id == *id)) + { + return Err(SiteReplicationProbeError::SiteReplicationUnavailable); + } + Ok(()) + } + + async fn cleanup_late_arrivals( + &self, + request: &SiteReplicationPerformanceRequest, + key: &str, + deadline: Instant, + ) -> Result<(), SiteReplicationProbeError> { + let mut clean = false; + loop { + if Instant::now() >= deadline { + return clean.then_some(()).ok_or(SiteReplicationProbeError::CleanupFailed); + } + self.delete_key_versions(&self.source, &request.scratch_bucket, key, deadline) + .await?; + self.delete_key_versions(&self.destination, &request.scratch_bucket, key, deadline) + .await?; + clean = true; + tokio::time::sleep(Duration::from_millis(100)).await; + } + } + + async fn delete_key_versions( + &self, + endpoint: &SiteReplicationEndpoint, + bucket: &str, + key: &str, + deadline: Instant, + ) -> Result { + let list_url = list_versions_url(&endpoint.endpoint, bucket, key)?; + let response = endpoint.send(Method::GET, list_url, Bytes::new(), deadline, None).await?; + if !response.status().is_success() { + return status_error(response.status()); + } + let body = drain_response(response, MAX_RESPONSE_BYTES, deadline, None).await?; + let listed: ListVersionsResult = + quick_xml::de::from_reader(body.as_slice()).map_err(|_| SiteReplicationProbeError::ProtocolFailure)?; + if listed.is_truncated { + return Err(SiteReplicationProbeError::CleanupFailed); + } + let versions = listed + .versions + .into_iter() + .chain(listed.delete_markers) + .filter(|version| version.key == key) + .collect::>(); + for version in &versions { + let url = object_url(&endpoint.endpoint, bucket, key, Some(&version.version_id))?; + let response = endpoint.send(Method::DELETE, url, Bytes::new(), deadline, None).await?; + if !response.status().is_success() && response.status() != StatusCode::NOT_FOUND { + return Err(SiteReplicationProbeError::CleanupFailed); + } + drain_response(response, 16_384, deadline, None).await?; + } + Ok(versions.len()) + } +} + +impl SiteReplicationEndpoint { + async fn send( + &self, + method: Method, + url: Url, + payload: Bytes, + deadline: Instant, + cancel: Option<&CancellationToken>, + ) -> Result { + let payload_hash = hex_lower(&Sha256::digest(&payload)); + let unsigned = http::Request::builder() + .method(method.clone()) + .uri(url.as_str()) + .header("x-amz-content-sha256", payload_hash) + .body(()) + .map_err(|_| SiteReplicationProbeError::ProtocolFailure)?; + let headers = rustfs_signer::try_sign_v4_headers( + unsigned.into_parts().0, + i64::try_from(payload.len()).map_err(|_| SiteReplicationProbeError::ProtocolFailure)?, + &self.access_key, + &self.secret_key, + &self.session_token, + "us-east-1", + ) + .map_err(|_| SiteReplicationProbeError::ProtocolFailure)?; + let sent = self.client.request(method, url).headers(headers).body(payload).send(); + let timed = tokio::time::timeout_at(tokio::time::Instant::from_std(deadline), sent); + let response = if let Some(cancel) = cancel { + tokio::select! { + () = cancel.cancelled() => return Err(SiteReplicationProbeError::Cancelled), + response = timed => response, + } + } else { + timed.await + }; + response.map_err(|_| SiteReplicationProbeError::TimedOut)?.map_err(|error| { + if error.is_timeout() { + SiteReplicationProbeError::TimedOut + } else if error.is_connect() { + SiteReplicationProbeError::EndpointUnavailable + } else { + SiteReplicationProbeError::ProtocolFailure + } + }) + } +} + +impl SiteReplicationProbe for S3SiteReplicationProbe { + fn probe<'a>( + &'a self, + request: &'a SiteReplicationPerformanceRequest, + cancel: &'a CancellationToken, + ) -> SiteReplicationProbeFuture<'a> { + Box::pin(self.execute(request, cancel)) + } +} + +#[derive(Debug, Error)] +pub enum SiteReplicationPerformanceError { + #[error("site_replication_performance_local_consent_required")] + ConsentRequired, + #[error("site_replication_performance_local_consent_expired")] + ConsentExpired, + #[error("site_replication_performance_request_expired")] + Expired, + #[error("site_replication_performance_invalid_request")] + InvalidRequest, + #[error("site_replication_performance_unsupported_version")] + UnsupportedVersion, + #[error("site_replication_performance_unsupported_capability")] + UnsupportedCapability, + #[error("site_replication_performance_limit_exceeded")] + LimitExceeded, + #[error("site_replication_performance_collection_cancelled")] + Cancelled, + #[error("site_replication_performance_busy")] + Busy, + #[error("site_replication_performance_invalid_endpoint")] + InvalidEndpoint, + #[error("site_replication_performance_invalid_root_certificate")] + InvalidRootCertificate, + #[error("site_replication_performance_invalid_credential")] + InvalidCredential, + #[error("site_replication_performance_transport_configuration")] + TransportConfiguration, + #[error("site_replication_performance_signing_failed")] + Signing, + #[error("site_replication_performance_encoding_failed")] + Encoding, + #[error("site_replication_performance_output_exists")] + AlreadyExists, + #[error("site_replication_performance_io_failed")] + Io(#[source] std::io::Error), + #[error("site_replication_performance_output_durability_failed")] + DurabilityAfterCommit(#[source] std::io::Error), +} + +pub async fn measure_site_replication( + request: &SiteReplicationPerformanceRequest, + probe: &impl SiteReplicationProbe, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + if SITE_REPLICATION_COLLECTOR_ACTIVE + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .is_err() + { + return Err(SiteReplicationPerformanceError::Busy); + } + let _guard = ActiveGuard; + if cancel.is_cancelled() { + return Ok(terminal_measurement( + request, + SiteReplicationOutcome::Cancelled, + SiteReplicationReasonCode::Cancelled, + SiteReplicationTargetReasonCode::Cancelled, + Duration::ZERO, + )); + } + let started = Instant::now(); + Ok(match probe.probe(request, cancel).await { + Ok(sample) + if sample.replicated_bytes == request.traffic_bytes + && sample.confirmed_objects == 1 + && !sample.duration.is_zero() => + { + success_measurement(request, sample) + } + Ok(_) => terminal_measurement( + request, + SiteReplicationOutcome::Failed, + SiteReplicationReasonCode::CollectionFailed, + SiteReplicationTargetReasonCode::DestinationUnconfirmed, + started.elapsed(), + ), + Err(SiteReplicationProbeError::Cancelled) => terminal_measurement( + request, + SiteReplicationOutcome::Cancelled, + SiteReplicationReasonCode::Cancelled, + SiteReplicationTargetReasonCode::Cancelled, + started.elapsed(), + ), + Err(error) => failed_measurement(request, started.elapsed(), error), + }) +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SignedSiteReplicationExport { + pub artifact_uid: String, + pub outcome: SiteReplicationOutcome, + pub reason_code: SiteReplicationReasonCode, + pub envelope_json: Vec, + pub envelope_signature: Vec, + pub result_json: Vec, + pub archive_bytes: Vec, + pub archive_sha256: String, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SavedSiteReplicationExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +pub fn sign_site_replication_export( + request: &SiteReplicationPerformanceRequest, + measurement: &SiteReplicationMeasurement, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + request.validate(unix_now()?)?; + check_cancel(cancel)?; + let result = &measurement.result; + let data_matches_outcome = (result.outcome == SiteReplicationOutcome::Succeeded) == result.data.is_some(); + if !data_matches_outcome + || result.run_uid != request.run_uid + || result.schema_version != SITE_REPLICATION_SCHEMA_VERSION + || result.tool_id != SITE_REPLICATION_TOOL_ID + || result.capability != SITE_REPLICATION_CAPABILITY + || measurement.target.source_alias != request.source_alias + || measurement.target.destination_alias != request.destination_alias + { + return Err(SiteReplicationPerformanceError::InvalidRequest); + } + let result_json = serde_json::to_vec(result).map_err(|_| SiteReplicationPerformanceError::Encoding)?; + if result_json.is_empty() || result_json.len() > MAX_RESULT_BYTES { + return Err(SiteReplicationPerformanceError::LimitExceeded); + } + let device_key_id = hex_lower(&Sha256::digest(key.public_key_der())); + let result_sha256 = hex_lower(&Sha256::digest(&result_json)); + let envelope = Envelope { + format_version: "rustfs.connect.diagnosticEnvelope/1", + protocol_version: "v1", + organization_name: &request.organization_name, + cluster_name: &request.cluster_name, + device_name: &request.device_name, + run_uid: &request.run_uid, + artifact_uid: &request.artifact_uid, + tool_id: SITE_REPLICATION_TOOL_ID, + schema_version: SITE_REPLICATION_SCHEMA_VERSION, + classification: "L2", + consent_uid: &request.consent.consent_uid, + policy_revision: request.consent.policy_revision, + produced_at: timestamp(request.produced_at_unix)?, + expires_at: timestamp(request.expires_at_unix)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(request.consent.nonce), + device_key_id: &device_key_id, + targets: Targets { + source_deployment: &request.cluster_name, + destination_deployment: &request.destination_cluster_name, + }, + payload: Payload { + path: RESULT_PATH, + media_type: "application/json", + size_bytes: u64::try_from(result_json.len()).map_err(|_| SiteReplicationPerformanceError::LimitExceeded)?, + sha256: &result_sha256, + }, + }; + let envelope_json = serde_json::to_vec(&envelope).map_err(|_| SiteReplicationPerformanceError::Encoding)?; + if envelope_json.is_empty() || envelope_json.len() > MAX_ENVELOPE_BYTES { + return Err(SiteReplicationPerformanceError::LimitExceeded); + } + let envelope_signature = signature_document(key, &device_key_id, &envelope_json)?; + let decompressed = result_json + .len() + .checked_add(envelope_json.len()) + .and_then(|size| size.checked_add(envelope_signature.len())) + .ok_or(SiteReplicationPerformanceError::LimitExceeded)?; + if decompressed > MAX_DECOMPRESSED_BYTES { + return Err(SiteReplicationPerformanceError::LimitExceeded); + } + check_cancel(cancel)?; + if unix_now()? >= request.expires_at_unix { + return Err(SiteReplicationPerformanceError::Expired); + } + let archive_bytes = archive(&envelope_json, &envelope_signature, &result_json)?; + if archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(SiteReplicationPerformanceError::LimitExceeded); + } + let archive_sha256 = hex_lower(&Sha256::digest(&archive_bytes)); + Ok(SignedSiteReplicationExport { + artifact_uid: request.artifact_uid.clone(), + outcome: result.outcome, + reason_code: result.reason_code, + envelope_json, + envelope_signature, + result_json, + archive_bytes, + archive_sha256, + }) +} + +pub fn save_signed_site_replication_export( + output: &Path, + export: &SignedSiteReplicationExport, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + if !uuid7(&export.artifact_uid) + || export.archive_bytes.is_empty() + || export.archive_bytes.len() > MAX_ARCHIVE_BYTES + || hex_lower(&Sha256::digest(&export.archive_bytes)) != export.archive_sha256 + { + return Err(SiteReplicationPerformanceError::InvalidRequest); + } + let parent = output + .parent() + .filter(|path| !path.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let filename = output + .file_name() + .ok_or(SiteReplicationPerformanceError::InvalidRequest)? + .to_string_lossy(); + let temporary = parent.join(format!(".{filename}.{}.partial", export.artifact_uid)); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + let mut file = options.open(&temporary).map_err(map_create_error)?; + let saved = (|| { + file.write_all(&export.archive_bytes) + .map_err(SiteReplicationPerformanceError::Io)?; + check_cancel(cancel)?; + file.sync_all().map_err(SiteReplicationPerformanceError::Io)?; + check_cancel(cancel)?; + fs::hard_link(&temporary, output).map_err(map_publish_error)?; + fs::remove_file(&temporary).map_err(SiteReplicationPerformanceError::DurabilityAfterCommit)?; + #[cfg(unix)] + File::open(parent) + .and_then(|directory| directory.sync_all()) + .map_err(SiteReplicationPerformanceError::DurabilityAfterCommit)?; + Ok(SavedSiteReplicationExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: u64::try_from(export.archive_bytes.len()) + .map_err(|_| SiteReplicationPerformanceError::LimitExceeded)?, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if saved.is_err() { + let _ = fs::remove_file(&temporary); + } + saved +} + +pub fn validate_site_replication_limits( + duration: Duration, + traffic_bytes: u64, + late_arrival_cleanup: Duration, +) -> Result<(), SiteReplicationPerformanceError> { + if duration.is_zero() + || duration.as_millis() == 0 + || duration > MAX_SITE_REPLICATION_DURATION + || traffic_bytes == 0 + || traffic_bytes > MAX_SITE_REPLICATION_TRAFFIC_BYTES + || late_arrival_cleanup.is_zero() + || late_arrival_cleanup > MAX_LATE_ARRIVAL_CLEANUP + || late_arrival_cleanup >= duration + { + return Err(SiteReplicationPerformanceError::LimitExceeded); + } + Ok(()) +} + +pub fn read_protected_site_replication_credential(path: &Path) -> Result, SiteReplicationPerformanceError> { + let metadata = fs::symlink_metadata(path).map_err(SiteReplicationPerformanceError::Io)?; + if metadata.file_type().is_symlink() || !metadata.is_file() { + return Err(SiteReplicationPerformanceError::InvalidCredential); + } + #[cfg(unix)] + { + use std::os::unix::fs::{MetadataExt as _, OpenOptionsExt as _, PermissionsExt as _}; + if metadata.uid() != process_uid() || metadata.permissions().mode() & 0o077 != 0 { + return Err(SiteReplicationPerformanceError::InvalidCredential); + } + let mut options = OpenOptions::new(); + options.read(true).custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW); + let mut file = options.open(path).map_err(SiteReplicationPerformanceError::Io)?; + let opened = file.metadata().map_err(SiteReplicationPerformanceError::Io)?; + if opened.dev() != metadata.dev() || opened.ino() != metadata.ino() { + return Err(SiteReplicationPerformanceError::InvalidCredential); + } + read_credential(&mut file) + } + #[cfg(not(unix))] + { + read_credential(&mut File::open(path).map_err(SiteReplicationPerformanceError::Io)?) + } +} + +fn read_credential(reader: &mut impl Read) -> Result, SiteReplicationPerformanceError> { + let mut bytes = Vec::with_capacity(256); + reader + .take(4_097) + .read_to_end(&mut bytes) + .map_err(SiteReplicationPerformanceError::Io)?; + while matches!(bytes.last(), Some(b'\n' | b'\r')) { + bytes.pop(); + } + let value = String::from_utf8(bytes).map_err(|_| SiteReplicationPerformanceError::InvalidCredential)?; + if value.is_empty() || value.len() > 4_096 || value.trim() != value || value.contains('\0') { + return Err(SiteReplicationPerformanceError::InvalidCredential); + } + Ok(Zeroizing::new(value)) +} + +impl SiteReplicationPerformanceRequest { + fn validate(&self, now: i64) -> Result<(), SiteReplicationPerformanceError> { + if self.schema_version != SITE_REPLICATION_SCHEMA_VERSION { + return Err(SiteReplicationPerformanceError::UnsupportedVersion); + } + if self.capability != SITE_REPLICATION_CAPABILITY { + return Err(SiteReplicationPerformanceError::UnsupportedCapability); + } + if !self.consent.confirmed || self.consent.policy_revision == 0 { + return Err(SiteReplicationPerformanceError::ConsentRequired); + } + if self.consent.expires_at_unix <= now || self.expires_at_unix > self.consent.expires_at_unix { + return Err(SiteReplicationPerformanceError::ConsentExpired); + } + let validity = self + .expires_at_unix + .checked_sub(self.produced_at_unix) + .ok_or(SiteReplicationPerformanceError::Expired)?; + if self.produced_at_unix > now.saturating_add(MAX_FUTURE_SKEW_SECONDS) + || validity <= 0 + || validity > MAX_VALIDITY_SECONDS + || self.expires_at_unix <= now + { + return Err(SiteReplicationPerformanceError::Expired); + } + validate_site_replication_limits(self.duration, self.traffic_bytes, self.late_arrival_cleanup)?; + if !uuid7(&self.run_uid) + || !uuid7(&self.artifact_uid) + || !uuid7(&self.consent.consent_uid) + || !resource_names_match(self) + || !opaque_alias(&self.source_alias) + || !opaque_alias(&self.destination_alias) + || self.source_alias == self.destination_alias + || !deployment_id(&self.source_deployment_id) + || !deployment_id(&self.destination_deployment_id) + || self.source_deployment_id == self.destination_deployment_id + || !scratch_bucket(&self.scratch_bucket) + || !lower_hex(&self.provenance.source_commit, 40) + || !lower_hex(&self.provenance.executable_sha256, 64) + || self.provenance.build_features.len() > MAX_BUILD_FEATURES + || !self.provenance.build_features.iter().all(|value| build_feature(value)) + || !rustfs_version(&self.provenance.rustfs_version) + { + return Err(SiteReplicationPerformanceError::InvalidRequest); + } + Ok(()) + } +} + +struct ActiveGuard; + +impl Drop for ActiveGuard { + fn drop(&mut self) { + SITE_REPLICATION_COLLECTOR_ACTIVE.store(false, Ordering::Release); + } +} + +fn success_measurement( + request: &SiteReplicationPerformanceRequest, + sample: SiteReplicationProbeMeasurement, +) -> SiteReplicationMeasurement { + let duration_millis = millis(sample.duration); + let max_lag = millis(sample.max_observed_lag); + SiteReplicationMeasurement { + result: result( + request, + SiteReplicationOutcome::Succeeded, + SiteReplicationReasonCode::Complete, + duration_millis, + 1, + Some(SiteReplicationPerformanceData { + replicated_bytes: sample.replicated_bytes, + confirmed_objects: sample.confirmed_objects, + duration_millis, + max_observed_lag_millis: max_lag, + error_count: 0, + }), + ), + target: target_result( + request, + SiteReplicationOutcome::Succeeded, + SiteReplicationTargetReasonCode::Complete, + sample.replicated_bytes, + sample.confirmed_objects, + duration_millis, + Some(max_lag), + ), + } +} + +fn failed_measurement( + request: &SiteReplicationPerformanceRequest, + elapsed: Duration, + error: SiteReplicationProbeError, +) -> SiteReplicationMeasurement { + let reason = if error == SiteReplicationProbeError::PermissionDenied { + SiteReplicationReasonCode::PermissionDenied + } else { + SiteReplicationReasonCode::CollectionFailed + }; + terminal_measurement(request, SiteReplicationOutcome::Failed, reason, target_reason(error), elapsed) +} + +fn terminal_measurement( + request: &SiteReplicationPerformanceRequest, + outcome: SiteReplicationOutcome, + reason: SiteReplicationReasonCode, + target_reason: SiteReplicationTargetReasonCode, + elapsed: Duration, +) -> SiteReplicationMeasurement { + let duration_millis = millis_allow_zero(elapsed); + SiteReplicationMeasurement { + result: result(request, outcome, reason, duration_millis, 0, None), + target: target_result(request, outcome, target_reason, 0, 0, duration_millis, None), + } +} + +fn result( + request: &SiteReplicationPerformanceRequest, + outcome: SiteReplicationOutcome, + reason_code: SiteReplicationReasonCode, + duration_millis: u64, + completed_units: u32, + data: Option, +) -> SiteReplicationDiagnosticResult { + SiteReplicationDiagnosticResult { + schema_version: SITE_REPLICATION_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: SITE_REPLICATION_TOOL_ID, + capability: SITE_REPLICATION_CAPABILITY, + outcome, + reason_code, + duration_millis: duration_millis.min(30_000), + provenance: request.provenance.clone(), + coverage: Coverage { + requested_units: 1, + completed_units, + unit: "WINDOW", + }, + data, + } +} + +fn target_result( + request: &SiteReplicationPerformanceRequest, + outcome: SiteReplicationOutcome, + reason_code: SiteReplicationTargetReasonCode, + replicated_bytes: u64, + confirmed_objects: u64, + duration_millis: u64, + max_observed_lag_millis: Option, +) -> SiteReplicationTargetResult { + SiteReplicationTargetResult { + source_alias: request.source_alias.clone(), + destination_alias: request.destination_alias.clone(), + outcome, + reason_code, + requested_bytes: request.traffic_bytes, + duration_millis, + concurrency: 1, + bytes_unit: "BYTE", + duration_unit: "MILLISECOND", + operation_count_unit: "OPERATION", + replicated_bytes, + confirmed_objects, + max_observed_lag_millis, + } +} + +fn target_reason(error: SiteReplicationProbeError) -> SiteReplicationTargetReasonCode { + match error { + SiteReplicationProbeError::SiteReplicationUnavailable => SiteReplicationTargetReasonCode::SiteReplicationUnavailable, + SiteReplicationProbeError::EndpointUnavailable => SiteReplicationTargetReasonCode::EndpointUnavailable, + SiteReplicationProbeError::PermissionDenied => SiteReplicationTargetReasonCode::PermissionDenied, + SiteReplicationProbeError::TimedOut => SiteReplicationTargetReasonCode::TimedOut, + SiteReplicationProbeError::Cancelled => SiteReplicationTargetReasonCode::Cancelled, + SiteReplicationProbeError::VersioningRequired => SiteReplicationTargetReasonCode::VersioningRequired, + SiteReplicationProbeError::DestinationUnconfirmed => SiteReplicationTargetReasonCode::DestinationUnconfirmed, + SiteReplicationProbeError::CleanupFailed => SiteReplicationTargetReasonCode::CleanupFailed, + SiteReplicationProbeError::ProtocolFailure => SiteReplicationTargetReasonCode::ProtocolFailure, + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct Envelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: &'static str, + schema_version: u16, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: &'a str, + targets: Targets<'a>, + payload: Payload<'a>, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct Targets<'a> { + source_deployment: &'a str, + destination_deployment: &'a str, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct Payload<'a> { + path: &'static str, + media_type: &'static str, + size_bytes: u64, + sha256: &'a str, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct EnvelopeSignature<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +fn signature_document(key: &DeviceIdentity, key_id: &str, envelope: &[u8]) -> Result, SiteReplicationPerformanceError> { + let pkcs8 = key.to_pkcs8_der().map_err(|_| SiteReplicationPerformanceError::Signing)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| SiteReplicationPerformanceError::Signing)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + envelope.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.extend_from_slice(envelope); + let signature: Signature = signing_key.sign(&input); + serde_json::to_vec(&EnvelopeSignature { + algorithm: "ES256", + key_id, + value: URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()), + }) + .map_err(|_| SiteReplicationPerformanceError::Encoding) +} + +fn archive(envelope: &[u8], signature: &[u8], result: &[u8]) -> Result, SiteReplicationPerformanceError> { + let cursor = Cursor::new(Vec::with_capacity(envelope.len() + signature.len() + result.len() + 512)); + let mut writer = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(OUTPUT_MODE); + for (name, bytes) in [(ENVELOPE_PATH, envelope), (SIGNATURE_PATH, signature), (RESULT_PATH, result)] { + writer + .start_file(name, options) + .map_err(|_| SiteReplicationPerformanceError::Encoding)?; + writer.write_all(bytes).map_err(SiteReplicationPerformanceError::Io)?; + } + writer + .finish() + .map(|cursor| cursor.into_inner()) + .map_err(|_| SiteReplicationPerformanceError::Encoding) +} + +fn timestamp(value: i64) -> Result { + OffsetDateTime::from_unix_timestamp(value) + .map_err(|_| SiteReplicationPerformanceError::InvalidRequest)? + .format(&Rfc3339) + .map_err(|_| SiteReplicationPerformanceError::Encoding) +} + +fn check_cancel(cancel: &CancellationToken) -> Result<(), SiteReplicationPerformanceError> { + if cancel.is_cancelled() { + Err(SiteReplicationPerformanceError::Cancelled) + } else { + Ok(()) + } +} + +fn map_create_error(error: std::io::Error) -> SiteReplicationPerformanceError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + SiteReplicationPerformanceError::AlreadyExists + } else { + SiteReplicationPerformanceError::Io(error) + } +} + +fn map_publish_error(error: std::io::Error) -> SiteReplicationPerformanceError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + SiteReplicationPerformanceError::AlreadyExists + } else { + SiteReplicationPerformanceError::Io(error) + } +} + +#[derive(Deserialize)] +struct SiteReplicationInfo { + enabled: bool, + #[serde(default)] + sites: Vec, +} + +#[derive(Deserialize)] +struct SiteInfo { + #[serde(rename = "deploymentID", alias = "deploymentId")] + deployment_id: String, +} + +#[derive(Deserialize)] +struct ListVersionsResult { + #[serde(rename = "IsTruncated", default)] + is_truncated: bool, + #[serde(rename = "Version", default)] + versions: Vec, + #[serde(rename = "DeleteMarker", default)] + delete_markers: Vec, +} + +#[derive(Deserialize)] +struct ListedVersion { + #[serde(rename = "Key")] + key: String, + #[serde(rename = "VersionId")] + version_id: String, +} + +async fn drain_response( + response: Response, + max_bytes: u64, + deadline: Instant, + cancel: Option<&CancellationToken>, +) -> Result, SiteReplicationProbeError> { + if response.content_length().is_some_and(|length| length > max_bytes) { + return Err(SiteReplicationProbeError::ProtocolFailure); + } + let max_bytes = usize::try_from(max_bytes).map_err(|_| SiteReplicationProbeError::ProtocolFailure)?; + let mut body = Vec::with_capacity(max_bytes.min(16_384)); + let mut stream = response.bytes_stream(); + loop { + let next = tokio::time::timeout_at(tokio::time::Instant::from_std(deadline), stream.next()); + let chunk = if let Some(cancel) = cancel { + tokio::select! { + () = cancel.cancelled() => return Err(SiteReplicationProbeError::Cancelled), + chunk = next => chunk, + } + } else { + next.await + } + .map_err(|_| SiteReplicationProbeError::TimedOut)?; + let Some(chunk) = chunk else { break }; + let chunk = chunk.map_err(|_| SiteReplicationProbeError::ProtocolFailure)?; + if body.len().saturating_add(chunk.len()) > max_bytes { + return Err(SiteReplicationProbeError::ProtocolFailure); + } + body.extend_from_slice(&chunk); + } + Ok(body) +} + +fn object_url(endpoint: &Url, bucket: &str, key: &str, version_id: Option<&str>) -> Result { + let mut url = endpoint + .join(&format!("{bucket}/{}", encode_path(key))) + .map_err(|_| SiteReplicationProbeError::ProtocolFailure)?; + if let Some(version_id) = version_id { + url.query_pairs_mut().append_pair("versionId", version_id); + } + Ok(url) +} + +fn list_versions_url(endpoint: &Url, bucket: &str, key: &str) -> Result { + let mut url = endpoint + .join(&format!("{bucket}/")) + .map_err(|_| SiteReplicationProbeError::ProtocolFailure)?; + url.query_pairs_mut().append_pair("versions", "").append_pair("prefix", key); + Ok(url) +} + +fn encode_path(value: &str) -> String { + value + .split('/') + .map(|segment| utf8_percent_encode(segment, NON_ALPHANUMERIC).to_string()) + .collect::>() + .join("/") +} + +fn deployment_endpoint(value: &str) -> Result { + let mut url = Url::parse(value).map_err(|_| SiteReplicationPerformanceError::InvalidEndpoint)?; + let local_http = url.scheme() == "http" + && url + .host_str() + .is_some_and(|host| host == "localhost" || host.parse::().is_ok_and(|ip| ip.is_loopback())); + if (url.scheme() != "https" && !local_http) + || url.cannot_be_a_base() + || !url.username().is_empty() + || url.password().is_some() + || url.path() != "/" + || url.query().is_some() + || url.fragment().is_some() + { + return Err(SiteReplicationPerformanceError::InvalidEndpoint); + } + if !url.path().ends_with('/') { + url.set_path("/"); + } + Ok(url) +} + +fn status_error(status: StatusCode) -> Result { + if matches!(status, StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN) { + Err(SiteReplicationProbeError::PermissionDenied) + } else { + Err(SiteReplicationProbeError::ProtocolFailure) + } +} + +fn resource_names_match(request: &SiteReplicationPerformanceRequest) -> bool { + let Some(organization_uid) = request.organization_name.strip_prefix("organizations/") else { + return false; + }; + let cluster_prefix = format!("{}/clusters/", request.organization_name); + let device_prefix = format!("{}/clusterDevices/", request.cluster_name); + uuid7(organization_uid) + && request.cluster_name.strip_prefix(&cluster_prefix).is_some_and(uuid7) + && request + .destination_cluster_name + .strip_prefix(&cluster_prefix) + .is_some_and(uuid7) + && request.destination_cluster_name != request.cluster_name + && request.device_name.strip_prefix(&device_prefix).is_some_and(uuid7) +} + +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn opaque_alias(value: &str) -> bool { + !value.is_empty() + && value.len() <= 128 + && value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || b"._:-".contains(&byte)) +} + +fn deployment_id(value: &str) -> bool { + !value.is_empty() + && value.len() <= 128 + && value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || b"-_.".contains(&byte)) +} + +fn scratch_bucket(value: &str) -> bool { + (3..=63).contains(&value.len()) + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'.' | b'-')) + && value.as_bytes().first().is_some_and(u8::is_ascii_alphanumeric) + && value.as_bytes().last().is_some_and(u8::is_ascii_alphanumeric) + && !value.contains("..") +} + +fn lower_hex(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_hexdigit() && !byte.is_ascii_uppercase()) +} + +fn build_feature(value: &str) -> bool { + !value.is_empty() + && value.len() <= 64 + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'-' | b'_')) +} + +fn rustfs_version(value: &str) -> bool { + !value.is_empty() + && value.len() <= 64 + && value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'-' | b'+')) +} + +fn millis(value: Duration) -> u64 { + millis_allow_zero(value).max(1) +} + +fn millis_allow_zero(value: Duration) -> u64 { + u64::try_from(value.as_millis()) + .unwrap_or(MAX_SAFE_INTEGER) + .min(MAX_SAFE_INTEGER) +} + +fn hex_lower(bytes: &[u8]) -> String { + let mut value = String::with_capacity(bytes.len() * 2); + for byte in bytes { + use std::fmt::Write as _; + let _ = write!(value, "{byte:02x}"); + } + value +} + +fn unix_now() -> Result { + let duration = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| SiteReplicationPerformanceError::InvalidRequest)?; + i64::try_from(duration.as_secs()).map_err(|_| SiteReplicationPerformanceError::InvalidRequest) +} + +#[cfg(unix)] +#[allow(unsafe_code)] +fn process_uid() -> u32 { + // SAFETY: geteuid has no pointer arguments or caller preconditions. + unsafe { libc::geteuid() } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::AtomicUsize; + + fn now() -> i64 { + i64::try_from(SystemTime::now().duration_since(UNIX_EPOCH).expect("current time").as_secs()) + .expect("current time fits i64") + } + + fn request() -> SiteReplicationPerformanceRequest { + let now = now(); + let organization = "organizations/019e3ae0-0000-7000-8000-000000000010"; + let cluster = format!("{organization}/clusters/019e3ae0-0000-7000-8000-000000000011"); + SiteReplicationPerformanceRequest { + organization_name: organization.to_owned(), + cluster_name: cluster.clone(), + destination_cluster_name: format!("{organization}/clusters/019e3ae0-0000-7000-8000-000000000016"), + device_name: format!("{cluster}/clusterDevices/019e3ae0-0000-7000-8000-000000000012"), + run_uid: "019e3ae0-0000-7000-8000-000000000013".to_owned(), + artifact_uid: "019e3ae0-0000-7000-8000-000000000014".to_owned(), + schema_version: 1, + capability: SITE_REPLICATION_CAPABILITY.to_owned(), + consent: LocalSiteReplicationConsent { + consent_uid: "019e3ae0-0000-7000-8000-000000000015".to_owned(), + policy_revision: 7, + expires_at_unix: now + 120, + confirmed: true, + nonce: [0x6b; 32], + }, + produced_at_unix: now, + expires_at_unix: now + 60, + duration: Duration::from_secs(2), + traffic_bytes: 65_536, + source_alias: "site-a".to_owned(), + source_deployment_id: "deployment-a".to_owned(), + destination_alias: "site-b".to_owned(), + destination_deployment_id: "deployment-b".to_owned(), + scratch_bucket: "connect-replication-scratch".to_owned(), + late_arrival_cleanup: Duration::from_millis(500), + provenance: SiteReplicationProvenance::new("a".repeat(40), "b".repeat(64), "1.0.0-rc.6", Vec::new()), + } + } + + struct CountingProbe { + calls: AtomicUsize, + result: Result, + } + + impl SiteReplicationProbe for CountingProbe { + fn probe<'a>( + &'a self, + _request: &'a SiteReplicationPerformanceRequest, + _cancel: &'a CancellationToken, + ) -> SiteReplicationProbeFuture<'a> { + self.calls.fetch_add(1, Ordering::Relaxed); + Box::pin(async move { self.result }) + } + } + + #[tokio::test] + async fn destination_confirmation_produces_frozen_aggregate_shape() { + let request = request(); + let probe = CountingProbe { + calls: AtomicUsize::new(0), + result: Ok(SiteReplicationProbeMeasurement { + replicated_bytes: 65_536, + confirmed_objects: 1, + duration: Duration::from_secs(1), + max_observed_lag: Duration::from_millis(250), + }), + }; + let measured = measure_site_replication(&request, &probe, &CancellationToken::new()) + .await + .expect("site replication measurement"); + let value = serde_json::to_value(&measured.result).expect("result JSON"); + assert_eq!(value["toolId"], SITE_REPLICATION_TOOL_ID); + assert_eq!(value["capability"], SITE_REPLICATION_CAPABILITY); + assert_eq!(value["outcome"], "SUCCEEDED"); + assert_eq!(value["data"]["replicatedBytes"], 65_536); + assert_eq!(value["data"]["confirmedObjects"], 1); + for forbidden in [ + "sourceEndpoint", + "destinationEndpoint", + "credentialFile", + "caFile", + "scratchBucket", + "cleanupVersionId", + "lateArrivalCleanup", + ] { + assert!(value["data"].get(forbidden).is_none(), "result leaked {forbidden}"); + } + let export = sign_site_replication_export(&request, &measured, &DeviceIdentity::generate(), &CancellationToken::new()) + .expect("signed site replication export"); + let envelope = String::from_utf8(export.envelope_json.clone()).expect("envelope UTF-8"); + let result = String::from_utf8(export.result_json).expect("result UTF-8"); + let envelope_value: serde_json::Value = serde_json::from_str(&envelope).expect("envelope JSON"); + assert_eq!(envelope_value["targets"]["sourceDeployment"], request.cluster_name); + assert_eq!(envelope_value["targets"]["destinationDeployment"], request.destination_cluster_name); + for secret in [ + "source.example", + "destination.example", + "connect-replication-scratch", + "deployment-a", + "deployment-b", + ] { + assert!(!envelope.contains(secret), "envelope leaked {secret}"); + assert!(!result.contains(secret), "result leaked {secret}"); + } + } + + #[tokio::test] + async fn invalid_consent_and_budgets_fail_before_probe() { + let probe = CountingProbe { + calls: AtomicUsize::new(0), + result: Err(SiteReplicationProbeError::ProtocolFailure), + }; + let mut invalid = request(); + invalid.consent.confirmed = false; + assert!(matches!( + measure_site_replication(&invalid, &probe, &CancellationToken::new()).await, + Err(SiteReplicationPerformanceError::ConsentRequired) + )); + invalid = request(); + invalid.traffic_bytes = MAX_SITE_REPLICATION_TRAFFIC_BYTES + 1; + assert!(matches!( + measure_site_replication(&invalid, &probe, &CancellationToken::new()).await, + Err(SiteReplicationPerformanceError::LimitExceeded) + )); + invalid = request(); + invalid.source_deployment_id = invalid.destination_deployment_id.clone(); + assert!(matches!( + measure_site_replication(&invalid, &probe, &CancellationToken::new()).await, + Err(SiteReplicationPerformanceError::InvalidRequest) + )); + invalid = request(); + invalid.destination_cluster_name = invalid.cluster_name.clone(); + assert!(matches!( + measure_site_replication(&invalid, &probe, &CancellationToken::new()).await, + Err(SiteReplicationPerformanceError::InvalidRequest) + )); + invalid = request(); + invalid.destination_cluster_name = format!("{}/clusters/not-a-uuid", invalid.organization_name); + assert!(matches!( + measure_site_replication(&invalid, &probe, &CancellationToken::new()).await, + Err(SiteReplicationPerformanceError::InvalidRequest) + )); + invalid = request(); + invalid.destination_cluster_name = + "organizations/019e3ae0-0000-7000-8000-000000000099/clusters/019e3ae0-0000-7000-8000-000000000016".to_owned(); + assert!(matches!( + measure_site_replication(&invalid, &probe, &CancellationToken::new()).await, + Err(SiteReplicationPerformanceError::InvalidRequest) + )); + assert_eq!(probe.calls.load(Ordering::Relaxed), 0); + } + + #[tokio::test] + async fn unconfirmed_and_cancelled_results_never_report_success() { + for (error, expected) in [ + (SiteReplicationProbeError::DestinationUnconfirmed, SiteReplicationOutcome::Failed), + (SiteReplicationProbeError::SiteReplicationUnavailable, SiteReplicationOutcome::Failed), + (SiteReplicationProbeError::Cancelled, SiteReplicationOutcome::Cancelled), + (SiteReplicationProbeError::CleanupFailed, SiteReplicationOutcome::Failed), + ] { + let probe = CountingProbe { + calls: AtomicUsize::new(0), + result: Err(error), + }; + let measured = measure_site_replication(&request(), &probe, &CancellationToken::new()) + .await + .expect("terminal measurement"); + assert_eq!(measured.result.outcome(), expected); + assert!(measured.result.data().is_none()); + } + } + + #[test] + fn cleanup_queries_are_version_specific_and_task_scoped() { + let endpoint = Url::parse("https://source.example/").expect("endpoint"); + let object = object_url(&endpoint, "scratch-bucket", "path/a b", Some("version+1")).expect("object URL"); + assert_eq!(object.as_str(), "https://source.example/scratch-bucket/path/a%20b?versionId=version%2B1"); + let list = list_versions_url(&endpoint, "scratch-bucket", "path/a b").expect("list URL"); + assert!(list.query_pairs().any(|(key, value)| key == "versions" && value.is_empty())); + assert!(list.query_pairs().any(|(key, value)| key == "prefix" && value == "path/a b")); + } + + #[test] + fn version_listing_parses_versions_and_delete_markers() { + let xml = br#"falsetask-keyv1task-keym1"#; + let listed: ListVersionsResult = quick_xml::de::from_reader(xml.as_slice()).expect("version listing"); + assert_eq!(listed.versions[0].version_id, "v1"); + assert_eq!(listed.delete_markers[0].version_id, "m1"); + } +} diff --git a/rustfs/src/connect/diagnostics/profile_cpu.rs b/rustfs/src/connect/diagnostics/profile_cpu.rs new file mode 100644 index 000000000..ac16e25bd --- /dev/null +++ b/rustfs/src/connect/diagnostics/profile_cpu.rs @@ -0,0 +1,1175 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded profile result and signed local export primitives. +//! +//! The optional Pyroscope pprof backend supplies an on-demand process sampler. +//! Raw frames stay local: the exported summary contains nonce-bound symbol IDs +//! and counts only, never symbol text, paths, addresses, or thread metadata. + +use std::fs::{self, File, OpenOptions}; +use std::io::{Cursor, Write as _}; +use std::path::Path; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; + +use base64_simd::URL_SAFE_NO_PAD; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use serde::Serialize; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use crate::connect::DeviceIdentity; + +pub const PROFILE_SCHEMA_VERSION: u16 = 1; +pub const CPU_PROFILE_CAPABILITY: &str = "profile.cpu@1"; +pub const MEMORY_PROFILE_CAPABILITY: &str = "profile.memory@1"; +pub const THREAD_PROFILE_CAPABILITY: &str = "profile.threads@1"; +pub const MAX_PROFILE_DURATION: Duration = Duration::from_secs(30); +pub const MAX_RESULT_BYTES: usize = 262_144; +pub const MAX_ENVELOPE_BYTES: usize = 16_384; +pub const MAX_ARCHIVE_BYTES: usize = 524_288; +pub const MAX_DECOMPRESSED_BYTES: usize = 278_528; +pub const MAX_BUILD_FEATURES: usize = 64; +pub const MAX_VALIDITY_SECONDS: i64 = 2_592_000; +pub const MAX_FUTURE_SKEW_SECONDS: i64 = 300; + +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1\0"; +const ENVELOPE_PATH: &str = "envelope.json"; +const SIGNATURE_PATH: &str = "envelope.sig"; +const RESULT_PATH: &str = "result.json"; +const OUTPUT_MODE: u32 = 0o600; + +static PROFILE_COLLECTOR_ACTIVE: AtomicBool = AtomicBool::new(false); + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +pub enum ProfileTool { + #[serde(rename = "profile.cpu")] + Cpu, + #[serde(rename = "profile.memory")] + Memory, + #[serde(rename = "profile.threads")] + Threads, +} + +impl ProfileTool { + pub const fn id(self) -> &'static str { + match self { + Self::Cpu => "profile.cpu", + Self::Memory => "profile.memory", + Self::Threads => "profile.threads", + } + } + + pub const fn capability(self) -> &'static str { + match self { + Self::Cpu => CPU_PROFILE_CAPABILITY, + Self::Memory => MEMORY_PROFILE_CAPABILITY, + Self::Threads => THREAD_PROFILE_CAPABILITY, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ProfileOutcome { + Succeeded, + Partial, + Failed, + Unsupported, + Cancelled, +} + +impl ProfileOutcome { + pub const fn as_str(self) -> &'static str { + match self { + Self::Succeeded => "SUCCEEDED", + Self::Partial => "PARTIAL", + Self::Failed => "FAILED", + Self::Unsupported => "UNSUPPORTED", + Self::Cancelled => "CANCELLED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ProfileReasonCode { + Complete, + LimitExceeded, + SourceUnavailable, + UnsupportedTool, + UnsupportedVersion, + UnsupportedPlatform, + Cancelled, + CounterReset, + InvalidInput, + CollectionFailed, +} + +impl ProfileReasonCode { + pub const fn as_str(self) -> &'static str { + match self { + Self::Complete => "COMPLETE", + Self::LimitExceeded => "LIMIT_EXCEEDED", + Self::SourceUnavailable => "SOURCE_UNAVAILABLE", + Self::UnsupportedTool => "UNSUPPORTED_TOOL", + Self::UnsupportedVersion => "UNSUPPORTED_VERSION", + Self::UnsupportedPlatform => "UNSUPPORTED_PLATFORM", + Self::Cancelled => "CANCELLED", + Self::CounterReset => "COUNTER_RESET", + Self::InvalidInput => "INVALID_INPUT", + Self::CollectionFailed => "COLLECTION_FAILED", + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ProfileProvenance { + repository: &'static str, + source_commit: String, + executable_sha256: String, + rustfs_version: String, + os_family: ProfileOsFamily, + architecture: ProfileArchitecture, + build_features: Vec, +} + +impl ProfileProvenance { + pub fn new( + source_commit: impl Into, + executable_sha256: impl Into, + rustfs_version: impl Into, + build_features: Vec, + ) -> Self { + Self { + repository: "rustfs/rustfs", + source_commit: source_commit.into(), + executable_sha256: executable_sha256.into(), + rustfs_version: rustfs_version.into(), + os_family: ProfileOsFamily::current(), + architecture: ProfileArchitecture::current(), + build_features, + } + } + + pub(crate) fn executable_sha256(&self) -> &str { + &self.executable_sha256 + } + + pub(crate) fn source_commit(&self) -> &str { + &self.source_commit + } + + pub(crate) fn rustfs_version(&self) -> &str { + &self.rustfs_version + } + + pub(crate) fn build_features(&self) -> &[String] { + &self.build_features + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ProfileOsFamily { + Linux, + Darwin, + Windows, + Freebsd, + Other, +} + +impl ProfileOsFamily { + fn current() -> Self { + match std::env::consts::OS { + "linux" => Self::Linux, + "macos" => Self::Darwin, + "windows" => Self::Windows, + "freebsd" => Self::Freebsd, + _ => Self::Other, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +pub enum ProfileArchitecture { + #[serde(rename = "x86_64")] + X86_64, + Aarch64, + Other, +} + +impl ProfileArchitecture { + fn current() -> Self { + match std::env::consts::ARCH { + "x86_64" => Self::X86_64, + "aarch64" => Self::Aarch64, + _ => Self::Other, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LocalProfileConsent { + pub consent_uid: String, + pub policy_revision: u64, + pub expires_at_unix: i64, + pub confirmed: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ProfileCaptureRequest { + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub schema_version: u16, + pub capability: String, + pub consent: LocalProfileConsent, + pub produced_at_unix: i64, + pub expires_at_unix: i64, + pub nonce: [u8; 32], + pub duration: Duration, + pub sample_period: Duration, + pub provenance: ProfileProvenance, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ProfileCoverage { + requested_units: u32, + completed_units: u32, + unit: &'static str, +} + +impl ProfileCoverage { + pub(super) const fn complete_window() -> Self { + Self { + requested_units: 1, + completed_units: 1, + unit: "WINDOW", + } + } + + const fn none() -> Self { + Self { + requested_units: 0, + completed_units: 0, + unit: "WINDOW", + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct CpuProfileData { + sample_period_micros: u64, + samples: Vec, + dropped_sample_count: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct CpuProfileSample { + symbol_id: String, + sample_count: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct MemoryProfileData { + scope: &'static str, + allocated_bytes: u64, + allocation_count: u64, + sample_period_micros: u64, +} + +impl MemoryProfileData { + pub(super) const fn allocation_aggregates(allocated_bytes: u64, allocation_count: u64, sample_period_micros: u64) -> Self { + Self { + scope: "ALLOCATION_AGGREGATES", + allocated_bytes, + allocation_count, + sample_period_micros, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ThreadProfileData { + scope: ThreadProfileScope, + states: Vec, +} + +impl ThreadProfileData { + #[cfg(target_os = "linux")] + pub(super) fn native(states: Vec) -> Self { + Self { + scope: ThreadProfileScope::NativeThreads, + states, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ThreadState { + Runnable, + Waiting, + Blocked, + Unknown, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ThreadStateCount { + state: ThreadState, + thread_count: u64, +} + +impl ThreadStateCount { + #[cfg(target_os = "linux")] + pub(super) const fn new(state: ThreadState, thread_count: u64) -> Self { + Self { state, thread_count } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ThreadProfileScope { + TokioRuntime, + NativeThreads, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(untagged)] +pub enum ProfileData { + Cpu(CpuProfileData), + Memory(MemoryProfileData), + Threads(ThreadProfileData), +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ProfileResult { + schema_version: u16, + run_uid: String, + tool_id: ProfileTool, + capability: &'static str, + outcome: ProfileOutcome, + reason_code: ProfileReasonCode, + duration_millis: u64, + provenance: ProfileProvenance, + coverage: ProfileCoverage, + data: Option, +} + +impl ProfileResult { + pub fn outcome(&self) -> ProfileOutcome { + self.outcome + } + + pub fn reason_code(&self) -> ProfileReasonCode { + self.reason_code + } + + pub fn data(&self) -> Option<&ProfileData> { + self.data.as_ref() + } + + pub(super) fn succeeded(request: &ProfileCaptureRequest, tool: ProfileTool, duration: Duration, data: ProfileData) -> Self { + Self { + schema_version: PROFILE_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: tool, + capability: tool.capability(), + outcome: ProfileOutcome::Succeeded, + reason_code: ProfileReasonCode::Complete, + duration_millis: u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).min(30_000), + provenance: request.provenance.clone(), + coverage: ProfileCoverage::complete_window(), + data: Some(data), + } + } + + #[cfg(all( + feature = "pyroscope", + any( + all(target_os = "macos", any(target_arch = "x86_64", target_arch = "aarch64")), + all( + target_os = "linux", + target_env = "gnu", + any(target_arch = "x86_64", target_arch = "aarch64") + ) + ) + ))] + fn partial(request: &ProfileCaptureRequest, tool: ProfileTool, duration: Duration, data: ProfileData) -> Self { + Self { + schema_version: PROFILE_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: tool, + capability: tool.capability(), + outcome: ProfileOutcome::Partial, + reason_code: ProfileReasonCode::LimitExceeded, + duration_millis: u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).min(30_000), + provenance: request.provenance.clone(), + coverage: ProfileCoverage::complete_window(), + data: Some(data), + } + } + + pub(super) fn unsupported(request: &ProfileCaptureRequest, tool: ProfileTool, reason_code: ProfileReasonCode) -> Self { + Self { + schema_version: PROFILE_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: tool, + capability: tool.capability(), + outcome: ProfileOutcome::Unsupported, + reason_code, + duration_millis: 0, + provenance: request.provenance.clone(), + coverage: ProfileCoverage::none(), + data: None, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SignedProfileExport { + pub artifact_uid: String, + pub tool: ProfileTool, + pub outcome: ProfileOutcome, + pub reason_code: ProfileReasonCode, + pub archive_bytes: Vec, + pub archive_sha256: String, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SavedProfileExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +#[derive(Debug, Error)] +pub enum ProfileError { + #[error("profile_local_consent_required")] + ConsentRequired, + #[error("profile_local_consent_expired")] + ConsentExpired, + #[error("profile_request_expired")] + Expired, + #[error("profile_invalid_request")] + InvalidRequest, + #[error("profile_unsupported_version")] + UnsupportedVersion, + #[error("profile_unsupported_capability")] + UnsupportedCapability, + #[error("profile_limit_exceeded")] + LimitExceeded, + #[error("profile_collection_cancelled")] + Cancelled, + #[error("profile_collection_timed_out")] + TimedOut, + #[error("profile_collection_already_running")] + Busy, + #[error("profile_source_unavailable")] + SourceUnavailable, + #[error("profile_counter_reset")] + CounterReset, + #[error("profile_collection_failed")] + CollectionFailed, + #[error("profile_export_signing_failed")] + Signing, + #[error("profile_export_exists")] + AlreadyExists, + #[error("profile_export_io_failed")] + Io(#[source] std::io::Error), + #[error("profile_export_encoding_failed")] + Encoding, + #[error("profile_export_durability_failed_after_commit")] + DurabilityAfterCommit(#[source] std::io::Error), +} + +pub async fn capture_cpu_profile( + request: &ProfileCaptureRequest, + cancel: &CancellationToken, +) -> Result { + request.validate(ProfileTool::Cpu, unix_now()?)?; + check_cancel(cancel)?; + + #[cfg(all( + feature = "pyroscope", + any( + all(target_os = "macos", any(target_arch = "x86_64", target_arch = "aarch64")), + all( + target_os = "linux", + target_env = "gnu", + any(target_arch = "x86_64", target_arch = "aarch64") + ) + ) + ))] + { + let _lease = CollectorLease::acquire()?; + let owned_request = request.clone(); + let owned_cancel = cancel.clone(); + let (data, elapsed) = tokio::task::spawn_blocking(move || local_cpu::collect(&owned_request, &owned_cancel)) + .await + .map_err(|_| ProfileError::CollectionFailed)??; + check_cancel(cancel)?; + let outcome = if data.dropped_sample_count == 0 { + ProfileResult::succeeded(request, ProfileTool::Cpu, elapsed, ProfileData::Cpu(data)) + } else { + ProfileResult::partial(request, ProfileTool::Cpu, elapsed, ProfileData::Cpu(data)) + }; + Ok(outcome) + } + + #[cfg(not(all( + feature = "pyroscope", + any( + all(target_os = "macos", any(target_arch = "x86_64", target_arch = "aarch64")), + all( + target_os = "linux", + target_env = "gnu", + any(target_arch = "x86_64", target_arch = "aarch64") + ) + ) + )))] + Ok(ProfileResult::unsupported(request, ProfileTool::Cpu, ProfileReasonCode::UnsupportedTool)) +} + +pub async fn export_cpu_profile( + request: &ProfileCaptureRequest, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + let result = capture_cpu_profile(request, cancel).await?; + encode_signed_profile_export(request, &result, key, cancel) +} + +#[cfg(all( + feature = "pyroscope", + any( + all(target_os = "macos", any(target_arch = "x86_64", target_arch = "aarch64")), + all( + target_os = "linux", + target_env = "gnu", + any(target_arch = "x86_64", target_arch = "aarch64") + ) + ) +))] +mod local_cpu { + use std::collections::HashMap; + use std::time::{Duration, Instant}; + + use pyroscope::backend::{BackendConfig, PprofConfig, ReportData, pprof_backend}; + use sha2::{Digest as _, Sha256}; + use tokio_util::sync::CancellationToken; + + use super::{CpuProfileData, CpuProfileSample, ProfileCaptureRequest, ProfileError, check_cancel, hex_lower}; + + const SYMBOL_DOMAIN: &[u8] = b"rustfs-connect-cpu-symbol-v1\0"; + const MAX_SAMPLE_RATE_HZ: u32 = 100; + const MAX_STACK_RECORDS: usize = 65_536; + const MAX_SAMPLES: u64 = 65_536; + const MAX_UNIQUE_SYMBOLS: usize = 4_096; + const MAX_OUTPUT_SYMBOLS: usize = 256; + const MAX_SYMBOL_BYTES: usize = 4_096; + + pub(super) fn collect( + request: &ProfileCaptureRequest, + cancel: &CancellationToken, + ) -> Result<(CpuProfileData, Duration), ProfileError> { + let sample_period_micros = u64::try_from(request.sample_period.as_micros()).map_err(|_| ProfileError::LimitExceeded)?; + let sample_rate = 1_000_000_u64 + .checked_div(sample_period_micros) + .ok_or(ProfileError::LimitExceeded)?; + let sample_rate = u32::try_from(sample_rate).map_err(|_| ProfileError::LimitExceeded)?; + if sample_rate == 0 || sample_rate > MAX_SAMPLE_RATE_HZ { + return Err(ProfileError::LimitExceeded); + } + + let started = Instant::now(); + let deadline = started.checked_add(request.duration).ok_or(ProfileError::LimitExceeded)?; + let mut backend = pprof_backend(PprofConfig { sample_rate }, BackendConfig::default()) + .initialize() + .map_err(|_| ProfileError::SourceUnavailable)?; + + let report_result = + wait_for_window(deadline, cancel).and_then(|()| backend.report().map_err(|_| ProfileError::CollectionFailed)); + let shutdown_result = backend.shutdown().map_err(|_| ProfileError::CollectionFailed); + let batch = report_result?; + shutdown_result?; + let ReportData::Reports(reports) = batch.data else { + return Err(ProfileError::SourceUnavailable); + }; + + let mut accumulator = Accumulator::new(request.nonce); + for report in reports { + for (stack, count) in report.data { + accumulator.record_stack(stack.frames.iter().filter_map(|frame| frame.name.as_deref()), count)?; + } + } + let actual_period_micros = 1_000_000_u64 + .checked_div(u64::from(sample_rate)) + .ok_or(ProfileError::LimitExceeded)?; + Ok((accumulator.finish(actual_period_micros)?, started.elapsed())) + } + + fn wait_for_window(deadline: Instant, cancel: &CancellationToken) -> Result<(), ProfileError> { + const POLL_INTERVAL: Duration = Duration::from_millis(10); + loop { + check_cancel(cancel)?; + let now = Instant::now(); + if now >= deadline { + return Ok(()); + } + std::thread::sleep(deadline.saturating_duration_since(now).min(POLL_INTERVAL)); + } + } + + struct Accumulator { + nonce: [u8; 32], + samples: HashMap, + stack_records: usize, + accepted_sample_count: u64, + dropped_sample_count: u64, + } + + impl Accumulator { + fn new(nonce: [u8; 32]) -> Self { + Self { + nonce, + samples: HashMap::new(), + stack_records: 0, + accepted_sample_count: 0, + dropped_sample_count: 0, + } + } + + fn record_stack<'a>(&mut self, symbols: impl Iterator, count: usize) -> Result<(), ProfileError> { + self.stack_records = self.stack_records.checked_add(1).ok_or(ProfileError::LimitExceeded)?; + let count = u64::try_from(count).map_err(|_| ProfileError::LimitExceeded)?; + if self.stack_records > MAX_STACK_RECORDS { + return self.drop_samples(count); + } + let accepted_count = count.min(MAX_SAMPLES.saturating_sub(self.accepted_sample_count)); + self.drop_samples(count.saturating_sub(accepted_count))?; + if accepted_count == 0 { + return Ok(()); + } + self.accepted_sample_count = self + .accepted_sample_count + .checked_add(accepted_count) + .ok_or(ProfileError::LimitExceeded)?; + let symbol = symbols + .into_iter() + .find(|symbol| !symbol.is_empty() && symbol.len() <= MAX_SYMBOL_BYTES) + .unwrap_or(""); + let symbol_id = symbol_id(&self.nonce, symbol); + if !self.samples.contains_key(&symbol_id) && self.samples.len() >= MAX_UNIQUE_SYMBOLS { + return self.drop_samples(accepted_count); + } + let samples = self.samples.entry(symbol_id).or_default(); + *samples = samples.checked_add(accepted_count).ok_or(ProfileError::LimitExceeded)?; + Ok(()) + } + + fn drop_samples(&mut self, count: u64) -> Result<(), ProfileError> { + self.dropped_sample_count = self + .dropped_sample_count + .checked_add(count) + .ok_or(ProfileError::LimitExceeded)?; + Ok(()) + } + + fn finish(self, sample_period_micros: u64) -> Result { + let mut samples = self + .samples + .into_iter() + .map(|(symbol_id, sample_count)| CpuProfileSample { symbol_id, sample_count }) + .collect::>(); + samples.sort_unstable_by(|left, right| { + right + .sample_count + .cmp(&left.sample_count) + .then_with(|| left.symbol_id.cmp(&right.symbol_id)) + }); + let mut dropped_sample_count = self.dropped_sample_count; + if samples.len() > MAX_OUTPUT_SYMBOLS { + dropped_sample_count = samples[MAX_OUTPUT_SYMBOLS..] + .iter() + .try_fold(dropped_sample_count, |total, sample| { + total.checked_add(sample.sample_count).ok_or(ProfileError::LimitExceeded) + })?; + samples.truncate(MAX_OUTPUT_SYMBOLS); + } + if samples.is_empty() { + return Err(ProfileError::SourceUnavailable); + } + Ok(CpuProfileData { + sample_period_micros, + samples, + dropped_sample_count, + }) + } + } + + fn symbol_id(nonce: &[u8; 32], symbol: &str) -> String { + let mut digest = Sha256::new(); + digest.update(SYMBOL_DOMAIN); + digest.update(nonce); + digest.update(symbol.as_bytes()); + format!("sha256:{}", hex_lower(&digest.finalize())) + } + + #[cfg(test)] + mod tests { + use super::*; + + #[test] + fn summary_uses_nonce_bound_ids_and_excludes_raw_symbols() { + let raw_symbol = "rustfs::storage::disk::read_object"; + let mut first = Accumulator::new([7; 32]); + first + .record_stack([raw_symbol].into_iter(), 9) + .expect("first stack should be recorded"); + let first = first.finish(10_000).expect("first summary should be produced"); + let mut second = Accumulator::new([8; 32]); + second + .record_stack([raw_symbol].into_iter(), 9) + .expect("second stack should be recorded"); + let second = second.finish(10_000).expect("second summary should be produced"); + + assert_ne!(first.samples[0].symbol_id, second.samples[0].symbol_id); + assert_eq!(first.samples[0].sample_count, 9); + let encoded = serde_json::to_string(&first).expect("CPU summary should serialize"); + assert!(!encoded.contains(raw_symbol)); + assert!(!encoded.contains("read_object")); + assert!(!encoded.contains('/')); + } + + #[test] + fn summary_bounds_samples_and_output_symbols() { + let mut accumulator = Accumulator::new([3; 32]); + for index in 0..=MAX_OUTPUT_SYMBOLS { + let symbol = format!("rustfs::bounded::{index}"); + accumulator + .record_stack([symbol.as_str()].into_iter(), 1) + .expect("bounded stack should be recorded"); + } + accumulator + .record_stack(["rustfs::large_count"].into_iter(), usize::MAX) + .expect("large count should be bounded"); + let summary = accumulator.finish(10_000).expect("bounded summary should be produced"); + + assert_eq!(summary.samples.len(), MAX_OUTPUT_SYMBOLS); + assert!(summary.dropped_sample_count > 0); + assert!(summary.samples.iter().map(|sample| sample.sample_count).sum::() <= MAX_SAMPLES); + } + + #[test] + fn window_observes_cancellation() { + let cancel = CancellationToken::new(); + cancel.cancel(); + assert!(matches!( + wait_for_window(Instant::now() + Duration::from_secs(1), &cancel), + Err(ProfileError::Cancelled) + )); + } + } +} + +pub fn encode_signed_profile_export( + request: &ProfileCaptureRequest, + result: &ProfileResult, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + let now = unix_now()?; + request.validate(result.tool_id, now)?; + check_cancel(cancel)?; + let valid_data = match result.outcome { + ProfileOutcome::Succeeded | ProfileOutcome::Partial => result.data.is_some(), + ProfileOutcome::Failed | ProfileOutcome::Unsupported | ProfileOutcome::Cancelled => result.data.is_none(), + }; + if !valid_data + || result.run_uid != request.run_uid + || result.schema_version != request.schema_version + || result.capability != request.capability + { + return Err(ProfileError::InvalidRequest); + } + + let result_bytes = serde_json::to_vec(result).map_err(|_| ProfileError::Encoding)?; + if result_bytes.is_empty() || result_bytes.len() > MAX_RESULT_BYTES { + return Err(ProfileError::LimitExceeded); + } + + let device_key_id = hex_lower(&Sha256::digest(key.public_key_der())); + let envelope = ProfileEnvelope { + format_version: "rustfs.connect.diagnosticEnvelope/1", + protocol_version: "v1", + organization_name: &request.organization_name, + cluster_name: &request.cluster_name, + device_name: &request.device_name, + run_uid: &request.run_uid, + artifact_uid: &request.artifact_uid, + tool_id: result.tool_id, + schema_version: PROFILE_SCHEMA_VERSION, + classification: "L3", + consent_uid: &request.consent.consent_uid, + policy_revision: request.consent.policy_revision, + produced_at: timestamp(request.produced_at_unix)?, + expires_at: timestamp(request.expires_at_unix)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(request.nonce), + device_key_id: &device_key_id, + payload: ProfilePayload { + path: RESULT_PATH, + media_type: "application/json", + size_bytes: result_bytes.len() as u64, + sha256: hex_lower(&Sha256::digest(&result_bytes)), + }, + }; + let envelope_bytes = serde_json::to_vec(&envelope).map_err(|_| ProfileError::Encoding)?; + if envelope_bytes.is_empty() || envelope_bytes.len() > MAX_ENVELOPE_BYTES { + return Err(ProfileError::LimitExceeded); + } + let signature_bytes = signature_document(key, &device_key_id, &envelope_bytes)?; + let decompressed = result_bytes + .len() + .checked_add(envelope_bytes.len()) + .and_then(|size| size.checked_add(signature_bytes.len())) + .ok_or(ProfileError::LimitExceeded)?; + if decompressed > MAX_DECOMPRESSED_BYTES { + return Err(ProfileError::LimitExceeded); + } + check_cancel(cancel)?; + if unix_now()? >= request.expires_at_unix { + return Err(ProfileError::Expired); + } + + let archive_bytes = archive(&envelope_bytes, &signature_bytes, &result_bytes)?; + if archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(ProfileError::LimitExceeded); + } + let archive_sha256 = hex_lower(&Sha256::digest(&archive_bytes)); + + Ok(SignedProfileExport { + artifact_uid: request.artifact_uid.clone(), + tool: result.tool_id, + outcome: result.outcome, + reason_code: result.reason_code, + archive_bytes, + archive_sha256, + }) +} + +pub fn save_signed_profile_export( + output: &Path, + export: &SignedProfileExport, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + let parent = output + .parent() + .filter(|path| !path.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let filename = output.file_name().ok_or(ProfileError::InvalidRequest)?.to_string_lossy(); + let temporary = parent.join(format!(".{filename}.{}.partial", export.artifact_uid)); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + + let mut file = options.open(&temporary).map_err(map_create_error)?; + let result = (|| { + file.write_all(&export.archive_bytes).map_err(ProfileError::Io)?; + check_cancel(cancel)?; + file.sync_all().map_err(ProfileError::Io)?; + check_cancel(cancel)?; + fs::hard_link(&temporary, output).map_err(map_publish_error)?; + if let Err(error) = fs::remove_file(&temporary) { + return Err(ProfileError::DurabilityAfterCommit(error)); + } + #[cfg(unix)] + if let Err(error) = File::open(parent).and_then(|directory| directory.sync_all()) { + return Err(ProfileError::DurabilityAfterCommit(error)); + } + Ok(SavedProfileExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: export.archive_bytes.len() as u64, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if result.is_err() { + let _ = fs::remove_file(&temporary); + } + result +} + +impl ProfileCaptureRequest { + pub(super) fn validate(&self, tool: ProfileTool, now_unix: i64) -> Result<(), ProfileError> { + if self.schema_version != PROFILE_SCHEMA_VERSION { + return Err(ProfileError::UnsupportedVersion); + } + if self.capability != tool.capability() { + return Err(ProfileError::UnsupportedCapability); + } + if !self.consent.confirmed || self.consent.policy_revision == 0 { + return Err(ProfileError::ConsentRequired); + } + if self.consent.expires_at_unix <= now_unix || self.expires_at_unix > self.consent.expires_at_unix { + return Err(ProfileError::ConsentExpired); + } + let validity = self + .expires_at_unix + .checked_sub(self.produced_at_unix) + .ok_or(ProfileError::Expired)?; + if self.produced_at_unix > now_unix.saturating_add(MAX_FUTURE_SKEW_SECONDS) + || validity <= 0 + || self.expires_at_unix <= now_unix + || validity > MAX_VALIDITY_SECONDS + { + return Err(ProfileError::Expired); + } + if self.duration.is_zero() + || self.duration > MAX_PROFILE_DURATION + || self.sample_period.is_zero() + || self.sample_period > self.duration + { + return Err(ProfileError::LimitExceeded); + } + if !uuid7(&self.run_uid) + || !uuid7(&self.artifact_uid) + || !uuid7(&self.consent.consent_uid) + || !resource_names_match(self) + || !lower_hex(&self.provenance.source_commit, 40) + || !lower_hex(&self.provenance.executable_sha256, 64) + || !version(&self.provenance.rustfs_version) + || self.provenance.build_features.len() > MAX_BUILD_FEATURES + || !self.provenance.build_features.iter().all(|feature| build_feature(feature)) + { + return Err(ProfileError::InvalidRequest); + } + Ok(()) + } +} + +pub(super) struct CollectorLease; + +impl CollectorLease { + pub(super) fn acquire() -> Result { + PROFILE_COLLECTOR_ACTIVE + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map(|_| Self) + .map_err(|_| ProfileError::Busy) + } +} + +impl Drop for CollectorLease { + fn drop(&mut self) { + PROFILE_COLLECTOR_ACTIVE.store(false, Ordering::Release); + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ProfileEnvelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: ProfileTool, + schema_version: u16, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: &'a str, + payload: ProfilePayload, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ProfilePayload { + path: &'static str, + media_type: &'static str, + size_bytes: u64, + sha256: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ProfileSignature<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +fn signature_document(key: &DeviceIdentity, key_id: &str, envelope: &[u8]) -> Result, ProfileError> { + let pkcs8 = key.to_pkcs8_der().map_err(|_| ProfileError::Signing)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| ProfileError::Signing)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + envelope.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.extend_from_slice(envelope); + let signature: Signature = signing_key.sign(&input); + let value = URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()); + serde_json::to_vec(&ProfileSignature { + algorithm: "ES256", + key_id, + value, + }) + .map_err(|_| ProfileError::Encoding) +} + +fn archive(envelope: &[u8], signature: &[u8], result: &[u8]) -> Result, ProfileError> { + let cursor = Cursor::new(Vec::with_capacity(envelope.len() + signature.len() + result.len() + 512)); + let mut writer = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(OUTPUT_MODE); + for (name, bytes) in [(ENVELOPE_PATH, envelope), (SIGNATURE_PATH, signature), (RESULT_PATH, result)] { + writer.start_file(name, options).map_err(|_| ProfileError::Encoding)?; + writer.write_all(bytes).map_err(ProfileError::Io)?; + } + writer + .finish() + .map(|cursor| cursor.into_inner()) + .map_err(|_| ProfileError::Encoding) +} + +fn resource_names_match(request: &ProfileCaptureRequest) -> bool { + let Some(organization_uid) = request.organization_name.strip_prefix("organizations/") else { + return false; + }; + if !uuid7(organization_uid) { + return false; + } + let cluster_prefix = format!("{}/clusters/", request.organization_name); + let Some(cluster_uid) = request.cluster_name.strip_prefix(&cluster_prefix) else { + return false; + }; + if !uuid7(cluster_uid) { + return false; + } + let device_prefix = format!("{}/clusterDevices/", request.cluster_name); + request.device_name.strip_prefix(&device_prefix).is_some_and(uuid7) +} + +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn lower_hex(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn build_feature(value: &str) -> bool { + value.len() <= 64 + && value.as_bytes().first().is_some_and(u8::is_ascii_lowercase) + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'_' | b'-')) +} + +fn version(value: &str) -> bool { + if value.is_empty() + || value.len() > 64 + || !value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'-')) + { + return false; + } + let (core, suffix) = value + .split_once('-') + .map_or((value, None), |(core, suffix)| (core, Some(suffix))); + if suffix.is_some_and(str::is_empty) { + return false; + } + let mut parts = core.split('.'); + parts.clone().count() == 3 && parts.all(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) +} + +fn timestamp(unix: i64) -> Result { + OffsetDateTime::from_unix_timestamp(unix) + .map_err(|_| ProfileError::InvalidRequest)? + .format(&Rfc3339) + .map_err(|_| ProfileError::InvalidRequest) +} + +pub(super) fn unix_now() -> Result { + let duration = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| ProfileError::InvalidRequest)?; + i64::try_from(duration.as_secs()).map_err(|_| ProfileError::InvalidRequest) +} + +pub(super) fn check_cancel(cancel: &CancellationToken) -> Result<(), ProfileError> { + if cancel.is_cancelled() { + Err(ProfileError::Cancelled) + } else { + Ok(()) + } +} + +fn hex_lower(bytes: &[u8]) -> String { + let mut value = String::with_capacity(bytes.len() * 2); + for byte in bytes { + use std::fmt::Write as _; + write!(&mut value, "{byte:02x}").expect("writing hexadecimal to a string cannot fail"); + } + value +} + +fn map_create_error(error: std::io::Error) -> ProfileError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + ProfileError::AlreadyExists + } else { + ProfileError::Io(error) + } +} + +fn map_publish_error(error: std::io::Error) -> ProfileError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + ProfileError::AlreadyExists + } else { + ProfileError::Io(error) + } +} diff --git a/rustfs/src/connect/diagnostics/profile_memory.rs b/rustfs/src/connect/diagnostics/profile_memory.rs new file mode 100644 index 000000000..0ff38a78e --- /dev/null +++ b/rustfs/src/connect/diagnostics/profile_memory.rs @@ -0,0 +1,163 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Mimalloc allocation aggregates for the bounded profile contract. +//! +//! Only cumulative allocated octets and allocation counts are sampled. Heap +//! bytes, addresses, stack traces, paths, symbols, and allocator debug text are +//! excluded from the result. + +use std::future::Future; +use std::pin::Pin; +use std::time::{Duration, Instant}; + +use serde_json::Value; +use tokio_util::sync::CancellationToken; + +use crate::connect::DeviceIdentity; + +use super::profile_cpu::{ + CollectorLease, MemoryProfileData, ProfileCaptureRequest, ProfileData, ProfileError, ProfileResult, ProfileTool, + SignedProfileExport, check_cancel, encode_signed_profile_export, unix_now, +}; + +const MAX_ALLOCATOR_STATS_BYTES: usize = 262_144; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct AllocationSnapshot { + total_allocated_bytes: u64, + allocation_count: u64, +} + +pub(crate) trait AllocationProfileSource: Send + Sync { + fn snapshot(&self) -> Result; +} + +struct MimallocProfileSource; + +impl AllocationProfileSource for MimallocProfileSource { + fn snapshot(&self) -> Result { + #[cfg(target_os = "windows")] + { + Err(ProfileError::SourceUnavailable) + } + #[cfg(not(target_os = "windows"))] + { + parse_allocator_stats(&rustfs_mimalloc::MiMalloc::stats_json()) + } + } +} + +pub async fn export_memory_profile( + request: &ProfileCaptureRequest, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + export_memory_profile_from(request, key, cancel, &MimallocProfileSource).await +} + +pub(crate) async fn export_memory_profile_from( + request: &ProfileCaptureRequest, + key: &DeviceIdentity, + cancel: &CancellationToken, + source: &dyn AllocationProfileSource, +) -> Result { + request.validate(ProfileTool::Memory, unix_now()?)?; + check_cancel(cancel)?; + let _lease = CollectorLease::acquire()?; + let started = Instant::now(); + let result = collect_window(request.sample_period, cancel, source); + let (before, after) = tokio::time::timeout(request.duration, result) + .await + .map_err(|_| ProfileError::TimedOut)??; + check_cancel(cancel)?; + + let allocated_bytes = after + .total_allocated_bytes + .checked_sub(before.total_allocated_bytes) + .ok_or(ProfileError::CounterReset)?; + let allocation_count = after + .allocation_count + .checked_sub(before.allocation_count) + .ok_or(ProfileError::CounterReset)?; + let elapsed = started.elapsed(); + let sample_period_micros = u64::try_from(elapsed.as_micros()).map_err(|_| ProfileError::LimitExceeded)?; + let data = MemoryProfileData::allocation_aggregates(allocated_bytes, allocation_count, sample_period_micros.max(1)); + let result = ProfileResult::succeeded(request, ProfileTool::Memory, elapsed, ProfileData::Memory(data)); + + encode_signed_profile_export(request, &result, key, cancel) +} + +fn collect_window<'a>( + sample_period: Duration, + cancel: &'a CancellationToken, + source: &'a dyn AllocationProfileSource, +) -> AllocationWindowFuture<'a> { + Box::pin(async move { + let before = source.snapshot()?; + tokio::select! { + () = cancel.cancelled() => return Err(ProfileError::Cancelled), + () = tokio::time::sleep(sample_period) => {} + } + let after = source.snapshot()?; + Ok((before, after)) + }) +} + +type AllocationWindowFuture<'a> = + Pin> + Send + 'a>>; + +pub(crate) fn parse_allocator_stats(stats: &str) -> Result { + if stats.is_empty() || stats.len() > MAX_ALLOCATOR_STATS_BYTES { + return Err(ProfileError::SourceUnavailable); + } + let value = serde_json::from_str::(stats).map_err(|_| ProfileError::SourceUnavailable)?; + let total_allocated_bytes = sum_metrics(&value, &["malloc_normal", "malloc_huge"], "total")?; + let allocation_count = sum_metrics(&value, &["malloc_normal_count", "malloc_huge_count"], "total")?; + Ok(AllocationSnapshot { + total_allocated_bytes, + allocation_count, + }) +} + +fn sum_metrics(value: &Value, metrics: &[&str], field: &str) -> Result { + metrics.iter().try_fold(0_u64, |sum, metric| { + let value = metric_field(value, metric, field).ok_or(ProfileError::SourceUnavailable)?; + sum.checked_add(value).ok_or(ProfileError::LimitExceeded) + }) +} + +fn metric_field(value: &Value, metric: &str, field: &str) -> Option { + match value { + Value::Object(fields) => { + if let Some(metric_value) = fields.get(metric) + && let Some(value) = numeric_field(metric_value, field) + { + return Some(value); + } + fields.values().find_map(|value| metric_field(value, metric, field)) + } + Value::Array(values) => values.iter().find_map(|value| metric_field(value, metric, field)), + _ => None, + } +} + +fn numeric_field(value: &Value, field: &str) -> Option { + match value { + Value::Number(number) => number.as_u64(), + Value::String(value) => value.parse().ok(), + Value::Object(fields) => fields.get(field).and_then(|value| numeric_field(value, field)), + _ => None, + } +} diff --git a/rustfs/src/connect/diagnostics/profile_threads.rs b/rustfs/src/connect/diagnostics/profile_threads.rs new file mode 100644 index 000000000..3fe15925d --- /dev/null +++ b/rustfs/src/connect/diagnostics/profile_threads.rs @@ -0,0 +1,200 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded thread-state profile collection. +//! +//! Linux native collection reads only the state byte from this process's +//! `/proc/self/task/*/stat` records. Thread names, identifiers, stacks, paths, +//! addresses, and raw procfs bytes cannot enter the exported result. Tokio +//! runtime state remains explicitly unsupported because Dial9 does not expose +//! the contract's RUNNABLE, WAITING, BLOCKED, and UNKNOWN counts. + +#[cfg(target_os = "linux")] +use std::fs::{self, File}; +#[cfg(target_os = "linux")] +use std::io::{ErrorKind, Read as _}; +#[cfg(target_os = "linux")] +use std::time::Instant; +use tokio_util::sync::CancellationToken; + +#[cfg(target_os = "linux")] +use super::profile_cpu::{CollectorLease, ProfileData, ThreadProfileData, ThreadState, ThreadStateCount}; +use super::profile_cpu::{ + ProfileCaptureRequest, ProfileError, ProfileReasonCode, ProfileResult, ProfileTool, SignedProfileExport, ThreadProfileScope, + check_cancel, encode_signed_profile_export, unix_now, +}; +use crate::connect::DeviceIdentity; + +#[cfg(target_os = "linux")] +const PROC_TASK_DIRECTORY: &str = "/proc/self/task"; +#[cfg(target_os = "linux")] +const MAX_NATIVE_THREADS: usize = 4_096; +#[cfg(target_os = "linux")] +const MAX_PROC_STAT_BYTES: u64 = 4_096; + +pub fn capture_thread_profile( + request: &ProfileCaptureRequest, + scope: ThreadProfileScope, + cancel: &CancellationToken, +) -> Result { + request.validate(ProfileTool::Threads, unix_now()?)?; + check_cancel(cancel)?; + + if scope == ThreadProfileScope::TokioRuntime { + return Ok(ProfileResult::unsupported( + request, + ProfileTool::Threads, + ProfileReasonCode::UnsupportedTool, + )); + } + + #[cfg(not(target_os = "linux"))] + return Ok(ProfileResult::unsupported( + request, + ProfileTool::Threads, + ProfileReasonCode::UnsupportedPlatform, + )); + + #[cfg(target_os = "linux")] + { + let _lease = CollectorLease::acquire()?; + let started = Instant::now(); + let deadline = started.checked_add(request.duration).ok_or(ProfileError::LimitExceeded)?; + let data = collect_native_thread_states(cancel, deadline)?; + check_cancel(cancel)?; + Ok(ProfileResult::succeeded( + request, + ProfileTool::Threads, + started.elapsed(), + ProfileData::Threads(data), + )) + } +} + +pub async fn export_thread_profile( + request: &ProfileCaptureRequest, + scope: ThreadProfileScope, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + let owned_request = request.clone(); + let owned_cancel = cancel.clone(); + let result = tokio::task::spawn_blocking(move || capture_thread_profile(&owned_request, scope, &owned_cancel)) + .await + .map_err(|_| ProfileError::CollectionFailed)??; + encode_signed_profile_export(request, &result, key, cancel) +} + +#[cfg(target_os = "linux")] +fn collect_native_thread_states(cancel: &CancellationToken, deadline: Instant) -> Result { + let entries = fs::read_dir(PROC_TASK_DIRECTORY).map_err(|_| ProfileError::SourceUnavailable)?; + let mut counts = [0_u64; 4]; + let mut visited = 0_usize; + + for entry in entries { + check_cancel(cancel)?; + if Instant::now() >= deadline { + return Err(ProfileError::TimedOut); + } + let entry = entry.map_err(|_| ProfileError::SourceUnavailable)?; + let name = entry.file_name(); + let Some(name) = name.to_str() else { + return Err(ProfileError::SourceUnavailable); + }; + if name.is_empty() || !name.bytes().all(|byte| byte.is_ascii_digit()) { + return Err(ProfileError::SourceUnavailable); + } + visited = visited.checked_add(1).ok_or(ProfileError::LimitExceeded)?; + if visited > MAX_NATIVE_THREADS { + return Err(ProfileError::LimitExceeded); + } + + let state = match read_proc_stat_state(&entry.path().join("stat"))? { + Some(state) => state, + None => continue, + }; + let index = match state { + ThreadState::Runnable => 0, + ThreadState::Waiting => 1, + ThreadState::Blocked => 2, + ThreadState::Unknown => 3, + }; + counts[index] = counts[index].checked_add(1).ok_or(ProfileError::LimitExceeded)?; + } + + if counts.iter().all(|count| *count == 0) { + return Err(ProfileError::SourceUnavailable); + } + + Ok(ThreadProfileData::native(vec![ + ThreadStateCount::new(ThreadState::Runnable, counts[0]), + ThreadStateCount::new(ThreadState::Waiting, counts[1]), + ThreadStateCount::new(ThreadState::Blocked, counts[2]), + ThreadStateCount::new(ThreadState::Unknown, counts[3]), + ])) +} + +#[cfg(target_os = "linux")] +fn read_proc_stat_state(path: &std::path::Path) -> Result, ProfileError> { + let file = match File::open(path) { + Ok(file) => file, + Err(error) if error.kind() == ErrorKind::NotFound => return Ok(None), + Err(_) => return Err(ProfileError::SourceUnavailable), + }; + let mut bytes = Vec::with_capacity(256); + file.take(MAX_PROC_STAT_BYTES + 1) + .read_to_end(&mut bytes) + .map_err(|_| ProfileError::SourceUnavailable)?; + if bytes.is_empty() || bytes.len() as u64 > MAX_PROC_STAT_BYTES { + return Err(ProfileError::SourceUnavailable); + } + parse_proc_stat_state(&bytes).map(Some) +} + +#[cfg(target_os = "linux")] +fn parse_proc_stat_state(stat: &[u8]) -> Result { + let closing = stat + .iter() + .rposition(|byte| *byte == b')') + .ok_or(ProfileError::SourceUnavailable)?; + let suffix = stat.get(closing + 1..).ok_or(ProfileError::SourceUnavailable)?; + let state = match suffix { + [b' ', state, b' ', ..] => *state, + _ => return Err(ProfileError::SourceUnavailable), + }; + Ok(match state { + b'R' => ThreadState::Runnable, + b'D' => ThreadState::Blocked, + b'S' | b'I' | b'T' | b't' | b'W' => ThreadState::Waiting, + _ => ThreadState::Unknown, + }) +} + +#[cfg(all(test, target_os = "linux"))] +mod tests { + use super::*; + + #[test] + fn proc_stat_parser_uses_only_the_kernel_state_byte() { + for (raw, expected) in [ + (b"123 (worker secret path) R 1 2".as_slice(), ThreadState::Runnable), + (b"123 (worker) S 1 2", ThreadState::Waiting), + (b"123 (worker) D 1 2", ThreadState::Blocked), + (b"123 (worker) Z 1 2", ThreadState::Unknown), + ] { + assert_eq!(parse_proc_stat_state(raw).expect("valid proc stat"), expected); + } + assert!(matches!(parse_proc_stat_state(b"123 malformed"), Err(ProfileError::SourceUnavailable))); + } +} diff --git a/rustfs/src/connect/diagnostics/receipt_delivery.rs b/rustfs/src/connect/diagnostics/receipt_delivery.rs new file mode 100644 index 000000000..f85cde976 --- /dev/null +++ b/rustfs/src/connect/diagnostics/receipt_delivery.rs @@ -0,0 +1,112 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::time::Duration; + +use serde::{Deserialize, Serialize}; + +use super::DiagnosticReceipt; +use crate::connect::config::HeartbeatConfig; +use crate::connect::heartbeat::HeartbeatError; +use crate::connect::telemetry::{TelemetryDelivery, TelemetryTransport}; + +const PROTOCOL_VERSION: &str = "v1"; + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct DiagnosticReceiptRequest<'a> { + protocol_version: &'static str, + request_id: &'a str, + #[serde(flatten)] + receipt: &'a DiagnosticReceipt, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct DiagnosticReceiptResponse { + accepted_version: String, +} + +pub(crate) enum DiagnosticReceiptDelivery { + Accepted, + Retry { retry_after: Option }, + AuthenticationStopped, + Rejected, +} + +pub(crate) struct DiagnosticReceiptSender { + transport: TelemetryTransport, +} + +impl DiagnosticReceiptSender { + pub(crate) fn new(config: HeartbeatConfig) -> Result { + Ok(Self { + transport: TelemetryTransport::new(config)?, + }) + } + + pub(crate) async fn send(&self, receipt: &DiagnosticReceipt) -> Result { + let request = DiagnosticReceiptRequest { + protocol_version: PROTOCOL_VERSION, + request_id: &receipt.receipt_id, + receipt, + }; + Ok(match self.transport.post("diagnosticExecutionReceipts", &request).await? { + TelemetryDelivery::Accepted { body, .. } => { + let response: DiagnosticReceiptResponse = serde_json::from_slice(&body).map_err(|_| HeartbeatError::Response)?; + if response.accepted_version != PROTOCOL_VERSION { + return Err(HeartbeatError::Response); + } + DiagnosticReceiptDelivery::Accepted + } + TelemetryDelivery::Retry { retry_after } => DiagnosticReceiptDelivery::Retry { retry_after }, + TelemetryDelivery::AuthenticationStopped { .. } => DiagnosticReceiptDelivery::AuthenticationStopped, + TelemetryDelivery::Rejected { .. } => DiagnosticReceiptDelivery::Rejected, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::connect::diagnostics::ReceiptOutcome; + + #[test] + fn request_is_the_versioned_frozen_receipt_with_receipt_id_as_idempotency_key() { + let receipt = DiagnosticReceipt { + receipt_id: "123e4567-e89b-42d3-a456-426614174001".to_owned(), + policy_revision: 8, + tool_id: "inventory.environment".to_owned(), + interval_started_at: "2030-01-01T00:00:00Z".to_owned(), + completed_at: "2030-01-01T00:00:07Z".to_owned(), + outcome: ReceiptOutcome::Failed, + attempt_count: 3, + reason: Some("connect_diagnostic_inventory_unavailable".to_owned()), + result_sha256: None, + result_bytes: None, + }; + let value = serde_json::to_value(DiagnosticReceiptRequest { + protocol_version: PROTOCOL_VERSION, + request_id: &receipt.receipt_id, + receipt: &receipt, + }) + .expect("receipt request"); + + assert_eq!(value["protocolVersion"], "v1"); + assert_eq!(value["requestId"], value["receiptId"]); + assert_eq!(value["attemptCount"], 3); + assert_eq!(value["outcome"], "FAILED"); + assert!(value.get("command").is_none()); + } +} diff --git a/rustfs/src/connect/diagnostics/schedule.rs b/rustfs/src/connect/diagnostics/schedule.rs new file mode 100644 index 000000000..59c4d686c --- /dev/null +++ b/rustfs/src/connect/diagnostics/schedule.rs @@ -0,0 +1,735 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Consent-bound scheduling for the small set of diagnostic producers compiled into RustFS. + +use std::fs; +use std::future::Future; +use std::io::{self, Write as _}; +use std::path::{Path, PathBuf}; +use std::pin::Pin; +use std::sync::Arc; +use std::time::Duration; + +use chrono::{DateTime, SecondsFormat, Utc}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use tokio::sync::{Mutex, watch}; +use tokio::task::JoinHandle; +use tokio_util::sync::CancellationToken; +use uuid::Uuid; + +use crate::connect::InventorySnapshot; +use crate::connect::environment::{ + ENVIRONMENT_CAPABILITY, ENVIRONMENT_SCHEMA_VERSION, EnvironmentCollectionRequest, EnvironmentInventory, collect_environment, +}; +use crate::connect::inventory::InventoryStateStore; + +const POLICY_CAPABILITY: &str = "diagnostics.policy.v1"; +const ENVIRONMENT_TOOL_ID: &str = "inventory.environment"; +const MIN_INTERVAL_SECONDS: u64 = 300; +const MAX_INTERVAL_SECONDS: u64 = 86_400; +const MAX_RETENTION_DAYS: u16 = 14; +const MAX_ATTEMPTS: u8 = 3; +const INITIAL_RETRY: Duration = Duration::from_secs(1); +const MAX_RETRY: Duration = Duration::from_secs(4); +const MAX_RESULT_BYTES: usize = 64 * 1024; +#[cfg(unix)] +const FILE_MODE: u32 = 0o600; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct DiagnosticCollectionPolicy { + policy_name: Option, + revision: u64, + support_state: SupportState, + desired_state: DesiredState, + reason_code: Option, + tool_ids: Vec, + interval_seconds: Option, + scope: Scope, + retention_days: Option, + consent_reference: Option, + consent_expires_at: Option, +} + +impl DiagnosticCollectionPolicy { + pub(crate) fn policy_sync_capability() -> &'static str { + POLICY_CAPABILITY + } + + pub(crate) fn stopped() -> Self { + Self { + policy_name: None, + revision: 0, + support_state: SupportState::Unsupported, + desired_state: DesiredState::Stopped, + reason_code: Some(ReasonCode::PolicyCapabilityUnsupported), + tool_ids: Vec::new(), + interval_seconds: None, + scope: Scope::Cluster, + retention_days: None, + consent_reference: None, + consent_expires_at: None, + } + } + + pub(crate) fn validate(&self) -> Result<(), DiagnosticScheduleError> { + if self.policy_name.as_ref().is_some_and(|value| value.len() > 512) + || self.tool_ids.len() > 19 + || self.tool_ids.iter().any(|value| value.is_empty() || value.len() > 64) + || self + .consent_reference + .as_ref() + .is_some_and(|value| value.is_empty() || value.len() > 512) + { + return Err(DiagnosticScheduleError::Policy); + } + if self.desired_state == DesiredState::Stopped { + return Ok(()); + } + let interval = self.interval_seconds.ok_or(DiagnosticScheduleError::Policy)?; + let retention = self.retention_days.ok_or(DiagnosticScheduleError::Policy)?; + let expires_at = self.consent_expiry()?; + if self.support_state != SupportState::Supported + || self.reason_code.is_some() + || !(MIN_INTERVAL_SECONDS..=MAX_INTERVAL_SECONDS).contains(&interval) + || !(1..=MAX_RETENTION_DAYS).contains(&retention) + || self.policy_name.is_none() + || self.consent_reference.is_none() + || self.tool_ids.is_empty() + || expires_at <= Utc::now() + { + return Err(DiagnosticScheduleError::Policy); + } + Ok(()) + } + + fn should_run(&self) -> bool { + self.support_state == SupportState::Supported && self.desired_state == DesiredState::Running + } + + fn consent_expiry(&self) -> Result, DiagnosticScheduleError> { + self.consent_expires_at + .as_deref() + .ok_or(DiagnosticScheduleError::Policy) + .and_then(parse_time) + } + + fn fingerprint(&self) -> Result { + let bytes = serde_json::to_vec(self).map_err(|_| DiagnosticScheduleError::Policy)?; + Ok(hex_simd::encode_to_string(Sha256::digest(bytes), hex_simd::AsciiCase::Lower)) + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum SupportState { + Supported, + Unsupported, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum DesiredState { + Running, + Stopped, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum ReasonCode { + PolicyCapabilityUnsupported, + ToolCapabilityUnsupported, + Disabled, + ConsentInactive, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum Scope { + Cluster, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct DiagnosticReceipt { + pub receipt_id: String, + pub policy_revision: u64, + pub tool_id: String, + pub interval_started_at: String, + pub completed_at: String, + pub outcome: ReceiptOutcome, + pub attempt_count: u8, + pub reason: Option, + pub result_sha256: Option, + pub result_bytes: Option, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum ReceiptOutcome { + Succeeded, + Failed, + Cancelled, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum DiagnosticScheduleStatus { + Waiting, + Running { + policy_revision: u64, + tool_id: String, + attempt: u8, + }, + Receipt(DiagnosticReceipt), + Failed { + reason: String, + }, + Stopped, +} + +#[derive(Debug, Error)] +pub enum DiagnosticScheduleError { + #[error("connect_diagnostic_policy_invalid")] + Policy, + #[error("connect_diagnostic_state_io")] + StateIo(#[source] io::Error), + #[error("connect_diagnostic_state_invalid")] + StateInvalid(#[source] serde_json::Error), + #[error("connect_diagnostic_state_corrupt")] + StateCorrupt, + #[error("connect_diagnostic_inventory_unavailable")] + InventoryUnavailable, + #[error("connect_diagnostic_collection_failed")] + CollectionFailed, + #[error("connect_diagnostic_result_too_large")] + ResultTooLarge, + #[error("connect_diagnostic_cancelled")] + Cancelled, +} + +#[derive(Clone, Debug, Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +struct ScheduleState { + policy_revision: Option, + policy_fingerprint: Option, + next_due_at: Option, + active_interval_started_at: Option, + last_receipt: Option, +} + +#[derive(Clone)] +struct StateStore { + path: PathBuf, + gate: Arc>, +} + +impl StateStore { + fn new(state_root: &Path) -> Self { + Self { + path: state_root.join("diagnostics/schedule.json"), + gate: Arc::new(Mutex::new(())), + } + } + + async fn read(&self) -> Result { + let _guard = self.gate.lock().await; + let path = self.path.clone(); + tokio::task::spawn_blocking(move || read_state(&path)) + .await + .map_err(|error| DiagnosticScheduleError::StateIo(io::Error::other(error)))? + } + + async fn write(&self, state: ScheduleState) -> Result<(), DiagnosticScheduleError> { + let _guard = self.gate.lock().await; + let path = self.path.clone(); + tokio::task::spawn_blocking(move || write_state(&path, &state)) + .await + .map_err(|error| DiagnosticScheduleError::StateIo(io::Error::other(error)))? + } +} + +type RunFuture = Pin, DiagnosticScheduleError>> + Send>>; +type Runner = Arc RunFuture + Send + Sync>; + +pub struct DiagnosticScheduleRuntime { + status: watch::Receiver, + receipts: watch::Receiver>, + task: JoinHandle<()>, +} + +impl DiagnosticScheduleRuntime { + pub fn status(&self) -> watch::Receiver { + self.status.clone() + } + + pub(crate) fn receipts(&self) -> watch::Receiver> { + self.receipts.clone() + } + + pub async fn shutdown(self) { + let _ = self.task.await; + } +} + +pub fn spawn_environment_schedule( + state_root: &Path, + policies: watch::Receiver, + shutdown: CancellationToken, +) -> Result { + let inventory_state_root = state_root.to_path_buf(); + let runner: Runner = Arc::new(move |cancel| { + let inventory_state_root = inventory_state_root.clone(); + Box::pin(async move { + let inventory = InventoryStateStore::from_state_root(&inventory_state_root) + .map_err(|_| DiagnosticScheduleError::InventoryUnavailable)?; + let result = run_environment_with_inventory(&inventory, &cancel).await?; + serde_json::to_vec(&result).map_err(|_| DiagnosticScheduleError::CollectionFailed) + }) + }); + Ok(spawn_schedule(StateStore::new(state_root), policies, shutdown, runner)) +} + +/// Run the only diagnostic producer currently compiled into RustFS once. +pub async fn run_local_environment_once( + inventory: &InventorySnapshot, + cancel: &CancellationToken, +) -> Result { + let request = + EnvironmentCollectionRequest::negotiate(ENVIRONMENT_SCHEMA_VERSION, ENVIRONMENT_CAPABILITY, Duration::from_secs(30)) + .map_err(|_| DiagnosticScheduleError::CollectionFailed)?; + collect_environment(inventory, request, cancel) + .await + .map_err(|error| match error { + crate::connect::EnvironmentError::Cancelled => DiagnosticScheduleError::Cancelled, + _ => DiagnosticScheduleError::CollectionFailed, + }) +} + +async fn run_environment_with_inventory( + inventory: &InventoryStateStore, + cancel: &CancellationToken, +) -> Result { + let persisted = inventory + .read_latest(Utc::now()) + .map_err(|_| DiagnosticScheduleError::InventoryUnavailable)?; + run_local_environment_once(&persisted.snapshot, cancel).await +} + +fn spawn_schedule( + store: StateStore, + mut policies: watch::Receiver, + shutdown: CancellationToken, + runner: Runner, +) -> DiagnosticScheduleRuntime { + let (status_tx, status_rx) = watch::channel(DiagnosticScheduleStatus::Waiting); + let (receipt_tx, receipt_rx) = watch::channel(None); + let task = tokio::spawn(async move { + if let Err(error) = run_collection_schedule(&store, &mut policies, &shutdown, &runner, &status_tx, &receipt_tx).await { + let _ = status_tx.send(DiagnosticScheduleStatus::Failed { + reason: error.to_string(), + }); + } + let _ = status_tx.send(DiagnosticScheduleStatus::Stopped); + }); + DiagnosticScheduleRuntime { + status: status_rx, + receipts: receipt_rx, + task, + } +} + +async fn run_collection_schedule( + store: &StateStore, + policies: &mut watch::Receiver, + shutdown: &CancellationToken, + runner: &Runner, + status: &watch::Sender, + receipts: &watch::Sender>, +) -> Result<(), DiagnosticScheduleError> { + let mut state = store.read().await?; + if state.last_receipt.is_some() { + let _ = receipts.send(state.last_receipt.clone()); + } + if let Some(started_at) = state.active_interval_started_at.take() { + let receipt = receipt( + state.policy_revision.unwrap_or_default(), + ENVIRONMENT_TOOL_ID, + started_at, + ReceiptOutcome::Failed, + 1, + Some("producer_restarted_during_collection".to_owned()), + None, + ); + state.last_receipt = Some(receipt.clone()); + store.write(state.clone()).await?; + let _ = status.send(DiagnosticScheduleStatus::Receipt(receipt)); + let _ = receipts.send(state.last_receipt.clone()); + } + + loop { + if shutdown.is_cancelled() { + break; + } + let policy = policies.borrow().clone(); + if let Err(error) = policy.validate() + && policy.should_run() + { + return Err(error); + } + if !policy.should_run() { + state.next_due_at = None; + state.active_interval_started_at = None; + store.write(state.clone()).await?; + let _ = status.send(DiagnosticScheduleStatus::Waiting); + if wait_for_policy(policies, shutdown).await { + break; + } + continue; + } + let fingerprint = policy.fingerprint()?; + match state.policy_revision { + Some(revision) if revision > policy.revision => { + if wait_for_policy(policies, shutdown).await { + break; + } + continue; + } + Some(revision) if revision == policy.revision && state.policy_fingerprint.as_deref() != Some(&fingerprint) => { + return Err(DiagnosticScheduleError::StateCorrupt); + } + Some(revision) if revision == policy.revision => {} + _ => { + state.policy_revision = Some(policy.revision); + state.policy_fingerprint = Some(fingerprint); + state.next_due_at = Some(now_string()); + store.write(state.clone()).await?; + } + } + let due = state + .next_due_at + .as_deref() + .ok_or(DiagnosticScheduleError::StateCorrupt) + .and_then(parse_time)?; + let expiry = policy.consent_expiry()?; + if due >= expiry { + state.next_due_at = None; + store.write(state.clone()).await?; + if wait_for_policy(policies, shutdown).await { + break; + } + continue; + } + let now = Utc::now(); + if due > now { + let wait = (due - now).to_std().map_err(|_| DiagnosticScheduleError::Policy)?; + tokio::select! { + biased; + () = shutdown.cancelled() => break, + changed = policies.changed() => if changed.is_err() { break; }, + () = tokio::time::sleep(wait) => {}, + } + continue; + } + let tool_id = match policy.tool_ids.as_slice() { + [tool_id] if tool_id == ENVIRONMENT_TOOL_ID => tool_id.clone(), + _ => return Err(DiagnosticScheduleError::Policy), + }; + let interval_started_at = due.to_rfc3339_opts(SecondsFormat::Secs, true); + state.active_interval_started_at = Some(interval_started_at.clone()); + state.next_due_at = Some( + (due + chrono::Duration::seconds(policy.interval_seconds.ok_or(DiagnosticScheduleError::Policy)? as i64)) + .to_rfc3339_opts(SecondsFormat::Secs, true), + ); + store.write(state.clone()).await?; + + let mut attempt = 1; + let mut retry = INITIAL_RETRY; + let receipt = loop { + let _ = status.send(DiagnosticScheduleStatus::Running { + policy_revision: policy.revision, + tool_id: tool_id.clone(), + attempt, + }); + let cancel = shutdown.child_token(); + let run = runner(cancel.clone()); + let result = tokio::select! { + biased; + () = shutdown.cancelled() => { + cancel.cancel(); + Err(DiagnosticScheduleError::Cancelled) + } + changed = policies.changed() => { + let _ = changed; + cancel.cancel(); + Err(DiagnosticScheduleError::Cancelled) + } + () = tokio::time::sleep_until(tokio::time::Instant::now() + expiry.signed_duration_since(Utc::now()).to_std().unwrap_or_default()) => { + cancel.cancel(); + Err(DiagnosticScheduleError::Cancelled) + } + result = run => result, + }; + match result { + Ok(bytes) if bytes.len() <= MAX_RESULT_BYTES => { + break receipt( + policy.revision, + &tool_id, + interval_started_at.clone(), + ReceiptOutcome::Succeeded, + attempt, + None, + Some(&bytes), + ); + } + Ok(_) => { + break receipt( + policy.revision, + &tool_id, + interval_started_at.clone(), + ReceiptOutcome::Failed, + attempt, + Some(DiagnosticScheduleError::ResultTooLarge.to_string()), + None, + ); + } + Err(DiagnosticScheduleError::Cancelled) => { + break receipt( + policy.revision, + &tool_id, + interval_started_at.clone(), + ReceiptOutcome::Cancelled, + attempt, + Some(DiagnosticScheduleError::Cancelled.to_string()), + None, + ); + } + Err(_error) if attempt < MAX_ATTEMPTS => { + tokio::select! { + biased; + () = shutdown.cancelled() => { + break receipt( + policy.revision, + &tool_id, + interval_started_at.clone(), + ReceiptOutcome::Cancelled, + attempt, + Some(DiagnosticScheduleError::Cancelled.to_string()), + None, + ); + } + changed = policies.changed() => { + let _ = changed; + break receipt( + policy.revision, + &tool_id, + interval_started_at.clone(), + ReceiptOutcome::Cancelled, + attempt, + Some(DiagnosticScheduleError::Cancelled.to_string()), + None, + ); + } + () = tokio::time::sleep(retry) => {} + } + attempt += 1; + retry = retry.saturating_mul(2).min(MAX_RETRY); + } + Err(error) => { + break receipt( + policy.revision, + &tool_id, + interval_started_at.clone(), + ReceiptOutcome::Failed, + attempt, + Some(error.to_string()), + None, + ); + } + } + }; + state.active_interval_started_at = None; + state.last_receipt = Some(receipt.clone()); + store.write(state.clone()).await?; + let _ = status.send(DiagnosticScheduleStatus::Receipt(receipt)); + let _ = receipts.send(state.last_receipt.clone()); + } + Ok(()) +} + +async fn wait_for_policy(policies: &mut watch::Receiver, shutdown: &CancellationToken) -> bool { + tokio::select! { + biased; + () = shutdown.cancelled() => true, + result = policies.changed() => result.is_err(), + } +} + +fn receipt( + revision: u64, + tool_id: &str, + interval_started_at: String, + outcome: ReceiptOutcome, + attempt_count: u8, + reason: Option, + bytes: Option<&[u8]>, +) -> DiagnosticReceipt { + DiagnosticReceipt { + receipt_id: Uuid::new_v4().to_string(), + policy_revision: revision, + tool_id: tool_id.to_owned(), + interval_started_at, + completed_at: now_string(), + outcome, + attempt_count, + reason, + result_sha256: bytes.map(|value| hex_simd::encode_to_string(Sha256::digest(value), hex_simd::AsciiCase::Lower)), + result_bytes: bytes.map(<[u8]>::len), + } +} + +fn now_string() -> String { + Utc::now().to_rfc3339_opts(SecondsFormat::Secs, true) +} + +fn parse_time(value: &str) -> Result, DiagnosticScheduleError> { + if value.len() != 20 || !value.ends_with('Z') { + return Err(DiagnosticScheduleError::Policy); + } + DateTime::parse_from_rfc3339(value) + .map(|value| value.with_timezone(&Utc)) + .map_err(|_| DiagnosticScheduleError::Policy) +} + +fn read_state(path: &Path) -> Result { + let bytes = match fs::read(path) { + Ok(bytes) => bytes, + Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(ScheduleState::default()), + Err(error) => return Err(DiagnosticScheduleError::StateIo(error)), + }; + if bytes.len() > MAX_RESULT_BYTES { + return Err(DiagnosticScheduleError::StateCorrupt); + } + serde_json::from_slice(&bytes).map_err(DiagnosticScheduleError::StateInvalid) +} + +fn write_state(path: &Path, state: &ScheduleState) -> Result<(), DiagnosticScheduleError> { + let bytes = serde_json::to_vec(state).map_err(DiagnosticScheduleError::StateInvalid)?; + let directory = path.parent().ok_or(DiagnosticScheduleError::StateCorrupt)?; + fs::create_dir_all(directory).map_err(DiagnosticScheduleError::StateIo)?; + let temp = path.with_extension(format!("tmp-{}", Uuid::new_v4())); + let mut options = fs::OpenOptions::new(); + options.create_new(true).write(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(FILE_MODE); + } + let mut file = options.open(&temp).map_err(DiagnosticScheduleError::StateIo)?; + let result = file + .write_all(&bytes) + .and_then(|()| file.sync_all()) + .and_then(|()| fs::rename(&temp, path)) + .map_err(DiagnosticScheduleError::StateIo); + if result.is_err() { + let _ = fs::remove_file(temp); + } + result +} + +#[cfg(test)] +mod tests { + use serde_json::json; + + use super::*; + + fn policy(revision: u64, running: bool) -> DiagnosticCollectionPolicy { + serde_json::from_value(json!({ + "policyName": if running { Some("organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/diagnosticCollectionPreference") } else { None }, + "revision": revision, + "supportState": "SUPPORTED", + "desiredState": if running { "RUNNING" } else { "STOPPED" }, + "reasonCode": if running { None::<&str> } else { Some("DISABLED") }, + "toolIds": if running { vec!["inventory.environment"] } else { vec![] }, + "intervalSeconds": if running { Some(300) } else { None }, + "scope": "CLUSTER", + "retentionDays": if running { Some(7) } else { None }, + "consentReference": if running { Some("organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/diagnosticCollectionConsents/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92") } else { None }, + "consentExpiresAt": if running { Some("2099-01-01T00:00:00Z") } else { None } + })) + .expect("policy") + } + + async fn wait_running(status: &mut watch::Receiver) { + tokio::time::timeout(Duration::from_secs(2), async { + while !matches!(&*status.borrow(), DiagnosticScheduleStatus::Running { .. }) { + status.changed().await.expect("schedule remains active"); + } + }) + .await + .expect("running"); + } + + fn blocking_runner() -> Runner { + Arc::new(|cancel| { + Box::pin(async move { + cancel.cancelled().await; + Err(DiagnosticScheduleError::Cancelled) + }) + }) + } + + #[tokio::test] + async fn disable_cancels_an_active_run_and_persists_receipt() { + let temp = tempfile::tempdir().expect("tempdir"); + let (policy_tx, policy_rx) = watch::channel(policy(1, true)); + let shutdown = CancellationToken::new(); + let mut runtime = spawn_schedule(StateStore::new(temp.path()), policy_rx, shutdown.clone(), blocking_runner()); + wait_running(&mut runtime.status).await; + policy_tx.send(policy(2, false)).expect("disable"); + tokio::time::timeout(Duration::from_secs(2), async { + loop { + let state = StateStore::new(temp.path()).read().await.expect("state"); + if state + .last_receipt + .is_some_and(|receipt| receipt.outcome == ReceiptOutcome::Cancelled) + { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + }) + .await + .expect("cancel receipt"); + let state = StateStore::new(temp.path()).read().await.expect("state"); + assert_eq!(state.last_receipt.expect("receipt").outcome, ReceiptOutcome::Cancelled); + shutdown.cancel(); + runtime.shutdown().await; + } + + #[tokio::test] + async fn shutdown_cancels_an_active_run_and_persists_receipt() { + let temp = tempfile::tempdir().expect("tempdir"); + let (_policy_tx, policy_rx) = watch::channel(policy(3, true)); + let shutdown = CancellationToken::new(); + let mut runtime = spawn_schedule(StateStore::new(temp.path()), policy_rx, shutdown.clone(), blocking_runner()); + wait_running(&mut runtime.status).await; + shutdown.cancel(); + runtime.shutdown().await; + let state = StateStore::new(temp.path()).read().await.expect("state"); + assert_eq!(state.last_receipt.expect("receipt").outcome, ReceiptOutcome::Cancelled); + } +} diff --git a/rustfs/src/connect/diagnostics/top_api.rs b/rustfs/src/connect/diagnostics/top_api.rs new file mode 100644 index 000000000..c89017e39 --- /dev/null +++ b/rustfs/src/connect/diagnostics/top_api.rs @@ -0,0 +1,1132 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Shared bounded result/export contract and the `top.api` producer. + +use std::fs::{self, File, OpenOptions}; +use std::io::{Cursor, Write as _}; +use std::path::Path; +use std::time::Duration; + +use base64_simd::URL_SAFE_NO_PAD; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use rand::{TryRng as _, rngs::SysRng}; +use rustfs_common::trace_bus::{ + TelemetryTraceEvent, TelemetryTraceOperation, TelemetryTraceStatus, subscribe_telemetry_trace_events, telemetry_trace_emit, + telemetry_trace_subscriber_count, +}; +use rustfs_io_metrics::install_s3_http_completion_observer; +use rustfs_s3_ops::S3Operation; +use serde::Serialize; +use sha2::{Digest as _, Sha256}; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio::sync::{Semaphore, SemaphorePermit}; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use crate::connect::identity::DeviceIdentity; + +pub const TOP_SCHEMA_VERSION: u8 = 1; +pub const TOP_API_CAPABILITY: &str = "top.api@1"; +pub const TOP_CLASSIFICATION: &str = "L3"; +pub const MAX_TOP_DURATION: Duration = Duration::from_secs(30); +pub const MAX_TOP_RESULT_BYTES: usize = 262_144; +pub const MAX_TOP_WORKING_MEMORY_BYTES: u64 = 67_108_864; +pub const MAX_TOP_RECORDS: u16 = 1_024; +pub const MAX_TOP_CPU_MILLIS: u64 = 5_000; +pub const MAX_TOP_TEMPORARY_BYTES: u64 = 2_097_152; +pub const MAX_TOP_TRAFFIC_BYTES: u64 = 1_048_576; +pub const MAX_TOP_BANDWIDTH_BYTES_PER_SECOND: u64 = 1_048_576; +pub const MAX_TOP_OPERATIONS: u16 = 1_024; +pub const MAX_SAFE_INTEGER: u64 = 9_007_199_254_740_991; + +const ENVELOPE_FORMAT: &str = "rustfs.connect.diagnosticEnvelope/1"; +const PROTOCOL_VERSION: &str = "v1"; +const SIGNATURE_ALGORITHM: &str = "ES256"; +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1"; +const MAX_ENVELOPE_BYTES: usize = 16_384; +const MAX_ARCHIVE_BYTES: usize = 524_288; +const MAX_DECOMPRESSED_BYTES: usize = 278_528; +pub const MAX_TOP_EXPORT_VALIDITY: Duration = Duration::from_secs(2_592_000); +const TOP_CAPTURE_WORKING_SET_BYTES: u64 = 1_048_576; +const ENVELOPE_PATH: &str = "envelope.json"; +const SIGNATURE_PATH: &str = "envelope.sig"; +const RESULT_PATH: &str = "result.json"; +const OUTPUT_MODE: u32 = 0o600; +static TOP_CAPTURE: Semaphore = Semaphore::const_new(1); + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct LocalTopConsent { + pub uid: String, + pub tool_id: String, + pub classification: String, + pub active: bool, + pub expires_at_unix: i64, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct TopCaptureScope { + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub policy_revision: u64, + pub run_expires_at_unix: i64, + pub executable_sha256: String, + pub build_features: Vec, + pub consent: LocalTopConsent, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct TopCaptureLimits { + pub max_duration_millis: u64, + pub max_result_bytes: usize, + pub max_working_memory_bytes: u64, + pub max_records: u16, + pub max_concurrency: u8, + pub max_cpu_millis: u64, + pub max_temporary_bytes: u64, + pub max_traffic_bytes: u64, + pub max_bandwidth_bytes_per_second: u64, + pub max_operations: u16, +} + +impl Default for TopCaptureLimits { + fn default() -> Self { + Self { + max_duration_millis: MAX_TOP_DURATION.as_millis() as u64, + max_result_bytes: MAX_TOP_RESULT_BYTES, + max_working_memory_bytes: MAX_TOP_WORKING_MEMORY_BYTES, + max_records: MAX_TOP_RECORDS, + max_concurrency: 1, + max_cpu_millis: MAX_TOP_CPU_MILLIS, + max_temporary_bytes: MAX_TOP_TEMPORARY_BYTES, + max_traffic_bytes: MAX_TOP_TRAFFIC_BYTES, + max_bandwidth_bytes_per_second: MAX_TOP_BANDWIDTH_BYTES_PER_SECOND, + max_operations: MAX_TOP_OPERATIONS, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct TopCaptureRequest { + pub scope: TopCaptureScope, + pub limits: TopCaptureLimits, + pub window: Duration, + pub export_validity: Duration, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum TopOutcome { + Succeeded, + Partial, + Failed, + Unsupported, + Cancelled, +} + +impl TopOutcome { + pub const fn as_str(self) -> &'static str { + match self { + Self::Succeeded => "SUCCEEDED", + Self::Partial => "PARTIAL", + Self::Failed => "FAILED", + Self::Unsupported => "UNSUPPORTED", + Self::Cancelled => "CANCELLED", + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum TopReasonCode { + Complete, + LimitExceeded, + SourceUnavailable, + UnsupportedTool, + UnsupportedPlatform, + Cancelled, + CounterReset, + CollectionFailed, +} + +impl TopReasonCode { + pub const fn as_str(self) -> &'static str { + match self { + Self::Complete => "COMPLETE", + Self::LimitExceeded => "LIMIT_EXCEEDED", + Self::SourceUnavailable => "SOURCE_UNAVAILABLE", + Self::UnsupportedTool => "UNSUPPORTED_TOOL", + Self::UnsupportedPlatform => "UNSUPPORTED_PLATFORM", + Self::Cancelled => "CANCELLED", + Self::CounterReset => "COUNTER_RESET", + Self::CollectionFailed => "COLLECTION_FAILED", + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TopProvenance { + repository: &'static str, + source_commit: String, + executable_sha256: String, + rustfs_version: &'static str, + os_family: &'static str, + architecture: &'static str, + build_features: Vec, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TopCoverage { + pub requested_units: u32, + pub completed_units: u32, + pub unit: &'static str, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TopResult { + pub schema_version: u8, + pub run_uid: String, + pub tool_id: &'static str, + pub capability: String, + pub outcome: TopOutcome, + pub reason_code: TopReasonCode, + pub duration_millis: u64, + pub provenance: TopProvenance, + pub coverage: TopCoverage, + pub data: Option, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum TopApiOperation { + GetObject, + PutObject, + HeadObject, + ListObjects, + InternalRpc, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TopApiData { + pub error_count: u64, + pub operation: TopApiOperation, + pub request_count: u64, + pub total_duration_micros: u64, + pub window_millis: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SignedTopExport { + pub artifact_uid: String, + pub archive_bytes: Vec, + pub archive_sha256: String, + pub envelope_json: Vec, + pub envelope_signature: Vec, + pub result_json: Vec, + pub result_sha256: String, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SavedTopExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +#[derive(Debug, PartialEq, Eq, thiserror::Error)] +pub enum TopCaptureError { + #[error("connect_top_consent_required")] + ConsentRequired, + #[error("connect_top_consent_expired")] + ConsentExpired, + #[error("connect_top_consent_scope_invalid")] + ConsentScope, + #[error("connect_top_scope_invalid")] + Scope, + #[error("connect_top_provenance_invalid")] + Provenance, + #[error("connect_top_limits_invalid")] + Limits, + #[error("connect_top_run_expired")] + Expired, + #[error("connect_top_result_invalid")] + Result, + #[error("connect_top_cancelled")] + Cancelled, + #[error("connect_top_result_too_large")] + ResultTooLarge, + #[error("connect_top_serialization_failed")] + Serialization, + #[error("connect_top_signing_failed")] + Signing, + #[error("connect_top_random_failed")] + Random, + #[error("connect_top_export_exists")] + AlreadyExists, + #[error("connect_top_io_failed: {0:?}")] + Io(std::io::ErrorKind), + #[error("connect_top_export_durability_failed_after_commit: {0:?}")] + DurabilityAfterCommit(std::io::ErrorKind), +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct DiagnosticEnvelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: &'static str, + schema_version: u8, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: String, + payload: DiagnosticPayload, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct DiagnosticPayload { + path: &'static str, + media_type: &'static str, + size_bytes: usize, + sha256: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct DiagnosticSignature { + algorithm: &'static str, + key_id: String, + value: String, +} + +pub async fn capture_top_api( + request: &TopCaptureRequest, + operation: TopApiOperation, + cancel: &CancellationToken, +) -> Result, TopCaptureError> { + const TOOL_ID: &str = "top.api"; + + request.validate_capture(TOOL_ID)?; + if cancel.is_cancelled() { + return request.cancelled(TOOL_ID); + } + let Some(_permit) = request.acquire(cancel).await? else { + return request.cancelled(TOOL_ID); + }; + install_top_api_s3_completion_observer(); + let mut subscription = subscribe_telemetry_trace_events(); + let source_operation = telemetry_operation(operation); + let started = tokio::time::Instant::now(); + let deadline = started + request.window; + let mut request_count = 0_u64; + let mut error_count = 0_u64; + let mut total_duration_micros = 0_u64; + + loop { + tokio::select! { + biased; + () = cancel.cancelled() => return request.cancelled(TOOL_ID), + () = tokio::time::sleep_until(deadline) => break, + received = subscription.recv() => match received { + Ok(event) if event.operation == source_operation => { + if request_count >= u64::from(request.limits.max_operations) + || request_count >= u64::from(request.limits.max_records) + { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::LimitExceeded); + } + let Ok(duration_micros) = u64::try_from(event.duration.as_micros()) else { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::CollectionFailed); + }; + let Some(next_duration) = total_duration_micros.checked_add(duration_micros) else { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::CollectionFailed); + }; + request_count += 1; + error_count += u64::from(event.status == TelemetryTraceStatus::Error); + total_duration_micros = next_duration; + } + Ok(_) => {} + Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::LimitExceeded); + } + Err(tokio::sync::broadcast::error::RecvError::Closed) => { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::SourceUnavailable); + } + } + } + } + + request.validate_scope(TOOL_ID)?; + if request_count > MAX_SAFE_INTEGER || error_count > MAX_SAFE_INTEGER || total_duration_micros > MAX_SAFE_INTEGER { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::CollectionFailed); + } + if request_count == 0 { + return request.unsupported(TOOL_ID, TopReasonCode::UnsupportedTool); + } + let window_millis = u64::try_from(request.window.as_millis()).map_err(|_| TopCaptureError::Limits)?; + request.succeeded( + TOOL_ID, + window_millis, + TopApiData { + operation, + request_count, + error_count, + window_millis, + total_duration_micros, + }, + ) +} + +fn install_top_api_s3_completion_observer() { + install_s3_http_completion_observer(top_api_trace_enabled, emit_s3_request_telemetry); +} + +fn top_api_trace_enabled() -> bool { + telemetry_trace_subscriber_count() != 0 +} + +fn emit_s3_request_telemetry(operation: S3Operation, duration: Duration, succeeded: bool) { + let Some(operation) = s3_telemetry_operation(operation) else { + return; + }; + let status = if succeeded { + TelemetryTraceStatus::Ok + } else { + TelemetryTraceStatus::Error + }; + telemetry_trace_emit(|| TelemetryTraceEvent::new(operation, duration, status)); +} + +const fn s3_telemetry_operation(operation: S3Operation) -> Option { + match operation { + S3Operation::GetObject => Some(TelemetryTraceOperation::GetObject), + S3Operation::PutObject => Some(TelemetryTraceOperation::PutObject), + S3Operation::HeadObject => Some(TelemetryTraceOperation::HeadObject), + S3Operation::ListObjects | S3Operation::ListObjectsV2 => Some(TelemetryTraceOperation::ListObjects), + _ => None, + } +} + +const fn telemetry_operation(operation: TopApiOperation) -> TelemetryTraceOperation { + match operation { + TopApiOperation::GetObject => TelemetryTraceOperation::GetObject, + TopApiOperation::PutObject => TelemetryTraceOperation::PutObject, + TopApiOperation::HeadObject => TelemetryTraceOperation::HeadObject, + TopApiOperation::ListObjects => TelemetryTraceOperation::ListObjects, + TopApiOperation::InternalRpc => TelemetryTraceOperation::InternalRpc, + } +} + +fn elapsed_millis(duration: Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).max(1) +} + +pub fn sign_top_export( + request: &TopCaptureRequest, + result: &TopResult, + identity: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + sign_top_export_inner(request, result, identity, cancel, None) +} + +pub(crate) fn sign_top_export_with_nonce( + request: &TopCaptureRequest, + result: &TopResult, + identity: &DeviceIdentity, + cancel: &CancellationToken, + nonce: [u8; 32], +) -> Result { + sign_top_export_inner(request, result, identity, cancel, Some(nonce)) +} + +fn sign_top_export_inner( + request: &TopCaptureRequest, + result: &TopResult, + identity: &DeviceIdentity, + cancel: &CancellationToken, + supplied_nonce: Option<[u8; 32]>, +) -> Result { + if !matches!(result.tool_id, "top.api" | "top.disk" | "top.locks" | "top.net" | "top.rpc") { + return Err(TopCaptureError::Result); + } + request.validate_scope(result.tool_id)?; + if cancel.is_cancelled() { + return Err(TopCaptureError::Cancelled); + } + if result.schema_version != TOP_SCHEMA_VERSION + || result.run_uid != request.scope.run_uid + || result.capability != format!("{}@{TOP_SCHEMA_VERSION}", result.tool_id) + || result.provenance != provenance(&request.scope)? + || !valid_export_outcome(result) + { + return Err(TopCaptureError::Result); + } + + let data = + serde_json::to_value(result.data.as_ref().ok_or(TopCaptureError::Result)?).map_err(|_| TopCaptureError::Serialization)?; + if !data.is_object() { + return Err(TopCaptureError::Result); + } + + let result_json = serde_json::to_vec(result).map_err(|_| TopCaptureError::Serialization)?; + if result_json.len() > request.limits.max_result_bytes || result_json.len() > MAX_TOP_RESULT_BYTES { + return Err(TopCaptureError::ResultTooLarge); + } + let result_sha256 = hex_lower(&Sha256::digest(&result_json)); + let now = OffsetDateTime::now_utc(); + let export_validity = time::Duration::try_from(request.export_validity).map_err(|_| TopCaptureError::Expired)?; + let requested_expiry = now + .checked_add(export_validity) + .ok_or(TopCaptureError::Expired)? + .unix_timestamp(); + let expires_at_unix = requested_expiry + .min(request.scope.run_expires_at_unix) + .min(request.scope.consent.expires_at_unix); + if expires_at_unix <= now.unix_timestamp() { + return Err(TopCaptureError::Expired); + } + + let nonce = if let Some(nonce) = supplied_nonce { + nonce + } else { + let mut nonce = [0u8; 32]; + SysRng.try_fill_bytes(&mut nonce).map_err(|_| TopCaptureError::Random)?; + nonce + }; + let device_key_id = hex_lower(&Sha256::digest(identity.public_key_der())); + let envelope = DiagnosticEnvelope { + format_version: ENVELOPE_FORMAT, + protocol_version: PROTOCOL_VERSION, + organization_name: &request.scope.organization_name, + cluster_name: &request.scope.cluster_name, + device_name: &request.scope.device_name, + run_uid: &request.scope.run_uid, + artifact_uid: &request.scope.artifact_uid, + tool_id: result.tool_id, + schema_version: TOP_SCHEMA_VERSION, + classification: TOP_CLASSIFICATION, + consent_uid: &request.scope.consent.uid, + policy_revision: request.scope.policy_revision, + produced_at: now + .replace_nanosecond(0) + .map_err(|_| TopCaptureError::Serialization)? + .format(&Rfc3339) + .map_err(|_| TopCaptureError::Serialization)?, + expires_at: OffsetDateTime::from_unix_timestamp(expires_at_unix) + .map_err(|_| TopCaptureError::Expired)? + .format(&Rfc3339) + .map_err(|_| TopCaptureError::Serialization)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(nonce), + device_key_id: device_key_id.clone(), + payload: DiagnosticPayload { + path: "result.json", + media_type: "application/json", + size_bytes: result_json.len(), + sha256: result_sha256.clone(), + }, + }; + let envelope_json = serde_json::to_vec(&envelope).map_err(|_| TopCaptureError::Serialization)?; + if envelope_json.len() > MAX_ENVELOPE_BYTES { + return Err(TopCaptureError::ResultTooLarge); + } + if cancel.is_cancelled() { + return Err(TopCaptureError::Cancelled); + } + + let key = identity.to_pkcs8_der().map_err(|_| TopCaptureError::Signing)?; + let signing_key = SigningKey::from_pkcs8_der(key.as_slice()).map_err(|_| TopCaptureError::Signing)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + 1 + envelope_json.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.push(0); + input.extend_from_slice(&envelope_json); + let signature: Signature = signing_key.sign(&input); + let signature = signature.normalize_s(); + let envelope_signature = serde_json::to_vec(&DiagnosticSignature { + algorithm: SIGNATURE_ALGORITHM, + key_id: device_key_id, + value: URL_SAFE_NO_PAD.encode_to_string(signature.to_bytes()), + }) + .map_err(|_| TopCaptureError::Serialization)?; + let decompressed_size = envelope_json + .len() + .checked_add(envelope_signature.len()) + .and_then(|size| size.checked_add(result_json.len())) + .ok_or(TopCaptureError::ResultTooLarge)?; + if decompressed_size > MAX_DECOMPRESSED_BYTES { + return Err(TopCaptureError::ResultTooLarge); + } + if cancel.is_cancelled() { + return Err(TopCaptureError::Cancelled); + } + let archive_bytes = archive(&envelope_json, &envelope_signature, &result_json)?; + if archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(TopCaptureError::ResultTooLarge); + } + let archive_sha256 = hex_lower(&Sha256::digest(&archive_bytes)); + + Ok(SignedTopExport { + artifact_uid: request.scope.artifact_uid.clone(), + archive_bytes, + archive_sha256, + envelope_json, + envelope_signature, + result_json, + result_sha256, + }) +} + +fn valid_export_outcome(result: &TopResult) -> bool { + if result.duration_millis == 0 + || result.duration_millis > MAX_TOP_DURATION.as_millis() as u64 + || result.coverage.unit != "WINDOW" + { + return false; + } + match result.outcome { + TopOutcome::Succeeded => { + result.reason_code == TopReasonCode::Complete + && result.data.is_some() + && result.coverage.requested_units == 1 + && result.coverage.completed_units == 1 + } + TopOutcome::Partial => { + matches!( + result.reason_code, + TopReasonCode::LimitExceeded | TopReasonCode::SourceUnavailable | TopReasonCode::CounterReset + ) && result.data.is_some() + && result.coverage.completed_units > 0 + && result.coverage.completed_units < result.coverage.requested_units + && result.coverage.requested_units <= 1_048_576 + } + TopOutcome::Failed | TopOutcome::Unsupported | TopOutcome::Cancelled => false, + } +} + +pub fn save_signed_top_export( + output: &Path, + export: &SignedTopExport, + cancel: &CancellationToken, +) -> Result { + if cancel.is_cancelled() { + return Err(TopCaptureError::Cancelled); + } + if !is_uuid_v7(&export.artifact_uid) { + return Err(TopCaptureError::Scope); + } + if export.archive_bytes.is_empty() + || export.archive_bytes.len() > MAX_ARCHIVE_BYTES + || hex_lower(&Sha256::digest(&export.archive_bytes)) != export.archive_sha256 + { + return Err(TopCaptureError::Result); + } + let parent = output + .parent() + .filter(|path| !path.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let filename = output.file_name().ok_or(TopCaptureError::Scope)?.to_string_lossy(); + let temporary = parent.join(format!(".{filename}.{}.partial", export.artifact_uid)); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + let mut file = options.open(&temporary).map_err(map_create_error)?; + let result = (|| { + file.write_all(&export.archive_bytes).map_err(io_error)?; + if cancel.is_cancelled() { + return Err(TopCaptureError::Cancelled); + } + file.sync_all().map_err(io_error)?; + if cancel.is_cancelled() { + return Err(TopCaptureError::Cancelled); + } + fs::hard_link(&temporary, output).map_err(map_publish_error)?; + if let Err(error) = fs::remove_file(&temporary) { + return Err(TopCaptureError::DurabilityAfterCommit(error.kind())); + } + #[cfg(unix)] + if let Err(error) = File::open(parent).and_then(|directory| directory.sync_all()) { + return Err(TopCaptureError::DurabilityAfterCommit(error.kind())); + } + Ok(SavedTopExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: export.archive_bytes.len() as u64, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if result.is_err() { + let _ = fs::remove_file(&temporary); + } + result +} + +impl TopCaptureRequest { + pub(crate) fn validate_capture(&self, tool_id: &'static str) -> Result<(), TopCaptureError> { + self.validate_scope(tool_id)?; + if self.window.is_zero() + || self.window > MAX_TOP_DURATION + || self.window.as_millis() > u128::from(self.limits.max_duration_millis) + { + return Err(TopCaptureError::Limits); + } + let remaining = self + .scope + .run_expires_at_unix + .min(self.scope.consent.expires_at_unix) + .checked_sub(unix_now()?) + .ok_or(TopCaptureError::Expired)?; + if remaining <= 0 || self.window.as_secs_f64().ceil() as i64 > remaining { + return Err(TopCaptureError::Expired); + } + Ok(()) + } + + pub(crate) fn validate_scope(&self, tool_id: &'static str) -> Result<(), TopCaptureError> { + let now = unix_now()?; + validate_resource_scope(&self.scope)?; + validate_limits(self.limits)?; + validate_provenance(&self.scope)?; + if !self.scope.consent.active { + return Err(TopCaptureError::ConsentRequired); + } + if self.scope.consent.classification != TOP_CLASSIFICATION || self.scope.consent.tool_id != tool_id { + return Err(TopCaptureError::ConsentScope); + } + if self.scope.consent.expires_at_unix <= now { + return Err(TopCaptureError::ConsentExpired); + } + if self.scope.run_expires_at_unix <= now { + return Err(TopCaptureError::Expired); + } + if self.export_validity.is_zero() || self.export_validity > MAX_TOP_EXPORT_VALIDITY { + return Err(TopCaptureError::Limits); + } + Ok(()) + } + + pub(crate) async fn acquire(&self, cancel: &CancellationToken) -> Result>, TopCaptureError> { + tokio::select! { + permit = TOP_CAPTURE.acquire() => Ok(Some(permit.map_err(|_| TopCaptureError::Limits)?)), + () = cancel.cancelled() => Ok(None), + } + } + + pub(crate) async fn wait_window(&self, tool_id: &'static str, cancel: &CancellationToken) -> Result { + tokio::select! { + () = tokio::time::sleep(self.window) => { + self.validate_scope(tool_id)?; + Ok(true) + } + () = cancel.cancelled() => Ok(false), + } + } + + pub(crate) fn succeeded( + &self, + tool_id: &'static str, + duration_millis: u64, + data: T, + ) -> Result, TopCaptureError> { + let result = TopResult { + schema_version: TOP_SCHEMA_VERSION, + run_uid: self.scope.run_uid.clone(), + tool_id, + capability: format!("{tool_id}@{TOP_SCHEMA_VERSION}"), + outcome: TopOutcome::Succeeded, + reason_code: TopReasonCode::Complete, + duration_millis, + provenance: provenance(&self.scope)?, + coverage: TopCoverage { + requested_units: 1, + completed_units: 1, + unit: "WINDOW", + }, + data: Some(data), + }; + self.bound_result(result) + } + + pub(crate) fn failed( + &self, + tool_id: &'static str, + duration_millis: u64, + reason_code: TopReasonCode, + ) -> Result, TopCaptureError> { + self.terminal(tool_id, duration_millis, TopOutcome::Failed, reason_code) + } + + pub(crate) fn unsupported( + &self, + tool_id: &'static str, + reason_code: TopReasonCode, + ) -> Result, TopCaptureError> { + self.terminal(tool_id, 0, TopOutcome::Unsupported, reason_code) + } + + pub(crate) fn cancelled(&self, tool_id: &'static str) -> Result, TopCaptureError> { + self.terminal(tool_id, 0, TopOutcome::Cancelled, TopReasonCode::Cancelled) + } + + fn terminal( + &self, + tool_id: &'static str, + duration_millis: u64, + outcome: TopOutcome, + reason_code: TopReasonCode, + ) -> Result, TopCaptureError> { + let result = TopResult { + schema_version: TOP_SCHEMA_VERSION, + run_uid: self.scope.run_uid.clone(), + tool_id, + capability: format!("{tool_id}@{TOP_SCHEMA_VERSION}"), + outcome, + reason_code, + duration_millis, + provenance: provenance(&self.scope)?, + coverage: TopCoverage { + requested_units: 1, + completed_units: 0, + unit: "WINDOW", + }, + data: None, + }; + self.bound_result(result) + } + + fn bound_result(&self, result: TopResult) -> Result, TopCaptureError> { + let size = serde_json::to_vec(&result).map_err(|_| TopCaptureError::Serialization)?.len(); + if size > self.limits.max_result_bytes || size > MAX_TOP_RESULT_BYTES { + return Err(TopCaptureError::ResultTooLarge); + } + Ok(result) + } +} + +fn validate_resource_scope(scope: &TopCaptureScope) -> Result<(), TopCaptureError> { + let organization_uid = scope + .organization_name + .strip_prefix("organizations/") + .filter(|tail| !tail.contains('/')) + .ok_or(TopCaptureError::Scope)?; + let cluster_prefix = format!("{}/clusters/", scope.organization_name); + let cluster_uid = scope + .cluster_name + .strip_prefix(&cluster_prefix) + .filter(|tail| !tail.contains('/')) + .ok_or(TopCaptureError::Scope)?; + let device_prefix = format!("{}/clusterDevices/", scope.cluster_name); + let device_uid = scope + .device_name + .strip_prefix(&device_prefix) + .filter(|tail| !tail.contains('/')) + .ok_or(TopCaptureError::Scope)?; + if scope.policy_revision == 0 + || ![ + organization_uid, + cluster_uid, + device_uid, + scope.run_uid.as_str(), + scope.artifact_uid.as_str(), + scope.consent.uid.as_str(), + ] + .into_iter() + .all(is_uuid_v7) + { + return Err(TopCaptureError::Scope); + } + Ok(()) +} + +fn validate_limits(limits: TopCaptureLimits) -> Result<(), TopCaptureError> { + if limits.max_duration_millis == 0 + || limits.max_duration_millis > MAX_TOP_DURATION.as_millis() as u64 + || limits.max_result_bytes == 0 + || limits.max_result_bytes > MAX_TOP_RESULT_BYTES + || limits.max_working_memory_bytes < TOP_CAPTURE_WORKING_SET_BYTES + || limits.max_working_memory_bytes > MAX_TOP_WORKING_MEMORY_BYTES + || limits.max_records == 0 + || limits.max_records > MAX_TOP_RECORDS + || limits.max_concurrency != 1 + || limits.max_cpu_millis == 0 + || limits.max_cpu_millis > MAX_TOP_CPU_MILLIS + || limits.max_temporary_bytes > MAX_TOP_TEMPORARY_BYTES + || limits.max_traffic_bytes > MAX_TOP_TRAFFIC_BYTES + || limits.max_bandwidth_bytes_per_second > MAX_TOP_BANDWIDTH_BYTES_PER_SECOND + || limits.max_operations == 0 + || limits.max_operations > MAX_TOP_OPERATIONS + { + return Err(TopCaptureError::Limits); + } + Ok(()) +} + +fn validate_provenance(scope: &TopCaptureScope) -> Result<(), TopCaptureError> { + if !lower_hex(&scope.executable_sha256, 64) + || scope.build_features.len() > 64 + || scope.build_features.iter().any(|feature| { + feature.is_empty() + || feature.len() > 64 + || !feature.as_bytes()[0].is_ascii_lowercase() + || !feature + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'_' || byte == b'-') + }) + { + return Err(TopCaptureError::Provenance); + } + let commit = crate::version::build::COMMIT_HASH; + if !lower_hex(commit, 40) { + return Err(TopCaptureError::Provenance); + } + Ok(()) +} + +fn provenance(scope: &TopCaptureScope) -> Result { + validate_provenance(scope)?; + let mut build_features = scope.build_features.clone(); + build_features.sort(); + build_features.dedup(); + Ok(TopProvenance { + repository: "rustfs/rustfs", + source_commit: crate::version::build::COMMIT_HASH.to_owned(), + executable_sha256: scope.executable_sha256.clone(), + rustfs_version: env!("CARGO_PKG_VERSION"), + os_family: match std::env::consts::OS { + "linux" => "LINUX", + "macos" => "DARWIN", + "windows" => "WINDOWS", + "freebsd" => "FREEBSD", + _ => "OTHER", + }, + architecture: match std::env::consts::ARCH { + "x86_64" => "x86_64", + "aarch64" => "aarch64", + _ => "other", + }, + build_features, + }) +} + +fn archive(envelope: &[u8], signature: &[u8], result: &[u8]) -> Result, TopCaptureError> { + let cursor = Cursor::new(Vec::with_capacity(envelope.len() + signature.len() + result.len() + 512)); + let mut writer = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(OUTPUT_MODE); + for (name, bytes) in [(ENVELOPE_PATH, envelope), (SIGNATURE_PATH, signature), (RESULT_PATH, result)] { + writer.start_file(name, options).map_err(|_| TopCaptureError::Serialization)?; + writer.write_all(bytes).map_err(io_error)?; + } + writer + .finish() + .map(|cursor| cursor.into_inner()) + .map_err(|_| TopCaptureError::Serialization) +} + +fn map_create_error(error: std::io::Error) -> TopCaptureError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + TopCaptureError::AlreadyExists + } else { + io_error(error) + } +} + +fn map_publish_error(error: std::io::Error) -> TopCaptureError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + TopCaptureError::AlreadyExists + } else { + io_error(error) + } +} + +fn io_error(error: std::io::Error) -> TopCaptureError { + TopCaptureError::Io(error.kind()) +} + +fn unix_now() -> Result { + Ok(OffsetDateTime::now_utc().unix_timestamp()) +} + +fn is_uuid_v7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn lower_hex(value: &str, len: usize) -> bool { + value.len() == len + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn hex_lower(bytes: &[u8]) -> String { + hex_simd::encode_to_string(bytes, hex_simd::AsciiCase::Lower) +} + +#[cfg(test)] +mod tests { + use super::*; + use rustfs_common::trace_bus::telemetry_trace_subscriber_count; + use rustfs_io_metrics::{record_s3_op, s3_http_metrics::S3HttpRequestGuard}; + use rustfs_s3_ops::S3Operation; + use serial_test::serial; + + fn capture_request(window: Duration) -> TopCaptureRequest { + let organization_uid = Uuid::now_v7(); + let cluster_uid = Uuid::now_v7(); + let device_uid = Uuid::now_v7(); + let organization_name = format!("organizations/{organization_uid}"); + let cluster_name = format!("{organization_name}/clusters/{cluster_uid}"); + let device_name = format!("{cluster_name}/clusterDevices/{device_uid}"); + let expires_at = OffsetDateTime::now_utc().unix_timestamp() + 60; + TopCaptureRequest { + scope: TopCaptureScope { + organization_name, + cluster_name, + device_name, + run_uid: Uuid::now_v7().to_string(), + artifact_uid: Uuid::now_v7().to_string(), + policy_revision: 1, + run_expires_at_unix: expires_at, + executable_sha256: "0".repeat(64), + build_features: Vec::new(), + consent: LocalTopConsent { + uid: Uuid::now_v7().to_string(), + tool_id: "top.api".to_owned(), + classification: TOP_CLASSIFICATION.to_owned(), + active: true, + expires_at_unix: expires_at, + }, + }, + limits: TopCaptureLimits::default(), + window, + export_validity: Duration::from_secs(60), + } + } + + async fn wait_for_subscription(previous: usize) { + tokio::time::timeout(Duration::from_secs(2), async { + while telemetry_trace_subscriber_count() <= previous { + tokio::task::yield_now().await; + } + }) + .await + .expect("top.api should subscribe to the runtime source"); + } + + async fn record_http_result(operation: S3Operation, status: u16) { + let mut guard = S3HttpRequestGuard::new("GET"); + guard.in_scope(|| record_s3_op(operation)); + tokio::task::yield_now().await; + guard.response(status); + } + + #[tokio::test] + #[serial] + async fn top_api_captures_real_s3_outcomes_and_excludes_other_operations() { + let request = capture_request(Duration::from_millis(25)); + let previous_subscribers = telemetry_trace_subscriber_count(); + let task_request = request.clone(); + let capture = + tokio::spawn( + async move { capture_top_api(&task_request, TopApiOperation::GetObject, &CancellationToken::new()).await }, + ); + wait_for_subscription(previous_subscribers).await; + + record_http_result(S3Operation::GetObject, 200).await; + record_http_result(S3Operation::GetObject, 503).await; + record_http_result(S3Operation::PutObject, 500).await; + + let result = capture.await.expect("capture task").expect("top.api result"); + assert_eq!(result.outcome, TopOutcome::Succeeded); + let data = result.data.as_ref().expect("successful capture data"); + assert_eq!(data.operation, TopApiOperation::GetObject); + assert_eq!(data.request_count, 2); + assert_eq!(data.error_count, 1); + assert!(data.total_duration_micros > 0); + let encoded = serde_json::to_value(data).expect("serialize top.api data"); + assert_eq!( + encoded.as_object().expect("top.api object").keys().collect::>(), + [ + "errorCount", + "operation", + "requestCount", + "totalDurationMicros", + "windowMillis" + ] + ); + let nonce = [7_u8; 32]; + let export = sign_top_export_with_nonce(&request, &result, &DeviceIdentity::generate(), &CancellationToken::new(), nonce) + .expect("signed top.api export"); + let envelope: serde_json::Value = serde_json::from_slice(&export.envelope_json).expect("diagnostic envelope"); + assert_eq!(envelope["nonce"], URL_SAFE_NO_PAD.encode_to_string(nonce)); + } + + #[tokio::test] + #[serial] + async fn top_api_cancellation_stops_without_exportable_data() { + let request = capture_request(Duration::from_secs(1)); + let cancel = CancellationToken::new(); + let task_cancel = cancel.clone(); + let previous_subscribers = telemetry_trace_subscriber_count(); + let capture = tokio::spawn(async move { capture_top_api(&request, TopApiOperation::GetObject, &task_cancel).await }); + wait_for_subscription(previous_subscribers).await; + cancel.cancel(); + + let result = capture.await.expect("capture task").expect("cancelled result"); + assert_eq!(result.outcome, TopOutcome::Cancelled); + assert_eq!(result.reason_code, TopReasonCode::Cancelled); + assert!(result.data.is_none()); + } + + #[tokio::test] + #[serial] + async fn top_api_refuses_to_publish_a_truncated_window() { + let mut request = capture_request(Duration::from_secs(1)); + request.limits.max_operations = 1; + request.limits.max_records = 1; + let previous_subscribers = telemetry_trace_subscriber_count(); + let capture = + tokio::spawn(async move { capture_top_api(&request, TopApiOperation::GetObject, &CancellationToken::new()).await }); + wait_for_subscription(previous_subscribers).await; + record_http_result(S3Operation::GetObject, 200).await; + record_http_result(S3Operation::GetObject, 200).await; + + let result = capture.await.expect("capture task").expect("limited result"); + assert_eq!(result.outcome, TopOutcome::Failed); + assert_eq!(result.reason_code, TopReasonCode::LimitExceeded); + assert!(result.data.is_none()); + } +} diff --git a/rustfs/src/connect/diagnostics/top_disk.rs b/rustfs/src/connect/diagnostics/top_disk.rs new file mode 100644 index 000000000..a506f85f2 --- /dev/null +++ b/rustfs/src/connect/diagnostics/top_disk.rs @@ -0,0 +1,130 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded process disk-I/O window backed by RustFS's existing process sampler. + +use serde::Serialize; +#[cfg(target_os = "linux")] +use tokio::time::Instant; +use tokio_util::sync::CancellationToken; + +use super::top_api::{MAX_SAFE_INTEGER, TopCaptureError, TopCaptureRequest, TopReasonCode, TopResult}; + +const TOOL_ID: &str = "top.disk"; +pub const TOP_DISK_CAPABILITY: &str = "top.disk@1"; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct DiskCounterSnapshot { + pub read_bytes: u64, + pub write_bytes: u64, + pub io_count: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TopDiskData { + pub resource_alias: &'static str, + pub read_bytes: u64, + pub write_bytes: u64, + pub io_count: u64, + pub window_millis: u64, +} + +pub async fn capture_top_disk( + request: &TopCaptureRequest, + cancel: &CancellationToken, +) -> Result, TopCaptureError> { + request.validate_capture(TOOL_ID)?; + if cancel.is_cancelled() { + return request.cancelled(TOOL_ID); + } + + #[cfg(not(target_os = "linux"))] + return request.unsupported(TOOL_ID, TopReasonCode::UnsupportedPlatform); + + #[cfg(target_os = "linux")] + { + let Some(_permit) = request.acquire(cancel).await? else { + return request.cancelled(TOOL_ID); + }; + let mut sampler = rustfs_io_metrics::ProcessSampler::new(); + let before = process_snapshot(&mut sampler)?; + let started = Instant::now(); + if !request.wait_window(TOOL_ID, cancel).await? { + return request.cancelled(TOOL_ID); + } + let after = process_snapshot(&mut sampler)?; + evaluate_disk_window(request, before, after, elapsed_millis(started.elapsed())) + } +} + +pub fn evaluate_disk_window( + request: &TopCaptureRequest, + before: DiskCounterSnapshot, + after: DiskCounterSnapshot, + window_millis: u64, +) -> Result, TopCaptureError> { + request.validate_scope(TOOL_ID)?; + if window_millis == 0 || window_millis > request.limits.max_duration_millis { + return Err(TopCaptureError::Limits); + } + let Some(read_bytes) = after.read_bytes.checked_sub(before.read_bytes) else { + return request.failed(TOOL_ID, window_millis, TopReasonCode::CollectionFailed); + }; + let Some(write_bytes) = after.write_bytes.checked_sub(before.write_bytes) else { + return request.failed(TOOL_ID, window_millis, TopReasonCode::CollectionFailed); + }; + let Some(io_count) = after.io_count.checked_sub(before.io_count) else { + return request.failed(TOOL_ID, window_millis, TopReasonCode::CollectionFailed); + }; + if [read_bytes, write_bytes, io_count] + .into_iter() + .any(|value| value > MAX_SAFE_INTEGER) + { + return request.failed(TOOL_ID, window_millis, TopReasonCode::CollectionFailed); + } + + request.succeeded( + TOOL_ID, + window_millis, + TopDiskData { + // The v1 contract excludes disk paths. This alias denotes the + // RustFS process-wide disk counter source for this one run. + resource_alias: "resource-1", + read_bytes, + write_bytes, + io_count, + window_millis, + }, + ) +} + +#[cfg(target_os = "linux")] +fn process_snapshot(sampler: &mut rustfs_io_metrics::ProcessSampler) -> Result { + let (_, snapshot) = sampler.snapshot_resource_and_system(); + let io_count = snapshot + .syscall_read_total + .checked_add(snapshot.syscall_write_total) + .ok_or(TopCaptureError::Result)?; + Ok(DiskCounterSnapshot { + read_bytes: snapshot.io_read_bytes, + write_bytes: snapshot.io_write_bytes, + io_count, + }) +} + +#[cfg(target_os = "linux")] +fn elapsed_millis(duration: std::time::Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).max(1) +} diff --git a/rustfs/src/connect/diagnostics/top_locks.rs b/rustfs/src/connect/diagnostics/top_locks.rs new file mode 100644 index 000000000..970d58b25 --- /dev/null +++ b/rustfs/src/connect/diagnostics/top_locks.rs @@ -0,0 +1,84 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Honest `top.locks` capability result. + +use serde::Serialize; +use tokio_util::sync::CancellationToken; + +use super::top_api::{MAX_SAFE_INTEGER, TopCaptureError, TopCaptureRequest, TopReasonCode, TopResult}; + +const TOOL_ID: &str = "top.locks"; +pub const TOP_LOCKS_CAPABILITY: &str = "top.locks@1"; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TopLocksData { + pub held_count: u64, + pub waiting_count: u64, + pub truncated: bool, +} + +pub async fn capture_top_locks( + request: &TopCaptureRequest, + cancel: &CancellationToken, +) -> Result, TopCaptureError> { + request.validate_capture(TOOL_ID)?; + if cancel.is_cancelled() { + return request.cancelled(TOOL_ID); + } + let Some(global_manager) = rustfs_lock::get_initialized_global_lock_manager() else { + return request.failed(TOOL_ID, 0, TopReasonCode::SourceUnavailable); + }; + let Some(manager) = global_manager.as_fast_lock_manager() else { + return request.unsupported(TOOL_ID, TopReasonCode::UnsupportedTool); + }; + let Some(_permit) = request.acquire(cancel).await? else { + return request.cancelled(TOOL_ID); + }; + if !request.wait_window(TOOL_ID, cancel).await? { + return request.cancelled(TOOL_ID); + } + let (held_count, waiting_count) = manager.current_lock_counts(); + // Report the admitted capture window. Scheduler wake-up jitter is not part + // of the measurement and must not turn an exactly bounded job into a + // LIMIT_EXCEEDED result. + let duration_millis = u64::try_from(request.window.as_millis()).unwrap_or(u64::MAX).max(1); + evaluate_lock_snapshot(request, held_count, waiting_count, duration_millis) +} + +pub fn evaluate_lock_snapshot( + request: &TopCaptureRequest, + held_count: u64, + waiting_count: u64, + duration_millis: u64, +) -> Result, TopCaptureError> { + request.validate_scope(TOOL_ID)?; + if duration_millis == 0 || duration_millis > request.limits.max_duration_millis { + return Err(TopCaptureError::Limits); + } + if held_count > MAX_SAFE_INTEGER || waiting_count > MAX_SAFE_INTEGER { + return request.failed(TOOL_ID, duration_millis, TopReasonCode::CollectionFailed); + } + + request.succeeded( + TOOL_ID, + duration_millis, + TopLocksData { + held_count, + waiting_count, + truncated: false, + }, + ) +} diff --git a/rustfs/src/connect/diagnostics/top_net.rs b/rustfs/src/connect/diagnostics/top_net.rs new file mode 100644 index 000000000..455ef1c74 --- /dev/null +++ b/rustfs/src/connect/diagnostics/top_net.rs @@ -0,0 +1,115 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded host-network window backed by a persistent OS interface snapshot. + +use serde::Serialize; +use sysinfo::Networks; +use tokio::time::Instant; +use tokio_util::sync::CancellationToken; + +use super::top_api::{MAX_SAFE_INTEGER, TopCaptureError, TopCaptureRequest, TopReasonCode, TopResult}; + +const TOOL_ID: &str = "top.net"; +pub const TOP_NET_CAPABILITY: &str = "top.net@1"; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct NetworkCounterSnapshot { + pub received_bytes: u64, + pub sent_bytes: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TopNetData { + pub received_bytes: u64, + pub sent_bytes: u64, + pub window_millis: u64, +} + +pub async fn capture_top_net( + request: &TopCaptureRequest, + cancel: &CancellationToken, +) -> Result, TopCaptureError> { + request.validate_capture(TOOL_ID)?; + if cancel.is_cancelled() { + return request.cancelled(TOOL_ID); + } + let Some(_permit) = request.acquire(cancel).await? else { + return request.cancelled(TOOL_ID); + }; + if !sysinfo::IS_SUPPORTED_SYSTEM { + return request.failed(TOOL_ID, 0, TopReasonCode::SourceUnavailable); + } + let mut networks = Networks::new(); + networks.refresh(true); + if networks.is_empty() { + return request.failed(TOOL_ID, 0, TopReasonCode::SourceUnavailable); + } + let started = Instant::now(); + if !request.wait_window(TOOL_ID, cancel).await? { + return request.cancelled(TOOL_ID); + } + let observed = rustfs_obs::metrics::stats_collector::collect_host_network_stats(&mut networks); + evaluate_network_window( + request, + NetworkCounterSnapshot { + received_bytes: 0, + sent_bytes: 0, + }, + NetworkCounterSnapshot { + received_bytes: observed.total_received, + sent_bytes: observed.total_transmitted, + }, + elapsed_millis(started.elapsed()), + ) +} + +pub fn evaluate_network_window( + request: &TopCaptureRequest, + before: NetworkCounterSnapshot, + after: NetworkCounterSnapshot, + window_millis: u64, +) -> Result, TopCaptureError> { + request.validate_scope(TOOL_ID)?; + if window_millis == 0 || window_millis > request.limits.max_duration_millis { + return Err(TopCaptureError::Limits); + } + let Some(received_bytes) = after.received_bytes.checked_sub(before.received_bytes) else { + return request.failed(TOOL_ID, window_millis, TopReasonCode::CollectionFailed); + }; + let Some(sent_bytes) = after.sent_bytes.checked_sub(before.sent_bytes) else { + return request.failed(TOOL_ID, window_millis, TopReasonCode::CollectionFailed); + }; + if received_bytes > MAX_SAFE_INTEGER || sent_bytes > MAX_SAFE_INTEGER { + return request.failed(TOOL_ID, window_millis, TopReasonCode::CollectionFailed); + } + if received_bytes == 0 && sent_bytes == 0 { + return request.failed(TOOL_ID, window_millis, TopReasonCode::SourceUnavailable); + } + + request.succeeded( + TOOL_ID, + window_millis, + TopNetData { + received_bytes, + sent_bytes, + window_millis, + }, + ) +} + +fn elapsed_millis(duration: std::time::Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).max(1) +} diff --git a/rustfs/src/connect/diagnostics/top_rpc.rs b/rustfs/src/connect/diagnostics/top_rpc.rs new file mode 100644 index 000000000..25d5dc58e --- /dev/null +++ b/rustfs/src/connect/diagnostics/top_rpc.rs @@ -0,0 +1,112 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded `top.rpc` aggregation over pre-classified internode HTTP RPC completions. + +use std::time::Duration; + +use rustfs_common::trace_bus::{TelemetryTraceOperation, TelemetryTraceStatus, subscribe_telemetry_trace_events}; +use serde::Serialize; +use tokio_util::sync::CancellationToken; + +use super::top_api::{MAX_SAFE_INTEGER, TopCaptureError, TopCaptureRequest, TopReasonCode, TopResult}; + +const TOOL_ID: &str = "top.rpc"; +pub const TOP_RPC_CAPABILITY: &str = "top.rpc@1"; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TopRpcData { + pub request_count: u64, + pub error_count: u64, + pub window_millis: u64, + pub total_duration_micros: u64, +} + +pub async fn capture_top_rpc( + request: &TopCaptureRequest, + cancel: &CancellationToken, +) -> Result, TopCaptureError> { + request.validate_capture(TOOL_ID)?; + if cancel.is_cancelled() { + return request.cancelled(TOOL_ID); + } + let Some(_permit) = request.acquire(cancel).await? else { + return request.cancelled(TOOL_ID); + }; + // The typed source is emitted only after an internode HTTP response has a + // final status. Tonic streams need a separate body-completion boundary. + let mut subscription = subscribe_telemetry_trace_events(); + let started = tokio::time::Instant::now(); + let deadline = started + request.window; + let mut request_count = 0_u64; + let mut error_count = 0_u64; + let mut total_duration_micros = 0_u64; + + loop { + tokio::select! { + biased; + () = cancel.cancelled() => return request.cancelled(TOOL_ID), + () = tokio::time::sleep_until(deadline) => break, + received = subscription.recv() => match received { + Ok(event) if event.operation == TelemetryTraceOperation::InternalRpc => { + if request_count >= u64::from(request.limits.max_operations) + || request_count >= u64::from(request.limits.max_records) + { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::LimitExceeded); + } + let Ok(duration_micros) = u64::try_from(event.duration.as_micros()) else { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::CollectionFailed); + }; + let Some(next_duration) = total_duration_micros.checked_add(duration_micros) else { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::CollectionFailed); + }; + request_count += 1; + error_count += u64::from(event.status == TelemetryTraceStatus::Error); + total_duration_micros = next_duration; + } + Ok(_) => {} + Err(tokio::sync::broadcast::error::RecvError::Lagged(_)) => { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::LimitExceeded); + } + Err(tokio::sync::broadcast::error::RecvError::Closed) => { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::SourceUnavailable); + } + } + } + } + + request.validate_scope(TOOL_ID)?; + if request_count > MAX_SAFE_INTEGER || error_count > MAX_SAFE_INTEGER || total_duration_micros > MAX_SAFE_INTEGER { + return request.failed(TOOL_ID, elapsed_millis(started.elapsed()), TopReasonCode::CollectionFailed); + } + if request_count == 0 { + return request.unsupported(TOOL_ID, TopReasonCode::UnsupportedTool); + } + let window_millis = u64::try_from(request.window.as_millis()).map_err(|_| TopCaptureError::Limits)?; + request.succeeded( + TOOL_ID, + window_millis, + TopRpcData { + request_count, + error_count, + window_millis, + total_duration_micros, + }, + ) +} + +fn elapsed_millis(duration: Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).max(1) +} diff --git a/rustfs/src/connect/diagnostics/trace_analysis.rs b/rustfs/src/connect/diagnostics/trace_analysis.rs new file mode 100644 index 000000000..8db444d04 --- /dev/null +++ b/rustfs/src/connect/diagnostics/trace_analysis.rs @@ -0,0 +1,149 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Deterministic local summaries over redacted telemetry spans. + +use serde::{Deserialize, Serialize}; +use thiserror::Error; +use tokio_util::sync::CancellationToken; + +use super::trace_record::{ + LocalTelemetryConsent, TelemetryOperation, TelemetryProducerError, TelemetrySpanStatus, acquire_telemetry_lease, +}; +use super::trace_replay::ReplayedTrace; + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct TraceAnalysis { + pub input_artifact_sha256: String, + pub span_count: u64, + pub error_count: u64, + pub total_duration_micros: u64, + pub operations: Vec, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct OperationSummary { + pub operation: TelemetryOperation, + pub span_count: u64, + pub error_count: u64, + pub total_duration_micros: u64, +} + +#[derive(Debug, Error, Eq, PartialEq)] +pub enum TraceAnalysisError { + #[error("another telemetry operation is already running")] + Busy, + #[error("local telemetry consent is expired")] + ConsentExpired, + #[error("telemetry analysis was cancelled")] + Cancelled, + #[error("telemetry analysis counters overflowed")] + CounterOverflow, +} + +pub fn analyze_trace( + replay: &ReplayedTrace, + consent: LocalTelemetryConsent, + cancel: &CancellationToken, +) -> Result { + consent.remaining().map_err(map_consent_error)?; + if cancel.is_cancelled() { + return Err(TraceAnalysisError::Cancelled); + } + let _lease = acquire_telemetry_lease().map_err(map_consent_error)?; + + let mut operations = [ + OperationSummary { + operation: TelemetryOperation::GetObject, + span_count: 0, + error_count: 0, + total_duration_micros: 0, + }, + OperationSummary { + operation: TelemetryOperation::PutObject, + span_count: 0, + error_count: 0, + total_duration_micros: 0, + }, + OperationSummary { + operation: TelemetryOperation::HeadObject, + span_count: 0, + error_count: 0, + total_duration_micros: 0, + }, + OperationSummary { + operation: TelemetryOperation::ListObjects, + span_count: 0, + error_count: 0, + total_duration_micros: 0, + }, + OperationSummary { + operation: TelemetryOperation::InternalRpc, + span_count: 0, + error_count: 0, + total_duration_micros: 0, + }, + ]; + let mut error_count = 0u64; + let mut total_duration_micros = 0u64; + for span in &replay.spans { + if cancel.is_cancelled() { + return Err(TraceAnalysisError::Cancelled); + } + let summary = &mut operations[operation_index(span.operation)]; + summary.span_count = summary.span_count.checked_add(1).ok_or(TraceAnalysisError::CounterOverflow)?; + summary.total_duration_micros = summary + .total_duration_micros + .checked_add(span.duration_micros) + .ok_or(TraceAnalysisError::CounterOverflow)?; + total_duration_micros = total_duration_micros + .checked_add(span.duration_micros) + .ok_or(TraceAnalysisError::CounterOverflow)?; + if span.status == TelemetrySpanStatus::Error { + summary.error_count = summary + .error_count + .checked_add(1) + .ok_or(TraceAnalysisError::CounterOverflow)?; + error_count = error_count.checked_add(1).ok_or(TraceAnalysisError::CounterOverflow)?; + } + } + let span_count = u64::try_from(replay.spans.len()).map_err(|_| TraceAnalysisError::CounterOverflow)?; + Ok(TraceAnalysis { + input_artifact_sha256: replay.input_artifact_sha256.clone(), + span_count, + error_count, + total_duration_micros, + operations: operations.into_iter().filter(|summary| summary.span_count != 0).collect(), + }) +} + +const fn operation_index(operation: TelemetryOperation) -> usize { + match operation { + TelemetryOperation::GetObject => 0, + TelemetryOperation::PutObject => 1, + TelemetryOperation::HeadObject => 2, + TelemetryOperation::ListObjects => 3, + TelemetryOperation::InternalRpc => 4, + } +} + +fn map_consent_error(error: TelemetryProducerError) -> TraceAnalysisError { + match error { + TelemetryProducerError::Busy => TraceAnalysisError::Busy, + TelemetryProducerError::ConsentExpired => TraceAnalysisError::ConsentExpired, + _ => TraceAnalysisError::CounterOverflow, + } +} diff --git a/rustfs/src/connect/diagnostics/trace_otlp.rs b/rustfs/src/connect/diagnostics/trace_otlp.rs new file mode 100644 index 000000000..510310b87 --- /dev/null +++ b/rustfs/src/connect/diagnostics/trace_otlp.rs @@ -0,0 +1,271 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Customer-side OTLP/HTTP forwarding with bounded payloads and local secrets. + +use std::time::{Duration, Instant}; + +use opentelemetry_proto::tonic::collector::trace::v1::ExportTraceServiceRequest; +use prost::Message as _; +use reqwest::{ + Client, StatusCode, Url, + header::{self, HeaderMap}, +}; +use serde::{Deserialize, Serialize}; +use thiserror::Error; +use tokio_util::sync::CancellationToken; + +use super::trace_record::{ + LocalTelemetryConsent, MAX_TELEMETRY_DURATION, TelemetryArtifactRequest, TelemetryDiagnosticResult, TelemetryLease, + TelemetryProducerError, TelemetryTool, acquire_telemetry_lease, ensure_result_size, +}; + +pub const MAX_OTLP_BODY_BYTES: usize = 1_048_576; + +/// An encoded OTLP/HTTP protobuf batch from the local OpenTelemetry adapter. +/// The bytes are never included in the diagnostic result or error values. +pub struct OtlpBatch { + body: Vec, + span_count: u64, + _lease: TelemetryLease, +} + +impl OtlpBatch { + pub fn new(body: Vec) -> Result { + if body.is_empty() || body.len() > MAX_OTLP_BODY_BYTES { + return Err(OtlpForwardError::InvalidBatch); + } + let lease = acquire_telemetry_lease().map_err(map_producer_error)?; + let request = ExportTraceServiceRequest::decode(body.as_slice()).map_err(|_| OtlpForwardError::InvalidBatch)?; + if request.resource_spans.len() > 1024 { + return Err(OtlpForwardError::InvalidBatch); + } + let scope_count = request.resource_spans.iter().try_fold(0_usize, |count, resource| { + count + .checked_add(resource.scope_spans.len()) + .filter(|count| *count <= 1024) + .ok_or(OtlpForwardError::InvalidBatch) + })?; + if scope_count == 0 { + return Err(OtlpForwardError::InvalidBatch); + } + let span_count = request + .resource_spans + .iter() + .flat_map(|resource| &resource.scope_spans) + .try_fold(0_u64, |count, scope| { + let spans = u64::try_from(scope.spans.len()).map_err(|_| OtlpForwardError::InvalidBatch)?; + count.checked_add(spans).ok_or(OtlpForwardError::InvalidBatch) + })?; + if span_count == 0 || span_count > 1024 { + return Err(OtlpForwardError::InvalidBatch); + } + Ok(Self { + body, + span_count, + _lease: lease, + }) + } +} + +pub async fn export_trace_otlp_result( + request: &TelemetryArtifactRequest, + endpoint: Url, + headers: LocalOtlpHeaders, + batch: OtlpBatch, + consent: LocalTelemetryConsent, + timeout: Duration, + cancel: &CancellationToken, +) -> Result, OtlpForwardError> { + let started = Instant::now(); + let receipt = export_trace_otlp(endpoint, headers, batch, consent, timeout, cancel).await?; + Ok(TelemetryDiagnosticResult::succeeded( + request, + TelemetryTool::Otlp, + started.elapsed(), + receipt, + )) +} + +/// Locally supplied collector authentication. This type has no `Debug` or +/// serialization implementation, keeping credentials out of results and logs. +pub struct LocalOtlpHeaders(HeaderMap); + +impl LocalOtlpHeaders { + pub fn new(headers: HeaderMap) -> Self { + Self(headers) + } +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct OtlpReceipt { + pub accepted_span_count: u64, + pub rejected_span_count: u64, + pub exported_bytes: u64, +} + +#[derive(Debug, Error, Eq, PartialEq)] +pub enum OtlpForwardError { + #[error("another telemetry operation is already running")] + Busy, + #[error("OTLP endpoint must be HTTPS or an HTTP loopback address without credentials, query or fragment")] + Endpoint, + #[error("OTLP batch must contain 1..1024 spans and 1..1048576 bytes")] + InvalidBatch, + #[error("OTLP forward timeout must be between 1ms and 30s")] + InvalidTimeout, + #[error("OTLP forwarding was cancelled")] + Cancelled, + #[error("local OTLP collector rejected the batch")] + Rejected, + #[error("local OTLP collector is unavailable")] + Unavailable, + #[error("OTLP receipt exceeds its result limit")] + ResultTooLarge, + #[error("local telemetry consent is expired")] + ConsentExpired, +} + +pub async fn export_trace_otlp( + endpoint: Url, + headers: LocalOtlpHeaders, + batch: OtlpBatch, + consent: LocalTelemetryConsent, + timeout: Duration, + cancel: &CancellationToken, +) -> Result { + validate_endpoint(&endpoint)?; + if timeout.is_zero() || timeout > MAX_TELEMETRY_DURATION { + return Err(OtlpForwardError::InvalidTimeout); + } + let remaining = consent.remaining().map_err(map_consent_error)?; + let consent_binds = remaining <= timeout; + let timeout = timeout.min(remaining); + let timeout_millis = usize::try_from(timeout.as_millis()).map_err(|_| OtlpForwardError::InvalidTimeout)?; + if timeout_millis == 0 { + return Err(OtlpForwardError::InvalidTimeout); + } + let deadline = tokio::time::Instant::now() + timeout; + let OtlpBatch { + body, + span_count, + _lease, + } = batch; + let byte_budget = MAX_OTLP_BODY_BYTES + .checked_mul(timeout_millis.min(1000)) + .and_then(|bytes| bytes.checked_div(1000)) + .ok_or(OtlpForwardError::InvalidBatch)?; + if body.len() > byte_budget { + return Err(OtlpForwardError::InvalidBatch); + } + let exported_bytes = u64::try_from(body.len()).map_err(|_| OtlpForwardError::InvalidBatch)?; + let mut headers = headers.0; + for transport_header in [ + header::HOST, + header::CONTENT_LENGTH, + header::CONTENT_TYPE, + header::CONNECTION, + header::TRANSFER_ENCODING, + ] { + headers.remove(transport_header); + } + let client = Client::builder() + .redirect(reqwest::redirect::Policy::none()) + .timeout(timeout) + .build() + .map_err(|_| OtlpForwardError::Unavailable)?; + + let request = client + .post(endpoint) + .headers(headers) + .header(reqwest::header::CONTENT_TYPE, "application/x-protobuf") + .body(body); + let mut response = tokio::select! { + biased; + _ = cancel.cancelled() => return Err(OtlpForwardError::Cancelled), + _ = tokio::time::sleep_until(deadline) => return Err(deadline_error(consent_binds)), + response = request.send() => response.map_err(|_| OtlpForwardError::Unavailable)?, + }; + if response.status() != StatusCode::OK { + return Err(if response.status().is_client_error() { + OtlpForwardError::Rejected + } else { + OtlpForwardError::Unavailable + }); + } + loop { + let chunk = tokio::select! { + biased; + _ = cancel.cancelled() => return Err(OtlpForwardError::Cancelled), + _ = tokio::time::sleep_until(deadline) => return Err(deadline_error(consent_binds)), + chunk = response.chunk() => chunk.map_err(|_| OtlpForwardError::Unavailable)?, + }; + let Some(chunk) = chunk else { + break; + }; + if !chunk.is_empty() { + // A successful OTLP response may carry partial-success details. + // Until the approved protobuf adapter is wired, refusing such a + // response avoids claiming every span was accepted. + return Err(OtlpForwardError::Rejected); + } + } + + let receipt = OtlpReceipt { + accepted_span_count: span_count, + rejected_span_count: 0, + exported_bytes, + }; + ensure_result_size(&receipt).map_err(|_| OtlpForwardError::ResultTooLarge)?; + Ok(receipt) +} + +fn deadline_error(consent_binds: bool) -> OtlpForwardError { + if consent_binds { + OtlpForwardError::ConsentExpired + } else { + OtlpForwardError::Unavailable + } +} + +fn validate_endpoint(endpoint: &Url) -> Result<(), OtlpForwardError> { + let loopback_http = endpoint.scheme() == "http" + && endpoint + .host_str() + .is_some_and(|host| matches!(host, "localhost" | "127.0.0.1" | "[::1]" | "::1")); + if (endpoint.scheme() != "https" && !loopback_http) + || !endpoint.username().is_empty() + || endpoint.password().is_some() + || endpoint.query().is_some() + || endpoint.fragment().is_some() + { + return Err(OtlpForwardError::Endpoint); + } + Ok(()) +} + +fn map_consent_error(error: TelemetryProducerError) -> OtlpForwardError { + match error { + TelemetryProducerError::ConsentExpired => OtlpForwardError::ConsentExpired, + _ => OtlpForwardError::Unavailable, + } +} + +fn map_producer_error(error: TelemetryProducerError) -> OtlpForwardError { + match error { + TelemetryProducerError::Busy => OtlpForwardError::Busy, + _ => OtlpForwardError::InvalidBatch, + } +} diff --git a/rustfs/src/connect/diagnostics/trace_record.rs b/rustfs/src/connect/diagnostics/trace_record.rs new file mode 100644 index 000000000..5478beab5 --- /dev/null +++ b/rustfs/src/connect/diagnostics/trace_record.rs @@ -0,0 +1,1118 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded, locally-authorized capture of already classified trace spans. +//! +//! The producer accepts only the five operations frozen by the Connect +//! diagnostic contract. Request paths, bucket/object names, HTTP headers and +//! arbitrary attributes cannot enter the captured representation. + +use std::fs::{self, File, OpenOptions}; +use std::io::{Cursor, Write as _}; +use std::path::Path; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use base64_simd::URL_SAFE_NO_PAD; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use rustfs_common::trace_bus::{ + TelemetryTraceEvent, TelemetryTraceOperation, TelemetryTraceStatus, subscribe_telemetry_trace_events, +}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use thiserror::Error; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio::sync::mpsc; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use crate::connect::DeviceIdentity; + +pub const MAX_TELEMETRY_DURATION: Duration = Duration::from_secs(30); +pub const MAX_TELEMETRY_SPANS: usize = 1024; +pub const MAX_TELEMETRY_RESULT_BYTES: usize = 262_144; +pub const MAX_SAFE_INTEGER: u64 = 9_007_199_254_740_991; +pub const TELEMETRY_SCHEMA_VERSION: u16 = 1; +pub const TELEMETRY_RECORD_CAPABILITY: &str = "telemetry.record@1"; +pub const TELEMETRY_OTLP_CAPABILITY: &str = "telemetry.otlp@1"; +pub const TELEMETRY_REPLAY_CAPABILITY: &str = "telemetry.replay@1"; + +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1\0"; +const ENVELOPE_PATH: &str = "envelope.json"; +const SIGNATURE_PATH: &str = "envelope.sig"; +const RESULT_PATH: &str = "result.json"; +const MAX_ARCHIVE_BYTES: usize = 524_288; +const MAX_ENVELOPE_BYTES: usize = 16_384; +const MAX_DECOMPRESSED_BYTES: usize = 278_528; +const MAX_VALIDITY_SECONDS: i64 = 2_592_000; +const MAX_FUTURE_SKEW_SECONDS: i64 = 300; +const MAX_BUILD_FEATURES: usize = 64; +static TELEMETRY_LEASED: AtomicBool = AtomicBool::new(false); +#[cfg(unix)] +const OUTPUT_MODE: u32 = 0o600; + +pub(super) struct TelemetryLease; + +impl Drop for TelemetryLease { + fn drop(&mut self) { + TELEMETRY_LEASED.store(false, Ordering::Release); + } +} + +pub(super) fn acquire_telemetry_lease() -> Result { + TELEMETRY_LEASED + .compare_exchange(false, true, Ordering::Acquire, Ordering::Relaxed) + .map(|_| TelemetryLease) + .map_err(|_| TelemetryProducerError::Busy) +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum TelemetryOperation { + GetObject, + PutObject, + HeadObject, + ListObjects, + InternalRpc, +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum TelemetrySpanStatus { + Ok, + Error, +} + +/// A classified observation supplied by a server-side adapter. +/// +/// Its closed shape deliberately has nowhere to retain a URL, header, object +/// name, trace attribute, or message body. The adapter must classify an event +/// before crossing this boundary. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct ObservedTelemetrySpan { + operation: TelemetryOperation, + duration: Duration, + status: TelemetrySpanStatus, +} + +impl ObservedTelemetrySpan { + pub fn new(operation: TelemetryOperation, duration: Duration, status: TelemetrySpanStatus) -> Self { + Self { + operation, + duration, + status, + } + } +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct TelemetrySpan { + pub operation: TelemetryOperation, + pub duration_micros: u64, + pub status: TelemetrySpanStatus, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct RecordedTrace { + pub spans: Vec, + pub dropped_span_count: u64, +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +pub enum TraceRecordCompletion { + Complete, + LimitExceeded, + SourceUnavailable, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +pub struct TraceRecordCapture { + pub data: RecordedTrace, + pub completion: TraceRecordCompletion, +} + +#[derive(Clone, Copy, Debug)] +pub struct LocalTelemetryConsent { + expires_at: Instant, +} + +impl LocalTelemetryConsent { + pub fn new(expires_at: Instant) -> Result { + if expires_at <= Instant::now() { + return Err(TelemetryProducerError::ConsentExpired); + } + Ok(Self { expires_at }) + } + + pub fn remaining(self) -> Result { + self.expires_at + .checked_duration_since(Instant::now()) + .filter(|remaining| !remaining.is_zero()) + .ok_or(TelemetryProducerError::ConsentExpired) + } + + fn capture_deadline(self, duration: Duration) -> Result { + let deadline = Instant::now() + duration; + if deadline > self.expires_at { + return Err(TelemetryProducerError::ConsentExpired); + } + Ok(deadline) + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct TelemetryArtifactConsent { + pub consent_uid: String, + pub policy_revision: u64, + pub expires_at_unix: i64, + pub confirmed: bool, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TelemetryProvenance { + repository: &'static str, + source_commit: String, + executable_sha256: String, + rustfs_version: String, + os_family: TelemetryOsFamily, + architecture: TelemetryArchitecture, + build_features: Vec, +} + +impl TelemetryProvenance { + pub fn new( + source_commit: impl Into, + executable_sha256: impl Into, + rustfs_version: impl Into, + build_features: Vec, + ) -> Self { + Self { + repository: "rustfs/rustfs", + source_commit: source_commit.into(), + executable_sha256: executable_sha256.into(), + rustfs_version: rustfs_version.into(), + os_family: TelemetryOsFamily::current(), + architecture: TelemetryArchitecture::current(), + build_features, + } + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum TelemetryOsFamily { + Linux, + Darwin, + Windows, + Freebsd, + Other, +} + +impl TelemetryOsFamily { + fn current() -> Self { + match std::env::consts::OS { + "linux" => Self::Linux, + "macos" => Self::Darwin, + "windows" => Self::Windows, + "freebsd" => Self::Freebsd, + _ => Self::Other, + } + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "lowercase")] +enum TelemetryArchitecture { + #[serde(rename = "x86_64")] + X86_64, + Aarch64, + Other, +} + +impl TelemetryArchitecture { + fn current() -> Self { + match std::env::consts::ARCH { + "x86_64" => Self::X86_64, + "aarch64" => Self::Aarch64, + _ => Self::Other, + } + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct TelemetryArtifactRequest { + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub schema_version: u16, + pub consent: TelemetryArtifactConsent, + pub produced_at_unix: i64, + pub expires_at_unix: i64, + pub nonce: [u8; 32], + pub provenance: TelemetryProvenance, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +pub enum TelemetryTool { + #[serde(rename = "telemetry.record")] + Record, + #[serde(rename = "telemetry.otlp")] + Otlp, + #[serde(rename = "telemetry.replay")] + Replay, +} + +impl TelemetryTool { + pub const fn id(self) -> &'static str { + match self { + Self::Record => "telemetry.record", + Self::Otlp => "telemetry.otlp", + Self::Replay => "telemetry.replay", + } + } + + pub const fn capability(self) -> &'static str { + match self { + Self::Record => TELEMETRY_RECORD_CAPABILITY, + Self::Otlp => TELEMETRY_OTLP_CAPABILITY, + Self::Replay => TELEMETRY_REPLAY_CAPABILITY, + } + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum TelemetryOutcome { + Succeeded, + Partial, + Failed, + Unsupported, + Cancelled, +} + +impl TelemetryOutcome { + pub const fn as_str(self) -> &'static str { + match self { + Self::Succeeded => "SUCCEEDED", + Self::Partial => "PARTIAL", + Self::Failed => "FAILED", + Self::Unsupported => "UNSUPPORTED", + Self::Cancelled => "CANCELLED", + } + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum TelemetryReasonCode { + Complete, + LimitExceeded, + SourceUnavailable, + PermissionDenied, + UnsupportedTool, + Cancelled, + InvalidInput, + CollectionFailed, + CounterReset, +} + +impl TelemetryReasonCode { + pub const fn as_str(self) -> &'static str { + match self { + Self::Complete => "COMPLETE", + Self::LimitExceeded => "LIMIT_EXCEEDED", + Self::SourceUnavailable => "SOURCE_UNAVAILABLE", + Self::PermissionDenied => "PERMISSION_DENIED", + Self::UnsupportedTool => "UNSUPPORTED_TOOL", + Self::Cancelled => "CANCELLED", + Self::InvalidInput => "INVALID_INPUT", + Self::CollectionFailed => "COLLECTION_FAILED", + Self::CounterReset => "COUNTER_RESET", + } + } +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TelemetryCoverage { + requested_units: u64, + completed_units: u64, + unit: &'static str, +} + +impl TelemetryCoverage { + pub const fn window(completed: bool) -> Self { + Self { + requested_units: 1, + completed_units: completed as u64, + unit: "WINDOW", + } + } + + const fn partial_windows() -> Self { + Self { + requested_units: 2, + completed_units: 1, + unit: "WINDOW", + } + } +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct TelemetryDiagnosticResult { + schema_version: u16, + run_uid: String, + tool_id: TelemetryTool, + capability: &'static str, + outcome: TelemetryOutcome, + reason_code: TelemetryReasonCode, + duration_millis: u64, + provenance: TelemetryProvenance, + coverage: TelemetryCoverage, + data: Option, +} + +impl TelemetryDiagnosticResult { + pub fn succeeded(request: &TelemetryArtifactRequest, tool: TelemetryTool, duration: Duration, data: T) -> Self { + Self { + schema_version: TELEMETRY_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: tool, + capability: tool.capability(), + outcome: TelemetryOutcome::Succeeded, + reason_code: TelemetryReasonCode::Complete, + duration_millis: bounded_duration_millis(duration), + provenance: request.provenance.clone(), + coverage: TelemetryCoverage::window(true), + data: Some(data), + } + } + + pub fn partial( + request: &TelemetryArtifactRequest, + tool: TelemetryTool, + duration: Duration, + reason_code: TelemetryReasonCode, + data: T, + ) -> Self { + Self { + schema_version: TELEMETRY_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: tool, + capability: tool.capability(), + outcome: TelemetryOutcome::Partial, + reason_code, + duration_millis: bounded_duration_millis(duration), + provenance: request.provenance.clone(), + coverage: TelemetryCoverage::partial_windows(), + data: Some(data), + } + } + + fn failed( + request: &TelemetryArtifactRequest, + tool: TelemetryTool, + duration: Duration, + reason_code: TelemetryReasonCode, + ) -> Self { + Self { + schema_version: TELEMETRY_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: tool, + capability: tool.capability(), + outcome: TelemetryOutcome::Failed, + reason_code, + duration_millis: bounded_duration_millis(duration), + provenance: request.provenance.clone(), + coverage: TelemetryCoverage::window(false), + data: None, + } + } + + pub fn unsupported(request: &TelemetryArtifactRequest, tool: TelemetryTool, duration: Duration) -> Self { + Self { + schema_version: TELEMETRY_SCHEMA_VERSION, + run_uid: request.run_uid.clone(), + tool_id: tool, + capability: tool.capability(), + outcome: TelemetryOutcome::Unsupported, + reason_code: TelemetryReasonCode::UnsupportedTool, + duration_millis: bounded_duration_millis(duration), + provenance: request.provenance.clone(), + coverage: TelemetryCoverage::window(false), + data: None, + } + } + + pub fn outcome(&self) -> TelemetryOutcome { + self.outcome + } + + pub fn reason_code(&self) -> TelemetryReasonCode { + self.reason_code + } + + pub fn data(&self) -> Option<&T> { + self.data.as_ref() + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct SignedTelemetryExport { + pub artifact_uid: String, + pub tool: TelemetryTool, + pub outcome: TelemetryOutcome, + pub reason_code: TelemetryReasonCode, + pub archive_bytes: Vec, + pub archive_sha256: String, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct SavedTelemetryExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +#[derive(Clone, Copy, Debug)] +pub struct TraceRecordLimits { + pub duration: Duration, + pub max_spans: usize, +} + +impl TraceRecordLimits { + pub fn validate(self) -> Result { + if self.duration.is_zero() || self.duration > MAX_TELEMETRY_DURATION { + return Err(TelemetryProducerError::InvalidDuration); + } + if self.max_spans == 0 || self.max_spans > MAX_TELEMETRY_SPANS { + return Err(TelemetryProducerError::InvalidSpanLimit); + } + Ok(self) + } +} + +#[derive(Debug, Error, Eq, PartialEq)] +pub enum TelemetryProducerError { + #[error("another telemetry operation is already running")] + Busy, + #[error("local telemetry consent is expired")] + ConsentExpired, + #[error("telemetry duration must be between 1ns and 30s")] + InvalidDuration, + #[error("telemetry span limit must be between 1 and 1024")] + InvalidSpanLimit, + #[error("telemetry capture was cancelled")] + Cancelled, + #[error("telemetry source closed before the requested window completed")] + SourceUnavailable, + #[error("telemetry duration exceeds the contract integer range")] + DurationOverflow, + #[error("telemetry result exceeds 262144 bytes")] + ResultTooLarge, +} + +#[derive(Debug, Error)] +pub enum TelemetryArtifactError { + #[error("telemetry_invalid_request")] + InvalidRequest, + #[error("telemetry_unsupported_version")] + UnsupportedVersion, + #[error("telemetry_local_consent_required")] + ConsentRequired, + #[error("telemetry_local_consent_expired")] + ConsentExpired, + #[error("telemetry_request_expired")] + Expired, + #[error("telemetry_collection_cancelled")] + Cancelled, + #[error("telemetry_limit_exceeded")] + LimitExceeded, + #[error("telemetry_export_signing_failed")] + Signing, + #[error("telemetry_export_exists")] + AlreadyExists, + #[error("telemetry_export_encoding_failed")] + Encoding, + #[error("telemetry_export_io_failed")] + Io(#[source] std::io::Error), + #[error("telemetry_export_durability_failed_after_commit")] + DurabilityAfterCommit(#[source] std::io::Error), +} + +/// Capture classified spans until the bounded window ends. +/// +/// Reaching `max_spans` stops capture immediately and increments the drop count +/// for every already queued observation that could be counted without waiting. +pub async fn record_trace( + mut source: mpsc::Receiver, + consent: LocalTelemetryConsent, + limits: TraceRecordLimits, + cancel: &CancellationToken, +) -> Result { + let limits = limits.validate()?; + if cancel.is_cancelled() { + return Err(TelemetryProducerError::Cancelled); + } + let deadline = consent.capture_deadline(limits.duration)?; + let _lease = acquire_telemetry_lease()?; + let mut spans = Vec::with_capacity(limits.max_spans.min(64)); + let mut dropped_span_count = 0u64; + let mut completion = TraceRecordCompletion::Complete; + + loop { + tokio::select! { + biased; + _ = cancel.cancelled() => return Err(TelemetryProducerError::Cancelled), + _ = tokio::time::sleep_until(deadline.into()) => break, + observed = source.recv() => { + let Some(observed) = observed else { + if spans.is_empty() { + return Err(TelemetryProducerError::SourceUnavailable); + } + completion = TraceRecordCompletion::SourceUnavailable; + break; + }; + if spans.len() == limits.max_spans { + completion = TraceRecordCompletion::LimitExceeded; + dropped_span_count = dropped_span_count.saturating_add(1).min(MAX_SAFE_INTEGER); + while source.try_recv().is_ok() { + dropped_span_count = dropped_span_count.saturating_add(1).min(MAX_SAFE_INTEGER); + } + break; + } + spans.push(to_contract_span(observed)?); + } + } + } + + let result = RecordedTrace { + spans, + dropped_span_count, + }; + ensure_result_size(&result)?; + Ok(TraceRecordCapture { + data: result, + completion, + }) +} + +/// Capture the process-local, pre-classified S3 and internode RPC trace source. +pub async fn record_trace_bus( + consent: LocalTelemetryConsent, + limits: TraceRecordLimits, + cancel: &CancellationToken, +) -> Result { + let limits = limits.validate()?; + if cancel.is_cancelled() { + return Err(TelemetryProducerError::Cancelled); + } + let deadline = consent.capture_deadline(limits.duration)?; + let _lease = acquire_telemetry_lease()?; + let mut subscription = subscribe_telemetry_trace_events(); + let mut spans = Vec::with_capacity(limits.max_spans.min(64)); + let mut dropped_span_count = 0u64; + let mut completion = TraceRecordCompletion::Complete; + loop { + tokio::select! { + biased; + _ = cancel.cancelled() => return Err(TelemetryProducerError::Cancelled), + _ = tokio::time::sleep_until(deadline.into()) => break, + received = subscription.recv() => match received { + Ok(event) => { + if spans.len() == limits.max_spans { + completion = TraceRecordCompletion::LimitExceeded; + dropped_span_count = dropped_span_count.saturating_add(1).min(MAX_SAFE_INTEGER); + drain_telemetry_drops(&mut subscription, &mut dropped_span_count); + break; + } + spans.push(to_contract_span(observe_trace_bus_event(&event))?); + } + Err(tokio::sync::broadcast::error::RecvError::Lagged(dropped)) => { + completion = TraceRecordCompletion::LimitExceeded; + dropped_span_count = dropped_span_count.saturating_add(dropped).min(MAX_SAFE_INTEGER); + break; + } + Err(tokio::sync::broadcast::error::RecvError::Closed) => { + if spans.is_empty() { + return Err(TelemetryProducerError::SourceUnavailable); + } + completion = TraceRecordCompletion::SourceUnavailable; + break; + } + } + } + } + + let result = RecordedTrace { + spans, + dropped_span_count, + }; + ensure_result_size(&result)?; + Ok(TraceRecordCapture { + data: result, + completion, + }) +} + +fn observe_trace_bus_event(event: &TelemetryTraceEvent) -> ObservedTelemetrySpan { + let operation = match event.operation { + TelemetryTraceOperation::GetObject => TelemetryOperation::GetObject, + TelemetryTraceOperation::PutObject => TelemetryOperation::PutObject, + TelemetryTraceOperation::HeadObject => TelemetryOperation::HeadObject, + TelemetryTraceOperation::ListObjects => TelemetryOperation::ListObjects, + TelemetryTraceOperation::InternalRpc => TelemetryOperation::InternalRpc, + }; + let status = match event.status { + TelemetryTraceStatus::Ok => TelemetrySpanStatus::Ok, + TelemetryTraceStatus::Error => TelemetrySpanStatus::Error, + }; + ObservedTelemetrySpan::new(operation, event.duration, status) +} + +fn drain_telemetry_drops(subscription: &mut rustfs_common::trace_bus::TelemetryTraceSubscription, dropped_span_count: &mut u64) { + loop { + match subscription.try_recv() { + Ok(_) => *dropped_span_count = dropped_span_count.saturating_add(1).min(MAX_SAFE_INTEGER), + Err(tokio::sync::broadcast::error::TryRecvError::Lagged(dropped)) => { + *dropped_span_count = dropped_span_count.saturating_add(dropped).min(MAX_SAFE_INTEGER); + } + Err(tokio::sync::broadcast::error::TryRecvError::Empty | tokio::sync::broadcast::error::TryRecvError::Closed) => { + break; + } + } + } +} + +pub fn record_diagnostic_result( + request: &TelemetryArtifactRequest, + capture: TraceRecordCapture, + duration: Duration, +) -> TelemetryDiagnosticResult { + match capture.completion { + TraceRecordCompletion::Complete => { + TelemetryDiagnosticResult::succeeded(request, TelemetryTool::Record, duration, capture.data) + } + TraceRecordCompletion::LimitExceeded => { + TelemetryDiagnosticResult::failed(request, TelemetryTool::Record, duration, TelemetryReasonCode::CollectionFailed) + } + TraceRecordCompletion::SourceUnavailable => { + TelemetryDiagnosticResult::failed(request, TelemetryTool::Record, duration, TelemetryReasonCode::SourceUnavailable) + } + } +} + +fn to_contract_span(observed: ObservedTelemetrySpan) -> Result { + let duration_micros = u64::try_from(observed.duration.as_micros()).map_err(|_| TelemetryProducerError::DurationOverflow)?; + if duration_micros > MAX_SAFE_INTEGER { + return Err(TelemetryProducerError::DurationOverflow); + } + Ok(TelemetrySpan { + operation: observed.operation, + duration_micros, + status: observed.status, + }) +} + +pub(crate) fn ensure_result_size(value: &impl Serialize) -> Result<(), TelemetryProducerError> { + let bytes = serde_json::to_vec(value).map_err(|_| TelemetryProducerError::ResultTooLarge)?; + if bytes.len() > MAX_TELEMETRY_RESULT_BYTES { + return Err(TelemetryProducerError::ResultTooLarge); + } + Ok(()) +} + +pub fn encode_signed_telemetry_export( + request: &TelemetryArtifactRequest, + result: &TelemetryDiagnosticResult, + key: &DeviceIdentity, + cancel: &CancellationToken, +) -> Result { + request.validate()?; + check_cancel(cancel)?; + if result.schema_version != request.schema_version || result.run_uid != request.run_uid { + return Err(TelemetryArtifactError::InvalidRequest); + } + if !matches!(result.outcome, TelemetryOutcome::Succeeded | TelemetryOutcome::Partial) { + return Err(TelemetryArtifactError::InvalidRequest); + } + let valid_publishable_shape = match result.outcome { + TelemetryOutcome::Succeeded => { + result.data.is_some() + && result.reason_code == TelemetryReasonCode::Complete + && result.coverage.requested_units == 1 + && result.coverage.completed_units == 1 + && result.coverage.unit == "WINDOW" + } + TelemetryOutcome::Partial => { + result.data.is_some() + && matches!( + result.reason_code, + TelemetryReasonCode::LimitExceeded + | TelemetryReasonCode::SourceUnavailable + | TelemetryReasonCode::CounterReset + ) + && result.coverage.requested_units == 2 + && result.coverage.completed_units == 1 + && result.coverage.unit == "WINDOW" + } + _ => false, + }; + if !valid_publishable_shape { + return Err(TelemetryArtifactError::InvalidRequest); + } + if result.capability != result.tool_id.capability() { + return Err(TelemetryArtifactError::InvalidRequest); + } + let result_bytes = serde_json::to_vec(result).map_err(|_| TelemetryArtifactError::Encoding)?; + if result_bytes.is_empty() || result_bytes.len() > MAX_TELEMETRY_RESULT_BYTES { + return Err(TelemetryArtifactError::LimitExceeded); + } + let device_key_id = hex_lower(&Sha256::digest(key.public_key_der())); + let envelope = TelemetryEnvelope { + format_version: "rustfs.connect.diagnosticEnvelope/1", + protocol_version: "v1", + organization_name: &request.organization_name, + cluster_name: &request.cluster_name, + device_name: &request.device_name, + run_uid: &request.run_uid, + artifact_uid: &request.artifact_uid, + tool_id: result.tool_id, + schema_version: TELEMETRY_SCHEMA_VERSION, + classification: "L3", + consent_uid: &request.consent.consent_uid, + policy_revision: request.consent.policy_revision, + produced_at: timestamp(request.produced_at_unix)?, + expires_at: timestamp(request.expires_at_unix)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(request.nonce), + device_key_id: &device_key_id, + payload: TelemetryPayload { + path: RESULT_PATH, + media_type: "application/json", + size_bytes: result_bytes.len() as u64, + sha256: hex_lower(&Sha256::digest(&result_bytes)), + }, + }; + let envelope_bytes = serde_json::to_vec(&envelope).map_err(|_| TelemetryArtifactError::Encoding)?; + if envelope_bytes.is_empty() || envelope_bytes.len() > MAX_ENVELOPE_BYTES { + return Err(TelemetryArtifactError::LimitExceeded); + } + let signature_bytes = signature_document(key, &device_key_id, &envelope_bytes)?; + let decompressed = result_bytes + .len() + .checked_add(envelope_bytes.len()) + .and_then(|size| size.checked_add(signature_bytes.len())) + .ok_or(TelemetryArtifactError::LimitExceeded)?; + if decompressed > MAX_DECOMPRESSED_BYTES { + return Err(TelemetryArtifactError::LimitExceeded); + } + check_cancel(cancel)?; + if unix_now()? >= request.expires_at_unix { + return Err(TelemetryArtifactError::Expired); + } + let archive_bytes = archive(&envelope_bytes, &signature_bytes, &result_bytes)?; + if archive_bytes.len() > MAX_ARCHIVE_BYTES { + return Err(TelemetryArtifactError::LimitExceeded); + } + Ok(SignedTelemetryExport { + artifact_uid: request.artifact_uid.clone(), + tool: result.tool_id, + outcome: result.outcome, + reason_code: result.reason_code, + archive_sha256: hex_lower(&Sha256::digest(&archive_bytes)), + archive_bytes, + }) +} + +pub fn save_signed_telemetry_export( + output: &Path, + export: &SignedTelemetryExport, + cancel: &CancellationToken, +) -> Result { + check_cancel(cancel)?; + if !uuid7(&export.artifact_uid) + || export.archive_bytes.is_empty() + || export.archive_bytes.len() > MAX_ARCHIVE_BYTES + || hex_lower(&Sha256::digest(&export.archive_bytes)) != export.archive_sha256 + { + return Err(TelemetryArtifactError::InvalidRequest); + } + let parent = output + .parent() + .filter(|path| !path.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let filename = output + .file_name() + .ok_or(TelemetryArtifactError::InvalidRequest)? + .to_string_lossy(); + let temporary = parent.join(format!(".{filename}.{}.partial", export.artifact_uid)); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + let mut file = options.open(&temporary).map_err(map_create_error)?; + let result = (|| { + file.write_all(&export.archive_bytes).map_err(TelemetryArtifactError::Io)?; + check_cancel(cancel)?; + file.sync_all().map_err(TelemetryArtifactError::Io)?; + check_cancel(cancel)?; + fs::hard_link(&temporary, output).map_err(map_publish_error)?; + if let Err(error) = fs::remove_file(&temporary) { + return Err(TelemetryArtifactError::DurabilityAfterCommit(error)); + } + #[cfg(unix)] + if let Err(error) = File::open(parent).and_then(|directory| directory.sync_all()) { + return Err(TelemetryArtifactError::DurabilityAfterCommit(error)); + } + Ok(SavedTelemetryExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: export.archive_bytes.len() as u64, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if result.is_err() { + let _ = fs::remove_file(temporary); + } + result +} + +impl TelemetryArtifactRequest { + pub fn validate(&self) -> Result<(), TelemetryArtifactError> { + self.validate_at(unix_now()?) + } + + fn validate_at(&self, now_unix: i64) -> Result<(), TelemetryArtifactError> { + if self.schema_version != TELEMETRY_SCHEMA_VERSION { + return Err(TelemetryArtifactError::UnsupportedVersion); + } + if !self.consent.confirmed || self.consent.policy_revision == 0 { + return Err(TelemetryArtifactError::ConsentRequired); + } + if self.consent.expires_at_unix <= now_unix || self.expires_at_unix > self.consent.expires_at_unix { + return Err(TelemetryArtifactError::ConsentExpired); + } + let validity = self + .expires_at_unix + .checked_sub(self.produced_at_unix) + .ok_or(TelemetryArtifactError::Expired)?; + if self.produced_at_unix > now_unix.saturating_add(MAX_FUTURE_SKEW_SECONDS) + || validity <= 0 + || self.expires_at_unix <= now_unix + || validity > MAX_VALIDITY_SECONDS + { + return Err(TelemetryArtifactError::Expired); + } + if !uuid7(&self.run_uid) + || !uuid7(&self.artifact_uid) + || !uuid7(&self.consent.consent_uid) + || !resource_names_match(self) + || !lower_hex(&self.provenance.source_commit, 40) + || !lower_hex(&self.provenance.executable_sha256, 64) + || !version(&self.provenance.rustfs_version) + || self.provenance.build_features.len() > MAX_BUILD_FEATURES + || !self.provenance.build_features.iter().all(|feature| build_feature(feature)) + { + return Err(TelemetryArtifactError::InvalidRequest); + } + Ok(()) + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct TelemetryEnvelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: TelemetryTool, + schema_version: u16, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: &'a str, + payload: TelemetryPayload, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct TelemetryPayload { + path: &'static str, + media_type: &'static str, + size_bytes: u64, + sha256: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct SignatureDocument<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +fn signature_document(key: &DeviceIdentity, key_id: &str, envelope: &[u8]) -> Result, TelemetryArtifactError> { + let pkcs8 = key.to_pkcs8_der().map_err(|_| TelemetryArtifactError::Signing)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| TelemetryArtifactError::Signing)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + envelope.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.extend_from_slice(envelope); + let signature: Signature = signing_key.sign(&input); + serde_json::to_vec(&SignatureDocument { + algorithm: "ES256", + key_id, + value: URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()), + }) + .map_err(|_| TelemetryArtifactError::Encoding) +} + +fn archive(envelope: &[u8], signature: &[u8], result: &[u8]) -> Result, TelemetryArtifactError> { + let cursor = Cursor::new(Vec::with_capacity(envelope.len() + signature.len() + result.len() + 512)); + let mut writer = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(0o600); + for (name, bytes) in [(ENVELOPE_PATH, envelope), (SIGNATURE_PATH, signature), (RESULT_PATH, result)] { + writer + .start_file(name, options) + .map_err(|_| TelemetryArtifactError::Encoding)?; + writer.write_all(bytes).map_err(TelemetryArtifactError::Io)?; + } + writer + .finish() + .map(|cursor| cursor.into_inner()) + .map_err(|_| TelemetryArtifactError::Encoding) +} + +fn resource_names_match(request: &TelemetryArtifactRequest) -> bool { + let Some(organization_uid) = request.organization_name.strip_prefix("organizations/") else { + return false; + }; + if !uuid7(organization_uid) { + return false; + } + let cluster_prefix = format!("{}/clusters/", request.organization_name); + let Some(cluster_uid) = request.cluster_name.strip_prefix(&cluster_prefix) else { + return false; + }; + if !uuid7(cluster_uid) { + return false; + } + let device_prefix = format!("{}/clusterDevices/", request.cluster_name); + request.device_name.strip_prefix(&device_prefix).is_some_and(uuid7) +} + +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn lower_hex(value: &str, length: usize) -> bool { + value.len() == length + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn version(value: &str) -> bool { + if value.is_empty() + || value.len() > 64 + || !value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'-')) + { + return false; + } + let (core, suffix) = value + .split_once('-') + .map_or((value, None), |(core, suffix)| (core, Some(suffix))); + if suffix.is_some_and(str::is_empty) { + return false; + } + let mut parts = core.split('.'); + parts.clone().count() == 3 && parts.all(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) +} + +fn build_feature(value: &str) -> bool { + !value.is_empty() + && value.len() <= 64 + && value.as_bytes()[0].is_ascii_lowercase() + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'_' | b'-')) +} + +fn timestamp(unix: i64) -> Result { + OffsetDateTime::from_unix_timestamp(unix) + .map_err(|_| TelemetryArtifactError::InvalidRequest)? + .format(&Rfc3339) + .map_err(|_| TelemetryArtifactError::InvalidRequest) +} + +fn unix_now() -> Result { + let duration = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| TelemetryArtifactError::InvalidRequest)?; + i64::try_from(duration.as_secs()).map_err(|_| TelemetryArtifactError::InvalidRequest) +} + +fn check_cancel(cancel: &CancellationToken) -> Result<(), TelemetryArtifactError> { + if cancel.is_cancelled() { + Err(TelemetryArtifactError::Cancelled) + } else { + Ok(()) + } +} + +fn bounded_duration_millis(duration: Duration) -> u64 { + u64::try_from(duration.as_millis()).unwrap_or(u64::MAX).min(30_000) +} + +fn hex_lower(bytes: &[u8]) -> String { + hex_simd::encode_to_string(bytes, hex_simd::AsciiCase::Lower) +} + +fn map_create_error(error: std::io::Error) -> TelemetryArtifactError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + TelemetryArtifactError::AlreadyExists + } else { + TelemetryArtifactError::Io(error) + } +} + +fn map_publish_error(error: std::io::Error) -> TelemetryArtifactError { + if error.kind() == std::io::ErrorKind::AlreadyExists { + TelemetryArtifactError::AlreadyExists + } else { + TelemetryArtifactError::Io(error) + } +} diff --git a/rustfs/src/connect/diagnostics/trace_replay.rs b/rustfs/src/connect/diagnostics/trace_replay.rs new file mode 100644 index 000000000..9690ac4d8 --- /dev/null +++ b/rustfs/src/connect/diagnostics/trace_replay.rs @@ -0,0 +1,127 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! In-memory replay of a locally reviewed telemetry record. +//! +//! This module never opens a path and never replays S3 operations. The caller +//! supplies reviewed bytes; replay validates the exact closed record shape and +//! returns only its redacted operation/timing/status observations. + +use std::time::Instant; + +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use thiserror::Error; +use tokio_util::sync::CancellationToken; + +use super::trace_record::{ + LocalTelemetryConsent, MAX_SAFE_INTEGER, MAX_TELEMETRY_RESULT_BYTES, MAX_TELEMETRY_SPANS, RecordedTrace, + TelemetryArtifactRequest, TelemetryDiagnosticResult, TelemetryProducerError, TelemetrySpan, TelemetryTool, + acquire_telemetry_lease, ensure_result_size, +}; + +pub struct LocallyReviewedTraceArtifact<'a> { + bytes: &'a [u8], +} + +impl<'a> LocallyReviewedTraceArtifact<'a> { + pub fn new(bytes: &'a [u8]) -> Result { + if bytes.is_empty() || bytes.len() > MAX_TELEMETRY_RESULT_BYTES { + return Err(TraceReplayError::InvalidArtifact); + } + Ok(Self { bytes }) + } +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct ReplayedTrace { + pub input_artifact_sha256: String, + pub spans: Vec, +} + +#[derive(Debug, Error, Eq, PartialEq)] +pub enum TraceReplayError { + #[error("another telemetry operation is already running")] + Busy, + #[error("reviewed telemetry artifact is invalid")] + InvalidArtifact, + #[error("telemetry replay was cancelled")] + Cancelled, + #[error("local telemetry consent is expired")] + ConsentExpired, + #[error("telemetry replay result exceeds 262144 bytes")] + ResultTooLarge, +} + +pub fn replay_trace( + artifact: LocallyReviewedTraceArtifact<'_>, + consent: LocalTelemetryConsent, + cancel: &CancellationToken, +) -> Result { + consent.remaining().map_err(map_consent_error)?; + if cancel.is_cancelled() { + return Err(TraceReplayError::Cancelled); + } + let _lease = acquire_telemetry_lease().map_err(map_consent_error)?; + let record: RecordedTrace = serde_json::from_slice(artifact.bytes).map_err(|_| TraceReplayError::InvalidArtifact)?; + if record.spans.len() > MAX_TELEMETRY_SPANS + || record.dropped_span_count > MAX_SAFE_INTEGER + || record.spans.iter().any(|span| span.duration_micros > MAX_SAFE_INTEGER) + { + return Err(TraceReplayError::InvalidArtifact); + } + if cancel.is_cancelled() { + return Err(TraceReplayError::Cancelled); + } + let result = ReplayedTrace { + input_artifact_sha256: hex_lower(&Sha256::digest(artifact.bytes)), + spans: record.spans, + }; + ensure_result_size(&result).map_err(|_| TraceReplayError::ResultTooLarge)?; + Ok(result) +} + +pub fn replay_trace_result( + request: &TelemetryArtifactRequest, + artifact: LocallyReviewedTraceArtifact<'_>, + consent: LocalTelemetryConsent, + cancel: &CancellationToken, +) -> Result, TraceReplayError> { + let started = Instant::now(); + let replay = replay_trace(artifact, consent, cancel)?; + Ok(TelemetryDiagnosticResult::succeeded( + request, + TelemetryTool::Replay, + started.elapsed(), + replay, + )) +} + +fn map_consent_error(error: TelemetryProducerError) -> TraceReplayError { + match error { + TelemetryProducerError::Busy => TraceReplayError::Busy, + TelemetryProducerError::ConsentExpired => TraceReplayError::ConsentExpired, + _ => TraceReplayError::InvalidArtifact, + } +} + +fn hex_lower(bytes: &[u8]) -> String { + use std::fmt::Write as _; + + bytes.iter().fold(String::with_capacity(bytes.len() * 2), |mut output, byte| { + let _ = write!(output, "{byte:02x}"); + output + }) +} diff --git a/rustfs/src/connect/diagnostics/trace_runtime.rs b/rustfs/src/connect/diagnostics/trace_runtime.rs new file mode 100644 index 000000000..a9418338f --- /dev/null +++ b/rustfs/src/connect/diagnostics/trace_runtime.rs @@ -0,0 +1,526 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Owner-only local transport between the telemetry CLI and the running server. + +use std::io; +use std::os::unix::fs::{FileTypeExt as _, MetadataExt as _, PermissionsExt as _}; +use std::path::Path; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use serde::{Deserialize, Serialize}; +use thiserror::Error; +use tokio::io::{AsyncBufReadExt as _, AsyncReadExt as _, AsyncWriteExt as _, BufReader}; +use tokio::net::{UnixListener, UnixStream}; +use tokio::task::{JoinHandle, JoinSet}; +use tokio_util::sync::CancellationToken; + +use super::{LocalTelemetryConsent, TelemetryProducerError, TraceRecordCapture, TraceRecordLimits, record_trace_bus}; + +const SOCKET_FILE: &str = "telemetry-record.sock"; +const PROTOCOL_VERSION: u16 = 1; +const MAX_REQUEST_BYTES: u64 = 1_024; +const MAX_RESPONSE_BYTES: u64 = 300_000; +const MAX_CONNECTIONS: usize = 8; +const REQUEST_TIMEOUT: Duration = Duration::from_secs(1); +const STATE_DIRECTORY_MODE: u32 = 0o700; +const SOCKET_MODE: u32 = 0o600; + +#[derive(Debug, Error)] +pub(crate) enum LocalTraceCaptureError { + #[error("telemetry server runtime is unavailable")] + RuntimeUnavailable, + #[error("telemetry server runtime state is not owner-only")] + StateSecurity, + #[error("telemetry server runtime protocol failed")] + Protocol, + #[error("telemetry server runtime I/O failed")] + Io(#[source] io::Error), + #[error(transparent)] + Producer(#[from] TelemetryProducerError), +} + +pub(crate) struct LocalTraceCaptureRuntime { + shutdown: CancellationToken, + task: Option>, +} + +impl LocalTraceCaptureRuntime { + pub async fn shutdown(mut self) { + self.shutdown.cancel(); + if let Some(task) = self.task.take() { + let _ = task.await; + } + } +} + +impl Drop for LocalTraceCaptureRuntime { + fn drop(&mut self) { + self.shutdown.cancel(); + } +} + +#[derive(Debug, Deserialize, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +struct CaptureRequest { + protocol_version: u16, + consent_expires_at_unix: i64, + duration_millis: u64, + max_spans: usize, +} + +#[derive(Clone, Copy, Debug, Deserialize, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum ProducerErrorCode { + Busy, + ConsentExpired, + InvalidDuration, + InvalidSpanLimit, + Cancelled, + SourceUnavailable, + DurationOverflow, + ResultTooLarge, +} + +impl From<&TelemetryProducerError> for ProducerErrorCode { + fn from(error: &TelemetryProducerError) -> Self { + match error { + TelemetryProducerError::Busy => Self::Busy, + TelemetryProducerError::ConsentExpired => Self::ConsentExpired, + TelemetryProducerError::InvalidDuration => Self::InvalidDuration, + TelemetryProducerError::InvalidSpanLimit => Self::InvalidSpanLimit, + TelemetryProducerError::Cancelled => Self::Cancelled, + TelemetryProducerError::SourceUnavailable => Self::SourceUnavailable, + TelemetryProducerError::DurationOverflow => Self::DurationOverflow, + TelemetryProducerError::ResultTooLarge => Self::ResultTooLarge, + } + } +} + +impl From for TelemetryProducerError { + fn from(code: ProducerErrorCode) -> Self { + match code { + ProducerErrorCode::Busy => Self::Busy, + ProducerErrorCode::ConsentExpired => Self::ConsentExpired, + ProducerErrorCode::InvalidDuration => Self::InvalidDuration, + ProducerErrorCode::InvalidSpanLimit => Self::InvalidSpanLimit, + ProducerErrorCode::Cancelled => Self::Cancelled, + ProducerErrorCode::SourceUnavailable => Self::SourceUnavailable, + ProducerErrorCode::DurationOverflow => Self::DurationOverflow, + ProducerErrorCode::ResultTooLarge => Self::ResultTooLarge, + } + } +} + +#[derive(Debug, Deserialize, Serialize)] +#[serde(deny_unknown_fields, tag = "status", rename_all = "SCREAMING_SNAKE_CASE")] +enum CaptureResponse { + Ok { capture: TraceRecordCapture }, + Error { code: ProducerErrorCode }, +} + +pub(crate) fn spawn_local_trace_capture_runtime( + state_root: &Path, + parent_shutdown: &CancellationToken, +) -> Result { + let owner = private_state_owner(state_root)?; + let socket_path = state_root.join(SOCKET_FILE); + let listener = bind_listener(&socket_path, owner)?; + let socket_identity = socket_identity(&socket_path, owner)?; + let shutdown = parent_shutdown.child_token(); + let task_shutdown = shutdown.clone(); + let task = tokio::spawn(async move { + run_listener(listener, owner, task_shutdown).await; + remove_own_socket(&socket_path, socket_identity); + }); + Ok(LocalTraceCaptureRuntime { + shutdown, + task: Some(task), + }) +} + +pub(crate) async fn request_local_trace_capture( + state_root: &Path, + consent_expires_at_unix: i64, + limits: TraceRecordLimits, + cancel: &CancellationToken, +) -> Result { + limits.validate()?; + let owner = private_state_owner(state_root)?; + let socket_path = state_root.join(SOCKET_FILE); + socket_identity(&socket_path, owner)?; + let stream = UnixStream::connect(&socket_path).await.map_err(|error| match error.kind() { + io::ErrorKind::NotFound | io::ErrorKind::ConnectionRefused => LocalTraceCaptureError::RuntimeUnavailable, + _ => LocalTraceCaptureError::Io(error), + })?; + let (reader, mut writer) = stream.into_split(); + let request = CaptureRequest { + protocol_version: PROTOCOL_VERSION, + consent_expires_at_unix, + duration_millis: u64::try_from(limits.duration.as_millis()).map_err(|_| TelemetryProducerError::InvalidDuration)?, + max_spans: limits.max_spans, + }; + let mut encoded = serde_json::to_vec(&request).map_err(|_| LocalTraceCaptureError::Protocol)?; + encoded.push(b'\n'); + if encoded.len() as u64 > MAX_REQUEST_BYTES { + return Err(LocalTraceCaptureError::Protocol); + } + writer.write_all(&encoded).await.map_err(LocalTraceCaptureError::Io)?; + + let response = async move { + let mut bytes = Vec::new(); + reader + .take(MAX_RESPONSE_BYTES + 1) + .read_to_end(&mut bytes) + .await + .map_err(LocalTraceCaptureError::Io)?; + if bytes.is_empty() || bytes.len() as u64 > MAX_RESPONSE_BYTES { + return Err(LocalTraceCaptureError::Protocol); + } + serde_json::from_slice::(&bytes).map_err(|_| LocalTraceCaptureError::Protocol) + }; + let response = tokio::select! { + biased; + _ = cancel.cancelled() => return Err(TelemetryProducerError::Cancelled.into()), + response = response => response?, + }; + match response { + CaptureResponse::Ok { capture } => Ok(capture), + CaptureResponse::Error { code } => Err(TelemetryProducerError::from(code).into()), + } +} + +async fn run_listener(listener: UnixListener, owner: u32, shutdown: CancellationToken) { + let mut connections = JoinSet::new(); + loop { + tokio::select! { + biased; + _ = shutdown.cancelled() => break, + Some(_) = connections.join_next(), if !connections.is_empty() => {} + accepted = listener.accept() => match accepted { + Ok((stream, _)) => { + if connections.len() < MAX_CONNECTIONS + && stream.peer_cred().is_ok_and(|credentials| credentials.uid() == owner) + { + connections.spawn(handle_connection(stream, shutdown.clone())); + } + } + Err(_) => break, + }, + } + } + while connections.join_next().await.is_some() {} +} + +async fn handle_connection(stream: UnixStream, shutdown: CancellationToken) { + let (reader, mut writer) = stream.into_split(); + let mut reader = BufReader::new(reader); + let request = tokio::select! { + biased; + _ = shutdown.cancelled() => return, + result = tokio::time::timeout(REQUEST_TIMEOUT, read_request(&mut reader)) => match result { + Ok(Ok(request)) => request, + Ok(Err(_)) | Err(_) => return, + }, + }; + let limits = TraceRecordLimits { + duration: Duration::from_millis(request.duration_millis), + max_spans: request.max_spans, + }; + let capture = async { + if request.protocol_version != PROTOCOL_VERSION { + return Err(TelemetryProducerError::SourceUnavailable); + } + limits.validate()?; + let remaining = request + .consent_expires_at_unix + .checked_sub(unix_now().map_err(|_| TelemetryProducerError::ConsentExpired)?) + .and_then(|seconds| u64::try_from(seconds).ok()) + .filter(|seconds| *seconds > 0) + .ok_or(TelemetryProducerError::ConsentExpired)?; + let expires_at = Instant::now() + .checked_add(Duration::from_secs(remaining)) + .ok_or(TelemetryProducerError::ConsentExpired)?; + let consent = LocalTelemetryConsent::new(expires_at)?; + record_trace_bus(consent, limits, &shutdown).await + }; + tokio::pin!(capture); + let mut unexpected = [0_u8; 1]; + let disconnected = reader.read(&mut unexpected); + tokio::pin!(disconnected); + let result = tokio::select! { + biased; + _ = shutdown.cancelled() => Err(TelemetryProducerError::Cancelled), + _ = &mut disconnected => return, + result = &mut capture => result, + }; + let response = match result { + Ok(capture) => CaptureResponse::Ok { capture }, + Err(error) => CaptureResponse::Error { + code: ProducerErrorCode::from(&error), + }, + }; + if let Ok(bytes) = serde_json::to_vec(&response) { + let write_response = async { + writer.write_all(&bytes).await?; + writer.shutdown().await + }; + tokio::select! { + biased; + _ = shutdown.cancelled() => {} + _ = tokio::time::timeout(REQUEST_TIMEOUT, write_response) => {} + } + } +} + +async fn read_request(reader: &mut BufReader) -> Result { + let mut bytes = Vec::new(); + reader + .take(MAX_REQUEST_BYTES + 1) + .read_until(b'\n', &mut bytes) + .await + .map_err(LocalTraceCaptureError::Io)?; + if bytes.is_empty() || bytes.len() as u64 > MAX_REQUEST_BYTES || bytes.pop() != Some(b'\n') { + return Err(LocalTraceCaptureError::Protocol); + } + serde_json::from_slice(&bytes).map_err(|_| LocalTraceCaptureError::Protocol) +} + +fn private_state_owner(path: &Path) -> Result { + let metadata = std::fs::symlink_metadata(path).map_err(LocalTraceCaptureError::Io)?; + if metadata.file_type().is_symlink() || !metadata.is_dir() || metadata.permissions().mode() & 0o777 != STATE_DIRECTORY_MODE { + return Err(LocalTraceCaptureError::StateSecurity); + } + Ok(metadata.uid()) +} + +fn bind_listener(path: &Path, owner: u32) -> Result { + match UnixListener::bind(path) { + Ok(listener) => seal_listener(path, owner, listener), + Err(error) if error.kind() == io::ErrorKind::AddrInUse => { + let identity = socket_identity(path, owner)?; + match std::os::unix::net::UnixStream::connect(path) { + Ok(_) => Err(LocalTraceCaptureError::RuntimeUnavailable), + Err(connect_error) if connect_error.kind() == io::ErrorKind::ConnectionRefused => { + remove_own_socket(path, identity); + let listener = UnixListener::bind(path).map_err(LocalTraceCaptureError::Io)?; + seal_listener(path, owner, listener) + } + Err(connect_error) => Err(LocalTraceCaptureError::Io(connect_error)), + } + } + Err(error) => Err(LocalTraceCaptureError::Io(error)), + } +} + +fn seal_listener(path: &Path, owner: u32, listener: UnixListener) -> Result { + let metadata = std::fs::symlink_metadata(path).map_err(LocalTraceCaptureError::Io)?; + if !metadata.file_type().is_socket() || metadata.uid() != owner { + return Err(LocalTraceCaptureError::StateSecurity); + } + let identity = (metadata.dev(), metadata.ino()); + let sealed = std::fs::set_permissions(path, std::fs::Permissions::from_mode(SOCKET_MODE)) + .map_err(LocalTraceCaptureError::Io) + .and_then(|()| socket_identity(path, owner)); + match sealed { + Ok(sealed_identity) if sealed_identity == identity => Ok(listener), + Ok(_) => Err(LocalTraceCaptureError::StateSecurity), + Err(error) => { + drop(listener); + remove_own_socket(path, identity); + Err(error) + } + } +} + +fn socket_identity(path: &Path, owner: u32) -> Result<(u64, u64), LocalTraceCaptureError> { + let metadata = std::fs::symlink_metadata(path).map_err(|error| match error.kind() { + io::ErrorKind::NotFound => LocalTraceCaptureError::RuntimeUnavailable, + _ => LocalTraceCaptureError::Io(error), + })?; + if !metadata.file_type().is_socket() || metadata.uid() != owner || metadata.permissions().mode() & 0o777 != SOCKET_MODE { + return Err(LocalTraceCaptureError::StateSecurity); + } + Ok((metadata.dev(), metadata.ino())) +} + +fn remove_own_socket(path: &Path, identity: (u64, u64)) { + if std::fs::symlink_metadata(path).is_ok_and(|metadata| (metadata.dev(), metadata.ino()) == identity) { + let _ = std::fs::remove_file(path); + } +} + +fn unix_now() -> io::Result { + let duration = SystemTime::now().duration_since(UNIX_EPOCH).map_err(io::Error::other)?; + i64::try_from(duration.as_secs()).map_err(io::Error::other) +} + +#[cfg(test)] +mod tests { + use std::os::unix::fs::PermissionsExt as _; + use std::time::{Duration, SystemTime, UNIX_EPOCH}; + + use rustfs_common::trace_bus::{ + TelemetryTraceEvent, TelemetryTraceOperation, TelemetryTraceStatus, telemetry_trace_emit, + telemetry_trace_subscriber_count, + }; + use serial_test::serial; + use tokio_util::sync::CancellationToken; + + use super::{LocalTraceCaptureError, request_local_trace_capture, spawn_local_trace_capture_runtime}; + use crate::connect::{TelemetryOperation, TelemetryProducerError, TraceRecordCompletion, TraceRecordLimits}; + + async fn wait_for_subscriber() { + tokio::time::timeout(Duration::from_secs(1), async { + while telemetry_trace_subscriber_count() == 0 { + tokio::task::yield_now().await; + } + }) + .await + .expect("telemetry subscriber"); + } + + #[tokio::test] + #[serial] + async fn local_runtime_captures_server_process_events() { + let state = tempfile::tempdir().expect("state"); + std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).expect("private state"); + let shutdown = CancellationToken::new(); + let runtime = spawn_local_trace_capture_runtime(state.path(), &shutdown).expect("runtime"); + let request_state = state.path().to_path_buf(); + let consent_expiry = SystemTime::now().duration_since(UNIX_EPOCH).expect("clock").as_secs() as i64 + 5; + let request = tokio::spawn(async move { + request_local_trace_capture( + &request_state, + consent_expiry, + TraceRecordLimits { + duration: Duration::from_millis(40), + max_spans: 8, + }, + &CancellationToken::new(), + ) + .await + }); + wait_for_subscriber().await; + assert!(telemetry_trace_emit(|| TelemetryTraceEvent::new( + TelemetryTraceOperation::GetObject, + Duration::from_micros(7), + TelemetryTraceStatus::Ok, + ))); + let capture = request.await.expect("request task").expect("capture"); + assert_eq!(capture.data.spans.len(), 1); + assert_eq!(capture.data.spans[0].operation, TelemetryOperation::GetObject); + runtime.shutdown().await; + } + + #[tokio::test] + #[serial] + async fn local_runtime_rejects_non_private_state() { + let state = tempfile::tempdir().expect("state"); + std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o755)).expect("public state"); + assert!(spawn_local_trace_capture_runtime(state.path(), &CancellationToken::new()).is_err()); + } + + #[tokio::test] + #[serial] + async fn local_runtime_enforces_consent_and_span_limits() { + let state = tempfile::tempdir().expect("state"); + std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).expect("private state"); + let shutdown = CancellationToken::new(); + let runtime = spawn_local_trace_capture_runtime(state.path(), &shutdown).expect("runtime"); + let now = SystemTime::now().duration_since(UNIX_EPOCH).expect("clock").as_secs() as i64; + + let expired = request_local_trace_capture( + state.path(), + now - 1, + TraceRecordLimits { + duration: Duration::from_millis(10), + max_spans: 1, + }, + &CancellationToken::new(), + ) + .await + .expect_err("expired consent"); + assert!(matches!( + expired, + LocalTraceCaptureError::Producer(TelemetryProducerError::ConsentExpired) + )); + + let request_state = state.path().to_path_buf(); + let request = tokio::spawn(async move { + request_local_trace_capture( + &request_state, + now + 5, + TraceRecordLimits { + duration: Duration::from_secs(1), + max_spans: 1, + }, + &CancellationToken::new(), + ) + .await + }); + wait_for_subscriber().await; + for _ in 0..3 { + assert!(telemetry_trace_emit(|| TelemetryTraceEvent::new( + TelemetryTraceOperation::InternalRpc, + Duration::from_micros(9), + TelemetryTraceStatus::Error, + ))); + } + let capture = request.await.expect("request task").expect("bounded capture"); + assert_eq!(capture.completion, TraceRecordCompletion::LimitExceeded); + assert_eq!(capture.data.spans.len(), 1); + assert_eq!(capture.data.dropped_span_count, 2); + runtime.shutdown().await; + } + + #[tokio::test] + #[serial] + async fn local_runtime_releases_capture_when_the_client_stops() { + let state = tempfile::tempdir().expect("state"); + std::fs::set_permissions(state.path(), std::fs::Permissions::from_mode(0o700)).expect("private state"); + let shutdown = CancellationToken::new(); + let runtime = spawn_local_trace_capture_runtime(state.path(), &shutdown).expect("runtime"); + let now = SystemTime::now().duration_since(UNIX_EPOCH).expect("clock").as_secs() as i64; + let request_state = state.path().to_path_buf(); + let cancel = CancellationToken::new(); + let task_cancel = cancel.clone(); + let request = tokio::spawn(async move { + request_local_trace_capture( + &request_state, + now + 5, + TraceRecordLimits { + duration: Duration::from_secs(1), + max_spans: 8, + }, + &task_cancel, + ) + .await + }); + wait_for_subscriber().await; + cancel.cancel(); + assert!(matches!( + request.await.expect("request task"), + Err(LocalTraceCaptureError::Producer(TelemetryProducerError::Cancelled)) + )); + tokio::time::timeout(Duration::from_secs(1), async { + while telemetry_trace_subscriber_count() != 0 { + tokio::task::yield_now().await; + } + }) + .await + .expect("server capture stops after disconnect"); + runtime.shutdown().await; + } +} diff --git a/rustfs/src/connect/diagnostics/trace_runtime_unsupported.rs b/rustfs/src/connect/diagnostics/trace_runtime_unsupported.rs new file mode 100644 index 000000000..af35d7742 --- /dev/null +++ b/rustfs/src/connect/diagnostics/trace_runtime_unsupported.rs @@ -0,0 +1,57 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::io; +use std::path::Path; + +use thiserror::Error; +use tokio_util::sync::CancellationToken; + +use super::{TelemetryProducerError, TraceRecordCapture, TraceRecordLimits}; + +#[derive(Debug, Error)] +pub(crate) enum LocalTraceCaptureError { + #[error("telemetry server runtime is unavailable")] + RuntimeUnavailable, + #[error("telemetry server runtime state is not owner-only")] + StateSecurity, + #[error("telemetry server runtime protocol failed")] + Protocol, + #[error("telemetry server runtime I/O failed")] + Io(#[source] io::Error), + #[error(transparent)] + Producer(#[from] TelemetryProducerError), +} + +pub(crate) struct LocalTraceCaptureRuntime; + +impl LocalTraceCaptureRuntime { + pub async fn shutdown(self) {} +} + +pub(crate) fn spawn_local_trace_capture_runtime( + _state_root: &Path, + _parent_shutdown: &CancellationToken, +) -> Result { + Err(LocalTraceCaptureError::RuntimeUnavailable) +} + +pub(crate) async fn request_local_trace_capture( + _state_root: &Path, + _consent_expires_at_unix: i64, + _limits: TraceRecordLimits, + _cancel: &CancellationToken, +) -> Result { + Err(LocalTraceCaptureError::RuntimeUnavailable) +} diff --git a/rustfs/src/connect/environment.rs b/rustfs/src/connect/environment.rs new file mode 100644 index 000000000..bdde134a4 --- /dev/null +++ b/rustfs/src/connect/environment.rs @@ -0,0 +1,864 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Bounded, identifier-free deployment environment inventory. + +use std::collections::BTreeSet; +use std::fs::{self, File, OpenOptions}; +use std::io::{Cursor, Write as _}; +use std::path::Path; +use std::sync::{Arc, LazyLock}; +use std::time::Duration; + +use base64_simd::URL_SAFE_NO_PAD; +use p256::ecdsa::{Signature, SigningKey, signature::Signer as _}; +use p256::pkcs8::DecodePrivateKey as _; +use serde::Serialize; +use sha2::{Digest as _, Sha256}; +use sysinfo::{Disks, Networks, RefreshKind, System}; +use thiserror::Error; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use tokio::sync::Semaphore; +use tokio_util::sync::CancellationToken; +use uuid::{Uuid, Variant, Version}; +use zip::{CompressionMethod, ZipWriter, write::SimpleFileOptions}; + +use super::{DeviceIdentity, inventory::InventorySnapshot}; + +pub const ENVIRONMENT_CAPABILITY: &str = "inventory.environment@1"; +pub const ENVIRONMENT_SCHEMA_VERSION: u16 = 1; +pub const MAX_ENVIRONMENT_DURATION: Duration = Duration::from_secs(30); + +static SYSTEM_SCAN_PERMIT: LazyLock> = LazyLock::new(|| Arc::new(Semaphore::new(1))); + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct EnvironmentCollectionRequest { + timeout: Duration, +} + +impl EnvironmentCollectionRequest { + pub fn negotiate(schema_version: u16, capability: &str, timeout: Duration) -> Result { + if schema_version != ENVIRONMENT_SCHEMA_VERSION { + return Err(EnvironmentError::UnsupportedVersion); + } + if capability != ENVIRONMENT_CAPABILITY { + return Err(EnvironmentError::UnsupportedCapability); + } + if timeout.is_zero() || timeout > MAX_ENVIRONMENT_DURATION { + return Err(EnvironmentError::InvalidTimeout); + } + Ok(Self { timeout }) + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize)] +#[serde(rename_all = "UPPERCASE")] +pub enum EnvironmentOsFamily { + Linux, + Darwin, + Windows, + Freebsd, + Other, +} + +impl EnvironmentOsFamily { + fn current() -> Self { + match std::env::consts::OS { + "linux" => Self::Linux, + "macos" => Self::Darwin, + "windows" => Self::Windows, + "freebsd" => Self::Freebsd, + _ => Self::Other, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize)] +#[serde(rename_all = "lowercase")] +pub enum EnvironmentFilesystemType { + Ext4, + Xfs, + Zfs, + Apfs, + Other, +} + +impl EnvironmentFilesystemType { + fn from_reported(value: &str) -> Self { + match value.to_ascii_lowercase().as_str() { + "ext4" => Self::Ext4, + "xfs" => Self::Xfs, + "zfs" => Self::Zfs, + "apfs" => Self::Apfs, + _ => Self::Other, + } + } +} + +fn filesystem_types<'a>(reported: impl IntoIterator) -> Result, EnvironmentError> { + let values = reported + .into_iter() + .map(EnvironmentFilesystemType::from_reported) + .collect::>() + .into_iter() + .collect::>(); + if values.is_empty() { + return Err(EnvironmentError::SourceUnavailable(EnvironmentSource::Filesystem)); + } + Ok(values) +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +pub struct EnvironmentInventory { + node_count: u16, + drive_count: u32, + os_family: EnvironmentOsFamily, + filesystem_types: Vec, +} + +impl EnvironmentInventory { + pub fn node_count(&self) -> u16 { + self.node_count + } + + pub fn drive_count(&self) -> u32 { + self.drive_count + } + + pub fn os_family(&self) -> EnvironmentOsFamily { + self.os_family + } + + pub fn filesystem_types(&self) -> &[EnvironmentFilesystemType] { + &self.filesystem_types + } +} + +const SIGNATURE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1\0"; +const OUTPUT_MODE: u32 = 0o600; + +#[derive(Clone, Debug)] +pub struct EnvironmentExportRequest { + pub confirmed: bool, + pub organization_name: String, + pub cluster_name: String, + pub device_name: String, + pub run_uid: String, + pub artifact_uid: String, + pub consent_uid: String, + pub policy_revision: u64, + pub produced_at_unix: i64, + pub expires_at_unix: i64, + pub nonce: [u8; 32], + pub source_commit: String, + pub executable_sha256: String, + pub rustfs_version: String, + pub build_features: Vec, +} + +#[derive(Clone, Debug)] +pub struct SignedEnvironmentExport { + pub artifact_uid: String, + pub archive_bytes: Vec, + pub archive_sha256: String, +} + +#[derive(Clone, Debug)] +pub struct SavedEnvironmentExport { + pub artifact_uid: String, + pub archive_size_bytes: u64, + pub archive_sha256: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct EnvironmentResult<'a> { + schema_version: u16, + run_uid: &'a str, + tool_id: &'static str, + capability: &'static str, + outcome: &'static str, + reason_code: &'static str, + duration_millis: u64, + provenance: EnvironmentProvenance<'a>, + coverage: EnvironmentCoverage, + data: &'a EnvironmentInventory, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct EnvironmentProvenance<'a> { + repository: &'static str, + source_commit: &'a str, + executable_sha256: &'a str, + rustfs_version: &'a str, + os_family: EnvironmentOsFamily, + architecture: &'static str, + build_features: &'a [String], +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct EnvironmentCoverage { + requested_units: u8, + completed_units: u8, + unit: &'static str, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct EnvironmentEnvelope<'a> { + format_version: &'static str, + protocol_version: &'static str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + run_uid: &'a str, + artifact_uid: &'a str, + tool_id: &'static str, + schema_version: u16, + classification: &'static str, + consent_uid: &'a str, + policy_revision: u64, + produced_at: String, + expires_at: String, + nonce: String, + device_key_id: &'a str, + payload: EnvironmentPayload, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct EnvironmentPayload { + path: &'static str, + media_type: &'static str, + size_bytes: u64, + sha256: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct EnvironmentSignature<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +pub fn sign_environment_inventory( + inventory: &EnvironmentInventory, + request: &EnvironmentExportRequest, + key: &DeviceIdentity, + duration: Duration, + cancel: &CancellationToken, +) -> Result { + if cancel.is_cancelled() { + return Err(EnvironmentError::Cancelled); + } + let now = OffsetDateTime::now_utc().unix_timestamp(); + if !request.confirmed + || duration.is_zero() + || duration > MAX_ENVIRONMENT_DURATION + || request.policy_revision == 0 + || request.produced_at_unix > now.saturating_add(300) + || request.expires_at_unix <= now + || request.expires_at_unix <= request.produced_at_unix + || request.expires_at_unix - request.produced_at_unix > 2_592_000 + || !uuid7(&request.run_uid) + || !uuid7(&request.artifact_uid) + || !uuid7(&request.consent_uid) + || !request.organization_name.starts_with("organizations/") + || !request + .cluster_name + .starts_with(&(request.organization_name.clone() + "/clusters/")) + || !request + .device_name + .starts_with(&(request.cluster_name.clone() + "/clusterDevices/")) + || !lower_hex(&request.source_commit, 40) + || !lower_hex(&request.executable_sha256, 64) + || !version(&request.rustfs_version) + || request.build_features.len() > 64 + || !request.build_features.iter().all(|feature| build_feature(feature)) + { + return Err(EnvironmentError::InvalidExport); + } + let result = EnvironmentResult { + schema_version: 1, + run_uid: &request.run_uid, + tool_id: "inventory.environment", + capability: ENVIRONMENT_CAPABILITY, + outcome: "SUCCEEDED", + reason_code: "COMPLETE", + duration_millis: u64::try_from(duration.as_millis()).unwrap_or(30_000).clamp(1, 30_000), + provenance: EnvironmentProvenance { + repository: "rustfs/rustfs", + source_commit: &request.source_commit, + executable_sha256: &request.executable_sha256, + rustfs_version: &request.rustfs_version, + os_family: EnvironmentOsFamily::current(), + architecture: match std::env::consts::ARCH { + "x86_64" => "x86_64", + "aarch64" => "aarch64", + _ => "other", + }, + build_features: &request.build_features, + }, + coverage: EnvironmentCoverage { + requested_units: 1, + completed_units: 1, + unit: "RESOURCE", + }, + data: inventory, + }; + let result_bytes = serde_json::to_vec(&result).map_err(|_| EnvironmentError::ExportEncoding)?; + let key_id = hex_lower(&Sha256::digest(key.public_key_der())); + let envelope = EnvironmentEnvelope { + format_version: "rustfs.connect.diagnosticEnvelope/1", + protocol_version: "v1", + organization_name: &request.organization_name, + cluster_name: &request.cluster_name, + device_name: &request.device_name, + run_uid: &request.run_uid, + artifact_uid: &request.artifact_uid, + tool_id: "inventory.environment", + schema_version: 1, + classification: "L1", + consent_uid: &request.consent_uid, + policy_revision: request.policy_revision, + produced_at: timestamp(request.produced_at_unix)?, + expires_at: timestamp(request.expires_at_unix)?, + nonce: URL_SAFE_NO_PAD.encode_to_string(request.nonce), + device_key_id: &key_id, + payload: EnvironmentPayload { + path: "result.json", + media_type: "application/json", + size_bytes: result_bytes.len() as u64, + sha256: hex_lower(&Sha256::digest(&result_bytes)), + }, + }; + let envelope_bytes = serde_json::to_vec(&envelope).map_err(|_| EnvironmentError::ExportEncoding)?; + let pkcs8 = key.to_pkcs8_der().map_err(|_| EnvironmentError::ExportSigning)?; + let signing_key = SigningKey::from_pkcs8_der(pkcs8.as_slice()).map_err(|_| EnvironmentError::ExportSigning)?; + let mut input = Vec::with_capacity(SIGNATURE_DOMAIN.len() + envelope_bytes.len()); + input.extend_from_slice(SIGNATURE_DOMAIN); + input.extend_from_slice(&envelope_bytes); + let signature: Signature = signing_key.sign(&input); + let signature_bytes = serde_json::to_vec(&EnvironmentSignature { + algorithm: "ES256", + key_id: &key_id, + value: URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()), + }) + .map_err(|_| EnvironmentError::ExportEncoding)?; + if cancel.is_cancelled() { + return Err(EnvironmentError::Cancelled); + } + let cursor = Cursor::new(Vec::new()); + let mut zip = ZipWriter::new(cursor); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .unix_permissions(OUTPUT_MODE); + for (name, bytes) in [ + ("envelope.json", envelope_bytes.as_slice()), + ("envelope.sig", signature_bytes.as_slice()), + ("result.json", result_bytes.as_slice()), + ] { + zip.start_file(name, options).map_err(|_| EnvironmentError::ExportEncoding)?; + zip.write_all(bytes).map_err(|_| EnvironmentError::ExportIo)?; + } + let archive_bytes = zip.finish().map_err(|_| EnvironmentError::ExportEncoding)?.into_inner(); + if archive_bytes.len() > 65_536 { + return Err(EnvironmentError::InvalidExport); + } + Ok(SignedEnvironmentExport { + artifact_uid: request.artifact_uid.clone(), + archive_sha256: hex_lower(&Sha256::digest(&archive_bytes)), + archive_bytes, + }) +} + +pub fn save_signed_environment_export( + output: &Path, + export: &SignedEnvironmentExport, + cancel: &CancellationToken, +) -> Result { + if cancel.is_cancelled() { + return Err(EnvironmentError::Cancelled); + } + let parent = output + .parent() + .filter(|p| !p.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + let temporary = parent.join(format!( + ".{}.{}.partial", + output.file_name().ok_or(EnvironmentError::InvalidExport)?.to_string_lossy(), + export.artifact_uid + )); + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(OUTPUT_MODE); + } + let mut file = options.open(&temporary).map_err(|_| EnvironmentError::ExportIo)?; + let saved = (|| { + file.write_all(&export.archive_bytes) + .map_err(|_| EnvironmentError::ExportIo)?; + if cancel.is_cancelled() { + return Err(EnvironmentError::Cancelled); + } + file.sync_all().map_err(|_| EnvironmentError::ExportIo)?; + fs::hard_link(&temporary, output).map_err(|_| EnvironmentError::ExportIo)?; + fs::remove_file(&temporary).map_err(|_| EnvironmentError::ExportIo)?; + #[cfg(unix)] + File::open(parent) + .and_then(|d| d.sync_all()) + .map_err(|_| EnvironmentError::ExportIo)?; + Ok(SavedEnvironmentExport { + artifact_uid: export.artifact_uid.clone(), + archive_size_bytes: export.archive_bytes.len() as u64, + archive_sha256: export.archive_sha256.clone(), + }) + })(); + if saved.is_err() { + let _ = fs::remove_file(&temporary); + } + saved +} + +fn timestamp(unix: i64) -> Result { + OffsetDateTime::from_unix_timestamp(unix) + .map_err(|_| EnvironmentError::InvalidExport)? + .format(&Rfc3339) + .map_err(|_| EnvironmentError::ExportEncoding) +} +fn uuid7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|u| u.get_version() == Some(Version::SortRand) && u.get_variant() == Variant::RFC4122) +} +fn lower_hex(value: &str, len: usize) -> bool { + value.len() == len && value.bytes().all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) +} + +fn version(value: &str) -> bool { + if value.is_empty() + || value.len() > 64 + || !value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'-')) + { + return false; + } + let (core, suffix) = value + .split_once('-') + .map_or((value, None), |(core, suffix)| (core, Some(suffix))); + if suffix.is_some_and(str::is_empty) { + return false; + } + let mut parts = core.split('.'); + parts.clone().count() == 3 && parts.all(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) +} + +fn build_feature(value: &str) -> bool { + !value.is_empty() + && value.len() <= 64 + && value.as_bytes()[0].is_ascii_lowercase() + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || matches!(byte, b'_' | b'-')) +} +fn hex_lower(bytes: &[u8]) -> String { + const H: &[u8; 16] = b"0123456789abcdef"; + let mut out = String::with_capacity(bytes.len() * 2); + for b in bytes { + out.push(H[(b >> 4) as usize] as char); + out.push(H[(b & 15) as usize] as char); + } + out +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum EnvironmentSource { + Filesystem, + Cpu, + Memory, + Network, +} + +#[derive(Debug, Error, PartialEq, Eq)] +pub enum EnvironmentError { + #[error("inventory_environment_unsupported_version")] + UnsupportedVersion, + #[error("inventory_environment_unsupported_capability")] + UnsupportedCapability, + #[error("inventory_environment_invalid_timeout")] + InvalidTimeout, + #[error("inventory_environment_source_unavailable")] + SourceUnavailable(EnvironmentSource), + #[error("inventory_environment_cancelled")] + Cancelled, + #[error("inventory_environment_timed_out")] + TimedOut, + #[error("inventory_environment_task_failed")] + TaskFailed, + #[error("inventory_environment_invalid_export")] + InvalidExport, + #[error("inventory_environment_export_encoding_failed")] + ExportEncoding, + #[error("inventory_environment_export_signing_failed")] + ExportSigning, + #[error("inventory_environment_export_io_failed")] + ExportIo, +} + +#[derive(Debug, PartialEq, Eq)] +pub(crate) struct HostEnvironment { + pub(crate) os_summary: String, + pub(crate) kernel_summary: String, + pub(crate) architecture: &'static str, + pub(crate) cores: usize, + pub(crate) total_memory_bytes: u64, + pub(crate) under_memory_pressure: bool, + pub(crate) filesystem_types: Vec, + pub(crate) interface_count: usize, + pub(crate) bond_count: usize, +} + +impl HostEnvironment { + fn collect() -> Result { + #[cfg(test)] + let scan = test_support::ScanGuard::start(); + #[cfg(test)] + if scan.controlled() { + return Ok(Self { + os_summary: "test-os".to_owned(), + kernel_summary: "test-kernel".to_owned(), + architecture: std::env::consts::ARCH, + cores: 1, + total_memory_bytes: 1, + under_memory_pressure: false, + filesystem_types: vec![EnvironmentFilesystemType::Other], + interface_count: 1, + bond_count: 0, + }); + } + + // Processes, names, addresses, paths, mount options and device labels are outside this schema. + let system = System::new_with_specifics(RefreshKind::everything().without_processes()); + let cores = system.cpus().len(); + if cores == 0 { + return Err(EnvironmentError::SourceUnavailable(EnvironmentSource::Cpu)); + } + let total_memory_bytes = system.total_memory(); + if total_memory_bytes == 0 { + return Err(EnvironmentError::SourceUnavailable(EnvironmentSource::Memory)); + } + let available_memory = system.available_memory(); + let disks = Disks::new_with_refreshed_list(); + let reported_filesystems = disks + .iter() + .map(|disk| disk.file_system().to_string_lossy()) + .collect::>(); + let filesystem_types = filesystem_types(reported_filesystems.iter().map(AsRef::as_ref))?; + let networks = Networks::new_with_refreshed_list(); + if networks.is_empty() { + return Err(EnvironmentError::SourceUnavailable(EnvironmentSource::Network)); + } + + Ok(Self { + os_summary: System::long_os_version().unwrap_or_else(|| "unknown".to_owned()), + kernel_summary: System::kernel_long_version(), + architecture: std::env::consts::ARCH, + cores, + total_memory_bytes, + under_memory_pressure: available_memory.saturating_mul(10) < total_memory_bytes, + filesystem_types, + interface_count: networks.len(), + bond_count: networks.keys().filter(|name| name.starts_with("bond")).count(), + }) + } +} + +pub async fn collect_environment( + inventory: &InventorySnapshot, + request: EnvironmentCollectionRequest, + cancel: &CancellationToken, +) -> Result { + let host = collect_host_environment(request.timeout, cancel).await?; + Ok(EnvironmentInventory { + node_count: inventory.node_count(), + drive_count: inventory.drive_count(), + os_family: EnvironmentOsFamily::current(), + filesystem_types: host.filesystem_types, + }) +} + +pub(crate) async fn collect_host_environment( + timeout: Duration, + cancel: &CancellationToken, +) -> Result { + let deadline = tokio::time::Instant::now() + timeout; + let permit = tokio::select! { + biased; + () = cancel.cancelled() => return Err(EnvironmentError::Cancelled), + result = tokio::time::timeout_at(deadline, SYSTEM_SCAN_PERMIT.clone().acquire_owned()) => { + match result { + Ok(Ok(permit)) => permit, + Ok(Err(_)) => return Err(EnvironmentError::TaskFailed), + Err(_) => return Err(EnvironmentError::TimedOut), + } + } + }; + let task = tokio::task::spawn_blocking(move || { + let _permit = permit; + HostEnvironment::collect() + }); + tokio::select! { + biased; + () = cancel.cancelled() => Err(EnvironmentError::Cancelled), + result = tokio::time::timeout_at(deadline, task) => { + match result { + Ok(Ok(environment)) => environment, + Ok(Err(_)) => Err(EnvironmentError::TaskFailed), + Err(_) => Err(EnvironmentError::TimedOut), + } + } + } +} + +#[cfg(test)] +mod test_support { + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::{Arc, Condvar, Mutex}; + + use tokio::sync::Semaphore; + + pub(super) static ACTIVE: AtomicUsize = AtomicUsize::new(0); + pub(super) static MAX_ACTIVE: AtomicUsize = AtomicUsize::new(0); + static CONTROLLED_SCAN: Mutex>> = Mutex::new(None); + + struct ScanBarrier { + reached: Semaphore, + completed: Semaphore, + released: Mutex, + release: Condvar, + } + + impl ScanBarrier { + fn new() -> Self { + Self { + reached: Semaphore::new(0), + completed: Semaphore::new(0), + released: Mutex::new(false), + release: Condvar::new(), + } + } + + fn wait(&self) { + self.reached.add_permits(1); + let mut released = self.released.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + while !*released { + released = self.release.wait(released).unwrap_or_else(|poisoned| poisoned.into_inner()); + } + } + + fn release(&self) { + *self.released.lock().unwrap_or_else(|poisoned| poisoned.into_inner()) = true; + self.release.notify_all(); + } + } + + pub(super) struct ControlledScan { + barrier: Arc, + } + + impl ControlledScan { + pub(super) async fn wait_until_reached(&self) { + let permit = self + .barrier + .reached + .acquire() + .await + .expect("controlled scan barrier should stay open"); + permit.forget(); + } + + pub(super) fn release(&self) { + self.barrier.release(); + } + + pub(super) async fn wait_until_completed(&self) { + let permit = self + .barrier + .completed + .acquire() + .await + .expect("controlled scan completion should stay open"); + permit.forget(); + } + } + + impl Drop for ControlledScan { + fn drop(&mut self) { + self.barrier.release(); + let mut controlled = CONTROLLED_SCAN.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + if controlled.as_ref().is_some_and(|barrier| Arc::ptr_eq(barrier, &self.barrier)) { + controlled.take(); + } + } + } + + pub(super) fn control_next_scan() -> ControlledScan { + let barrier = Arc::new(ScanBarrier::new()); + let mut controlled = CONTROLLED_SCAN.lock().unwrap_or_else(|poisoned| poisoned.into_inner()); + assert!(controlled.is_none(), "only one controlled system scan may be installed"); + *controlled = Some(Arc::clone(&barrier)); + ControlledScan { barrier } + } + + pub(super) struct ScanGuard { + barrier: Option>, + } + + impl ScanGuard { + pub(super) fn start() -> Self { + let active = ACTIVE.fetch_add(1, Ordering::SeqCst) + 1; + MAX_ACTIVE.fetch_max(active, Ordering::SeqCst); + let barrier = CONTROLLED_SCAN + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()) + .clone(); + if let Some(barrier) = &barrier { + barrier.wait(); + } + Self { barrier } + } + + pub(super) fn controlled(&self) -> bool { + self.barrier.is_some() + } + } + + impl Drop for ScanGuard { + fn drop(&mut self) { + ACTIVE.fetch_sub(1, Ordering::SeqCst); + if let Some(barrier) = &self.barrier { + barrier.completed.add_permits(1); + } + } + } +} + +#[cfg(test)] +mod tests { + use std::io::Read as _; + use std::sync::atomic::Ordering; + + use super::*; + + #[tokio::test(start_paused = true)] + async fn timed_out_and_cancelled_scans_remain_single_flight() { + test_support::MAX_ACTIVE.store(0, Ordering::SeqCst); + let controlled = test_support::control_next_scan(); + + let first = tokio::spawn(async { + let cancel = CancellationToken::new(); + collect_host_environment(Duration::from_millis(20), &cancel).await + }); + controlled.wait_until_reached().await; + tokio::time::advance(Duration::from_millis(20)).await; + assert_eq!(first.await.expect("first scan"), Err(EnvironmentError::TimedOut)); + assert_eq!(test_support::ACTIVE.load(Ordering::SeqCst), 1); + assert_eq!(SYSTEM_SCAN_PERMIT.available_permits(), 0); + + let second_cancel = CancellationToken::new(); + let second = collect_host_environment(Duration::from_secs(1), &second_cancel); + tokio::pin!(second); + assert!(futures::poll!(second.as_mut()).is_pending()); + assert_eq!(test_support::MAX_ACTIVE.load(Ordering::SeqCst), 1); + second_cancel.cancel(); + assert_eq!(second.await, Err(EnvironmentError::Cancelled)); + controlled.release(); + controlled.wait_until_completed().await; + assert_eq!(test_support::ACTIVE.load(Ordering::SeqCst), 0); + } + + #[test] + fn filesystem_projection_drops_paths_options_and_secret_like_values() { + assert_eq!( + filesystem_types(["xfs", "ext4", "/srv/customer-a", "rw,password=SYNTHETIC_SECRET_123"]), + Ok(vec![ + EnvironmentFilesystemType::Ext4, + EnvironmentFilesystemType::Xfs, + EnvironmentFilesystemType::Other, + ]) + ); + assert_eq!( + filesystem_types(std::iter::empty()), + Err(EnvironmentError::SourceUnavailable(EnvironmentSource::Filesystem)) + ); + } + + #[test] + fn signed_export_contains_only_the_allow_list_and_never_clobbers() { + let inventory = EnvironmentInventory { + node_count: 1, + drive_count: 0, + os_family: EnvironmentOsFamily::Linux, + filesystem_types: vec![EnvironmentFilesystemType::Xfs], + }; + let now = OffsetDateTime::now_utc().unix_timestamp(); + let request = EnvironmentExportRequest { + confirmed: true, + organization_name: "organizations/019e3ae0-0000-7000-8000-000000000001".into(), + cluster_name: "organizations/019e3ae0-0000-7000-8000-000000000001/clusters/019e3ae0-0000-7000-8000-000000000002".into(), + device_name: "organizations/019e3ae0-0000-7000-8000-000000000001/clusters/019e3ae0-0000-7000-8000-000000000002/clusterDevices/019e3ae0-0000-7000-8000-000000000003".into(), + run_uid: "019e3ae0-0000-7000-8000-000000000004".into(), + artifact_uid: "019e3ae0-0000-7000-8000-000000000005".into(), + consent_uid: "019e3ae0-0000-7000-8000-000000000006".into(), + policy_revision: 1, + produced_at_unix: now, + expires_at_unix: now + 60, + nonce: [7; 32], + source_commit: "a".repeat(40), + executable_sha256: "b".repeat(64), + rustfs_version: "1.0.0-rc.6".into(), + build_features: vec![], + }; + let cancel = CancellationToken::new(); + let mut unconfirmed = request.clone(); + unconfirmed.confirmed = false; + assert!(matches!( + sign_environment_inventory(&inventory, &unconfirmed, &DeviceIdentity::generate(), Duration::from_millis(1), &cancel,), + Err(EnvironmentError::InvalidExport) + )); + let export = + sign_environment_inventory(&inventory, &request, &DeviceIdentity::generate(), Duration::from_millis(1), &cancel) + .expect("signed export"); + let mut zip = zip::ZipArchive::new(Cursor::new(export.archive_bytes.as_slice())).expect("zip"); + let mut result = String::new(); + zip.by_name("result.json") + .expect("result") + .read_to_string(&mut result) + .expect("read"); + let value: serde_json::Value = serde_json::from_str(&result).expect("json"); + assert_eq!(value["data"]["driveCount"], 0); + assert_eq!(value["data"].as_object().expect("data").len(), 4); + assert!(!result.contains("secret")); + + let directory = tempfile::tempdir().expect("tempdir"); + let output = directory.path().join("environment.zip"); + save_signed_environment_export(&output, &export, &cancel).expect("save"); + assert!(save_signed_environment_export(&output, &export, &cancel).is_err()); + } +} diff --git a/rustfs/src/connect/heartbeat.rs b/rustfs/src/connect/heartbeat.rs index f71de4c2f..247d8242b 100644 --- a/rustfs/src/connect/heartbeat.rs +++ b/rustfs/src/connect/heartbeat.rs @@ -24,6 +24,8 @@ use uuid::Uuid; use super::config::HeartbeatConfig; use super::credential_store::CredentialStoreError; +use super::diagnostics::{CONNECT_DIAGNOSTIC_CAPABILITIES, DiagnosticCollectionPolicy, DiagnosticJobEnvelope}; +use super::environment::ENVIRONMENT_CAPABILITY; use super::identity::IdentityError; use super::identity_store::StoreError; use super::registration::CredentialValidationError; @@ -82,7 +84,7 @@ pub(crate) struct PendingHeartbeat { protocol_version: String, request_id: String, agent_version: String, - capabilities: [String; 1], + capabilities: Vec, sequence: u64, client_time: String, coarse_node_summary: CoarseNodeSummary, @@ -92,7 +94,24 @@ impl PendingHeartbeat { fn is_valid(&self) -> bool { self.protocol_version == PROTOCOL_VERSION && self.agent_version == AGENT_VERSION - && self.capabilities[0] == "heartbeat" + && (self.capabilities == ["heartbeat"] + || self.capabilities == ["heartbeat", DiagnosticCollectionPolicy::policy_sync_capability()] + || self.capabilities + == [ + "heartbeat", + DiagnosticCollectionPolicy::policy_sync_capability(), + ENVIRONMENT_CAPABILITY, + ] + || self.capabilities + == [ + "heartbeat", + DiagnosticCollectionPolicy::policy_sync_capability(), + ENVIRONMENT_CAPABILITY, + "jobs", + super::diagnostics::CPU_PROFILE_CAPABILITY, + ] + || self.capabilities == heartbeat_capabilities(false) + || self.capabilities == heartbeat_capabilities(true)) && self.sequence <= MAX_SEQUENCE && self.coarse_node_summary.is_valid() && is_exact_utc_seconds(&self.client_time) @@ -108,13 +127,29 @@ struct HeartbeatResponse { accepted_version: String, #[serde(default)] capability_hints: Vec, + #[serde(default)] + diagnostic_collection_policy: Option, + #[serde(default)] + diagnostic_job: Option, } pub(crate) enum Delivery { - Accepted { server_time: String }, - Retry { retry_after: Option }, - AuthenticationStopped { status: u16, reason: Option }, - Rejected { status: u16, reason: Option }, + Accepted { + server_time: String, + diagnostic_collection_policy: DiagnosticCollectionPolicy, + diagnostic_job: Option>, + }, + Retry { + retry_after: Option, + }, + AuthenticationStopped { + status: u16, + reason: Option, + }, + Rejected { + status: u16, + reason: Option, + }, } pub(crate) struct HeartbeatSender { @@ -143,8 +178,14 @@ impl HeartbeatSender { { return Err(HeartbeatError::Response); } + let policy = accepted + .diagnostic_collection_policy + .unwrap_or_else(DiagnosticCollectionPolicy::stopped); + policy.validate().map_err(|_| HeartbeatError::Response)?; Ok(Delivery::Accepted { server_time: accepted.server_time, + diagnostic_collection_policy: policy, + diagnostic_job: accepted.diagnostic_job.map(Box::new), }) } TelemetryDelivery::Retry { retry_after } => Ok(Delivery::Retry { retry_after }), @@ -157,6 +198,7 @@ impl HeartbeatSender { #[derive(Clone)] pub(crate) struct HeartbeatStateStore { path: PathBuf, + job_capable: bool, } #[derive(Default, Serialize, Deserialize)] @@ -167,8 +209,8 @@ struct HeartbeatState { } impl HeartbeatStateStore { - pub(crate) fn new(path: PathBuf) -> Self { - Self { path } + pub(crate) fn new(path: PathBuf, job_capable: bool) -> Self { + Self { path, job_capable } } pub(crate) fn try_runtime_lock(&self) -> Result { @@ -216,11 +258,12 @@ impl HeartbeatStateStore { if state.next_sequence > MAX_SEQUENCE { return Err(HeartbeatError::SequenceExhausted); } + let capabilities = heartbeat_capabilities(self.job_capable); let pending = PendingHeartbeat { protocol_version: PROTOCOL_VERSION.to_owned(), request_id: Uuid::new_v4().to_string(), agent_version: AGENT_VERSION.to_owned(), - capabilities: ["heartbeat".to_owned()], + capabilities, sequence: state.next_sequence, client_time: now.to_rfc3339_opts(SecondsFormat::Secs, true), coarse_node_summary: summary, @@ -280,6 +323,23 @@ impl HeartbeatStateStore { } } +fn heartbeat_capabilities(job_capable: bool) -> Vec { + let mut capabilities = vec![ + "heartbeat".to_owned(), + DiagnosticCollectionPolicy::policy_sync_capability().to_owned(), + ENVIRONMENT_CAPABILITY.to_owned(), + ]; + if job_capable { + capabilities.push("jobs".to_owned()); + } + capabilities.extend( + CONNECT_DIAGNOSTIC_CAPABILITIES + .iter() + .map(|capability| (*capability).to_owned()), + ); + capabilities +} + fn parent(path: &Path) -> Result<&Path, HeartbeatError> { path.parent() .ok_or_else(|| state_io(path, io::Error::new(io::ErrorKind::InvalidInput, "state path has no parent"))) @@ -364,6 +424,16 @@ pub enum HeartbeatError { Endpoint, #[error("Connect heartbeat root CA configuration is invalid")] RootCertificate, + #[error("Connect heartbeat proxy configuration is invalid")] + ProxyConfiguration, + #[error("Connect proxy authentication failed; verify the configured proxy credential files")] + ProxyAuthentication, + #[error( + "Connect proxy connection failed; verify proxy availability, credentials, the proxy allow-list, and the Connect endpoint" + )] + ProxyRejected, + #[error("Connect TLS peer certificate validation failed; verify the endpoint and configured root CA")] + TlsPeer, #[error("Connect heartbeat schedule is invalid")] Schedule, #[error("RustFS is not registered with Connect")] @@ -424,6 +494,10 @@ impl From for HeartbeatError { match error { TelemetryError::Endpoint => Self::Endpoint, TelemetryError::RootCertificate => Self::RootCertificate, + TelemetryError::ProxyConfiguration => Self::ProxyConfiguration, + TelemetryError::ProxyAuthentication => Self::ProxyAuthentication, + TelemetryError::ProxyRejected => Self::ProxyRejected, + TelemetryError::TlsPeer => Self::TlsPeer, TelemetryError::Schedule => Self::Schedule, TelemetryError::NotRegistered => Self::NotRegistered, TelemetryError::IdentityMissing => Self::IdentityMissing, diff --git a/rustfs/src/connect/license.rs b/rustfs/src/connect/license.rs new file mode 100644 index 000000000..a8c1df4cf --- /dev/null +++ b/rustfs/src/connect/license.rs @@ -0,0 +1,774 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Verification and local persistence for RustFS Connect service licenses. +//! +//! Service licenses are separate from Connect device credentials. This module +//! has no signing capability and does not affect S3 availability. + +use std::fs; +use std::io::{self, Write as _}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::{SystemTime, UNIX_EPOCH}; + +use base64_simd::URL_SAFE_NO_PAD; +use ed25519_dalek::{Signature, VerifyingKey}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use time::OffsetDateTime; +use time::format_description::well_known::Rfc3339; + +pub const LICENSE_PURPOSE: &str = "RUSTFS_CONNECT_SERVICE_LICENSE"; +pub const LICENSE_SCHEMA: &str = "rustfs.connect.serviceLicense/1"; +pub const LICENSE_ALGORITHM: &str = "Ed25519"; +pub const LICENSE_DOMAIN_SEPARATION_TAG: &str = "rustfs-connect-service-license-v1"; + +const INSTALLED_SCHEMA: &str = "rustfs.connect.installedServiceLicense/1"; +const MAX_ARTIFACT_BYTES: u64 = 64 * 1024; +const MAX_STATE_BYTES: u64 = MAX_ARTIFACT_BYTES + 1024; +const MAX_PAYLOAD_BYTES: usize = 32 * 1024; +const MAX_PUBLIC_KEY_BYTES: u64 = 256; +const LOCK_FILE: &str = ".service-license.lock"; + +#[cfg(unix)] +const STATE_FILE_MODE: u32 = 0o600; + +static STAGING_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum LicenseArtifactStatus { + Valid, + Missing, + InvalidConfiguration, + InvalidArtifact, + Unsupported, + UntrustedKey, + InvalidSignature, + WrongScope, + NotYetValid, + Expired, + Rollback, + SequenceConflict, + SupersessionRequired, + StateUnavailable, +} + +impl LicenseArtifactStatus { + pub const fn as_str(self) -> &'static str { + match self { + Self::Valid => "VALID", + Self::Missing => "MISSING", + Self::InvalidConfiguration => "INVALID_CONFIGURATION", + Self::InvalidArtifact => "INVALID_ARTIFACT", + Self::Unsupported => "UNSUPPORTED", + Self::UntrustedKey => "UNTRUSTED_KEY", + Self::InvalidSignature => "INVALID_SIGNATURE", + Self::WrongScope => "WRONG_SCOPE", + Self::NotYetValid => "NOT_YET_VALID", + Self::Expired => "EXPIRED", + Self::Rollback => "ROLLBACK", + Self::SequenceConflict => "SEQUENCE_CONFLICT", + Self::SupersessionRequired => "SUPERSESSION_REQUIRED", + Self::StateUnavailable => "STATE_UNAVAILABLE", + } + } +} + +impl std::fmt::Display for LicenseArtifactStatus { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str(self.as_str()) + } +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +pub struct LicenseClaims { + pub purpose: String, + pub schema: String, + pub algorithm: String, + pub key_id: String, + pub license_uid: String, + pub grant_uid: String, + pub sequence: u64, + pub issuer: String, + pub audience: String, + pub organization: String, + pub deployment: String, + pub plan_code: String, + pub policy_revision: String, + pub service_code: String, + pub issue_time: String, + pub not_before: String, + pub expire_time: String, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct LicenseReport { + pub status: LicenseArtifactStatus, + pub installed: bool, + pub idempotent: bool, + pub license: Option, + pub message: Option, +} + +impl LicenseReport { + pub const fn is_valid(&self) -> bool { + matches!(self.status, LicenseArtifactStatus::Valid) + } + + fn valid(license: LicenseClaims, installed: bool, idempotent: bool) -> Self { + Self { + status: LicenseArtifactStatus::Valid, + installed, + idempotent, + license: Some(license), + message: None, + } + } +} + +#[derive(Debug, thiserror::Error)] +#[error("{status}: {message}")] +pub struct LicenseArtifactError { + pub status: LicenseArtifactStatus, + pub message: String, + pub license: Option>, +} + +impl LicenseArtifactError { + pub fn report(self, installed: bool) -> LicenseReport { + LicenseReport { + status: self.status, + installed, + idempotent: false, + license: self.license.map(|license| *license), + message: Some(self.message), + } + } +} + +#[derive(Clone, Debug)] +pub struct LicenseVerificationContext { + verifying_key: VerifyingKey, + pub key_id: String, + pub issuer: String, + pub audience: String, + pub organization: String, + pub deployment: String, + pub service_code: String, + pub now_unix: i64, +} + +impl LicenseVerificationContext { + #[allow(clippy::too_many_arguments)] + pub fn from_public_key_file( + public_key_file: &Path, + key_id: String, + issuer: String, + audience: String, + organization: String, + deployment: String, + service_code: String, + ) -> Result { + let encoded = read_bounded_regular_file( + public_key_file, + MAX_PUBLIC_KEY_BYTES, + LicenseArtifactStatus::InvalidConfiguration, + "public key", + )?; + let encoded = std::str::from_utf8(&encoded) + .map_err(|_| failure(LicenseArtifactStatus::InvalidConfiguration, "the trusted public key is not UTF-8"))? + .trim(); + let public_key = decode_canonical_base64url(encoded, 32, LicenseArtifactStatus::InvalidConfiguration, "public key")?; + let public_key: [u8; 32] = public_key.try_into().map_err(|_| { + failure( + LicenseArtifactStatus::InvalidConfiguration, + "the trusted public key must contain 32 bytes", + ) + })?; + let now_unix = system_now()?; + Self::new(public_key, key_id, issuer, audience, organization, deployment, service_code, now_unix) + } + + #[allow(clippy::too_many_arguments)] + pub fn new( + public_key: [u8; 32], + key_id: String, + issuer: String, + audience: String, + organization: String, + deployment: String, + service_code: String, + now_unix: i64, + ) -> Result { + let verifying_key = VerifyingKey::from_bytes(&public_key) + .map_err(|_| failure(LicenseArtifactStatus::InvalidConfiguration, "the trusted Ed25519 public key is invalid"))?; + let actual_key_id = hex_simd::encode_to_string(Sha256::digest(public_key), hex_simd::AsciiCase::Lower); + if !is_lower_hex_sha256(&key_id) || key_id != actual_key_id { + return Err(failure( + LicenseArtifactStatus::InvalidConfiguration, + "the trusted key ID does not match the public key", + )); + } + if !is_bounded_label(&issuer, 128) || !is_bounded_label(&audience, 128) { + return Err(failure( + LicenseArtifactStatus::InvalidConfiguration, + "issuer and audience must be bounded identifiers", + )); + } + validate_resource_scope(&organization, &deployment).map_err(|message| { + failure( + LicenseArtifactStatus::InvalidConfiguration, + format!("the local license scope is invalid: {message}"), + ) + })?; + if !is_service_code(&service_code) { + return Err(failure(LicenseArtifactStatus::InvalidConfiguration, "the local service code is invalid")); + } + + Ok(Self { + verifying_key, + key_id, + issuer, + audience, + organization, + deployment, + service_code, + now_unix, + }) + } +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct LicenseArtifact { + payload: String, + signature: String, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +struct InstalledLicense { + schema: String, + artifact: LicenseArtifact, +} + +struct ValidatedLicense { + artifact: LicenseArtifact, + claims: LicenseClaims, +} + +pub fn apply_license_artifact( + artifact_path: &Path, + state_directory: &Path, + context: &LicenseVerificationContext, +) -> Result { + let artifact_bytes = read_license_artifact_file(artifact_path)?; + apply_license_bytes(&artifact_bytes, state_directory, context) +} + +pub(super) fn apply_license_bytes( + artifact_bytes: &[u8], + state_directory: &Path, + context: &LicenseVerificationContext, +) -> Result { + fs::create_dir_all(state_directory).map_err(|source| state_io(state_directory, source))?; + let _lock = lock_state(state_directory)?; + let candidate = validate_artifact(parse_artifact(artifact_bytes)?, context, true)?; + let state_path = state_path(state_directory, context); + + if let Some(current) = load_installed(&state_path, context)? { + match compare_sequence(&candidate, ¤t)? { + SequenceDecision::Idempotent => return Ok(LicenseReport::valid(candidate.claims, true, true)), + SequenceDecision::Replace => {} + } + } + + persist_installed(&state_path, &candidate.artifact)?; + Ok(LicenseReport::valid(candidate.claims, true, false)) +} + +pub fn verify_license_artifact( + artifact_path: &Path, + state_directory: &Path, + context: &LicenseVerificationContext, +) -> Result { + let artifact_bytes = read_license_artifact_file(artifact_path)?; + verify_license_bytes(&artifact_bytes, state_directory, context) +} + +pub(super) fn verify_license_bytes( + artifact_bytes: &[u8], + state_directory: &Path, + context: &LicenseVerificationContext, +) -> Result { + let candidate = validate_artifact(parse_artifact(artifact_bytes)?, context, true)?; + let state_path = state_path(state_directory, context); + if let Some(current) = load_installed(&state_path, context)? + && matches!(compare_sequence(&candidate, ¤t)?, SequenceDecision::Idempotent) + { + return Ok(LicenseReport::valid(candidate.claims, true, true)); + } + Ok(LicenseReport::valid(candidate.claims, false, false)) +} + +pub fn inspect_installed_license( + state_directory: &Path, + context: &LicenseVerificationContext, +) -> Result { + let state_path = state_path(state_directory, context); + let artifact = read_installed_artifact(&state_path)?.ok_or_else(|| { + failure( + LicenseArtifactStatus::Missing, + "no service license is installed for the requested deployment and service", + ) + })?; + let validated = validate_artifact(artifact, context, true).map_err(|mut error| { + if !matches!(error.status, LicenseArtifactStatus::Expired | LicenseArtifactStatus::NotYetValid) { + error.status = LicenseArtifactStatus::StateUnavailable; + error.message = "the installed service license could not be verified".to_owned(); + error.license = None; + } + error + })?; + Ok(LicenseReport::valid(validated.claims, true, false)) +} + +enum SequenceDecision { + Idempotent, + Replace, +} + +fn compare_sequence(candidate: &ValidatedLicense, current: &ValidatedLicense) -> Result { + if candidate.claims.grant_uid != current.claims.grant_uid { + return Err(failure_with_license( + LicenseArtifactStatus::SupersessionRequired, + "a different grant cannot replace the installed grant without a signed supersession claim", + candidate.claims.clone(), + )); + } + match candidate.claims.sequence.cmp(¤t.claims.sequence) { + std::cmp::Ordering::Less => Err(failure_with_license( + LicenseArtifactStatus::Rollback, + "the license sequence is older than the installed sequence", + candidate.claims.clone(), + )), + std::cmp::Ordering::Equal if candidate.artifact == current.artifact => Ok(SequenceDecision::Idempotent), + std::cmp::Ordering::Equal => Err(failure_with_license( + LicenseArtifactStatus::SequenceConflict, + "different license bytes use the installed sequence", + candidate.claims.clone(), + )), + std::cmp::Ordering::Greater => Ok(SequenceDecision::Replace), + } +} + +fn validate_artifact( + artifact: LicenseArtifact, + context: &LicenseVerificationContext, + check_time: bool, +) -> Result { + let payload = + decode_canonical_base64url(&artifact.payload, MAX_PAYLOAD_BYTES, LicenseArtifactStatus::InvalidArtifact, "payload")?; + let signature = decode_canonical_base64url(&artifact.signature, 64, LicenseArtifactStatus::InvalidArtifact, "signature")?; + let claims: LicenseClaims = serde_json::from_slice(&payload) + .map_err(|_| failure(LicenseArtifactStatus::InvalidArtifact, "the license payload is invalid"))?; + if !matches!(serde_json::to_vec(&claims), Ok(canonical) if canonical == payload) { + return Err(failure( + LicenseArtifactStatus::InvalidArtifact, + "the license payload is not canonical JSON", + )); + } + validate_claim_shape(&claims)?; + if claims.purpose != LICENSE_PURPOSE || claims.schema != LICENSE_SCHEMA || claims.algorithm != LICENSE_ALGORITHM { + return Err(failure( + LicenseArtifactStatus::Unsupported, + "the license purpose, schema, or algorithm is unsupported", + )); + } + if claims.key_id != context.key_id { + return Err(failure(LicenseArtifactStatus::UntrustedKey, "the license key ID is not trusted")); + } + + let signature = Signature::from_slice(&signature) + .map_err(|_| failure(LicenseArtifactStatus::InvalidArtifact, "the license signature has an invalid length"))?; + let mut signed = Vec::with_capacity(LICENSE_DOMAIN_SEPARATION_TAG.len() + 1 + payload.len()); + signed.extend_from_slice(LICENSE_DOMAIN_SEPARATION_TAG.as_bytes()); + signed.push(0); + signed.extend_from_slice(&payload); + context + .verifying_key + .verify_strict(&signed, &signature) + .map_err(|_| failure(LicenseArtifactStatus::InvalidSignature, "the license signature is invalid"))?; + + if claims.issuer != context.issuer + || claims.audience != context.audience + || claims.organization != context.organization + || claims.deployment != context.deployment + || claims.service_code != context.service_code + { + return Err(failure( + LicenseArtifactStatus::WrongScope, + "the license is not valid for the requested scope", + )); + } + + if check_time { + let not_before = parse_timestamp(&claims.not_before)?; + let expire_time = parse_timestamp(&claims.expire_time)?; + if context.now_unix < not_before { + return Err(failure_with_license( + LicenseArtifactStatus::NotYetValid, + "the license is not yet valid", + claims, + )); + } + if context.now_unix >= expire_time { + return Err(failure_with_license(LicenseArtifactStatus::Expired, "the license has expired", claims)); + } + } + + Ok(ValidatedLicense { artifact, claims }) +} + +fn validate_claim_shape(claims: &LicenseClaims) -> Result<(), LicenseArtifactError> { + if !is_lower_hex_sha256(&claims.key_id) + || !is_canonical_uuid_v7(&claims.license_uid) + || !is_canonical_uuid_v7(&claims.grant_uid) + || claims.sequence == 0 + || !is_bounded_label(&claims.issuer, 128) + || !is_bounded_label(&claims.audience, 128) + || !is_service_code(&claims.plan_code) + || !is_bounded_label(&claims.policy_revision, 128) + || !is_service_code(&claims.service_code) + { + return Err(failure( + LicenseArtifactStatus::InvalidArtifact, + "the license claims contain an invalid identifier", + )); + } + validate_resource_scope(&claims.organization, &claims.deployment) + .map_err(|_| failure(LicenseArtifactStatus::InvalidArtifact, "the license resource scope is invalid"))?; + let issue_time = parse_timestamp(&claims.issue_time)?; + let not_before = parse_timestamp(&claims.not_before)?; + let expire_time = parse_timestamp(&claims.expire_time)?; + if issue_time > not_before || issue_time >= expire_time || not_before >= expire_time { + return Err(failure(LicenseArtifactStatus::InvalidArtifact, "the license time interval is invalid")); + } + Ok(()) +} + +fn parse_timestamp(value: &str) -> Result { + if value.len() != 20 || !value.ends_with('Z') { + return Err(failure( + LicenseArtifactStatus::InvalidArtifact, + "license timestamps must use whole-second UTC RFC 3339", + )); + } + OffsetDateTime::parse(value, &Rfc3339) + .map(|time| time.unix_timestamp()) + .map_err(|_| failure(LicenseArtifactStatus::InvalidArtifact, "the license timestamp is invalid")) +} + +pub(super) fn read_license_artifact_file(path: &Path) -> Result, LicenseArtifactError> { + read_bounded_regular_file(path, MAX_ARTIFACT_BYTES, LicenseArtifactStatus::InvalidArtifact, "license artifact") +} + +fn parse_artifact(bytes: &[u8]) -> Result { + if bytes.is_empty() || bytes.len() as u64 > MAX_ARTIFACT_BYTES { + return Err(failure( + LicenseArtifactStatus::InvalidArtifact, + format!("the license artifact must be no larger than {MAX_ARTIFACT_BYTES} bytes"), + )); + } + serde_json::from_slice(bytes).map_err(|_| failure(LicenseArtifactStatus::InvalidArtifact, "the license artifact is invalid")) +} + +fn read_bounded_regular_file( + path: &Path, + maximum: u64, + status: LicenseArtifactStatus, + label: &str, +) -> Result, LicenseArtifactError> { + let metadata = fs::metadata(path).map_err(|source| failure(status, format!("the {label} could not be read: {source}")))?; + if !metadata.is_file() || metadata.len() > maximum { + return Err(failure( + status, + format!("the {label} must be a regular file no larger than {maximum} bytes"), + )); + } + fs::read(path).map_err(|source| failure(status, format!("the {label} could not be read: {source}"))) +} + +fn load_installed(path: &Path, context: &LicenseVerificationContext) -> Result, LicenseArtifactError> { + let Some(artifact) = read_installed_artifact(path)? else { + return Ok(None); + }; + validate_artifact(artifact, context, false).map(Some).map_err(|_| { + failure( + LicenseArtifactStatus::StateUnavailable, + "the installed service license could not be verified and was left untouched", + ) + }) +} + +fn read_installed_artifact(path: &Path) -> Result, LicenseArtifactError> { + let bytes = match fs::read(path) { + Ok(bytes) => bytes, + Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(None), + Err(source) => return Err(state_io(path, source)), + }; + check_state_mode(path)?; + if bytes.len() as u64 > MAX_STATE_BYTES { + return Err(failure( + LicenseArtifactStatus::StateUnavailable, + "the installed service license exceeds the size limit", + )); + } + let installed: InstalledLicense = serde_json::from_slice(&bytes) + .map_err(|_| failure(LicenseArtifactStatus::StateUnavailable, "the installed service license state is invalid"))?; + if installed.schema != INSTALLED_SCHEMA { + return Err(failure( + LicenseArtifactStatus::StateUnavailable, + "the installed service license state schema is unsupported", + )); + } + Ok(Some(installed.artifact)) +} + +fn persist_installed(path: &Path, artifact: &LicenseArtifact) -> Result<(), LicenseArtifactError> { + let parent = path + .parent() + .ok_or_else(|| failure(LicenseArtifactStatus::StateUnavailable, "the license state path has no parent directory"))?; + let bytes = serde_json::to_vec(&InstalledLicense { + schema: INSTALLED_SCHEMA.to_owned(), + artifact: artifact.clone(), + }) + .map_err(|_| failure(LicenseArtifactStatus::StateUnavailable, "the license state could not be encoded"))?; + let file_name = path + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or("service-license.json"); + let (temporary, mut file) = loop { + let temporary = parent.join(format!( + ".{file_name}.{}.{}.tmp", + std::process::id(), + STAGING_SEQUENCE.fetch_add(1, Ordering::Relaxed) + )); + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(STATE_FILE_MODE); + } + match options.open(&temporary) { + Ok(file) => break (temporary, file), + Err(source) if source.kind() == io::ErrorKind::AlreadyExists => continue, + Err(source) => return Err(state_io(&temporary, source)), + } + }; + let result = (|| -> io::Result<()> { + file.write_all(&bytes)?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + file.set_permissions(fs::Permissions::from_mode(STATE_FILE_MODE))?; + } + file.sync_all() + })(); + drop(file); + if let Err(source) = result { + let _ = fs::remove_file(&temporary); + return Err(state_io(&temporary, source)); + } + if let Err(source) = fs::rename(&temporary, path) { + let _ = fs::remove_file(&temporary); + return Err(state_io(path, source)); + } + sync_directory(parent).map_err(|source| state_io(parent, source))?; + Ok(()) +} + +fn lock_state(directory: &Path) -> Result { + let path = directory.join(LOCK_FILE); + let mut options = fs::OpenOptions::new(); + options.create(true).truncate(false).read(true).write(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(STATE_FILE_MODE); + } + let file = options.open(&path).map_err(|source| state_io(&path, source))?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + file.set_permissions(fs::Permissions::from_mode(STATE_FILE_MODE)) + .map_err(|source| state_io(&path, source))?; + } + file.lock().map_err(|source| state_io(&path, source))?; + Ok(file) +} + +fn state_path(directory: &Path, context: &LicenseVerificationContext) -> PathBuf { + let mut hasher = Sha256::new(); + hasher.update(context.organization.as_bytes()); + hasher.update([0]); + hasher.update(context.deployment.as_bytes()); + hasher.update([0]); + hasher.update(context.service_code.as_bytes()); + let scope = hex_simd::encode_to_string(hasher.finalize(), hex_simd::AsciiCase::Lower); + directory.join(format!("service-license-{scope}.json")) +} + +fn validate_resource_scope(organization: &str, deployment: &str) -> Result<(), &'static str> { + let organization_uid = organization + .strip_prefix("organizations/") + .ok_or("organization resource name is invalid")?; + if !is_canonical_uuid_v7(organization_uid) { + return Err("organization UID is invalid"); + } + let cluster_uid = deployment + .strip_prefix(organization) + .and_then(|suffix| suffix.strip_prefix("/clusters/")) + .ok_or("deployment resource name is invalid")?; + if !is_canonical_uuid_v7(cluster_uid) { + return Err("deployment UID is invalid"); + } + Ok(()) +} + +fn is_canonical_uuid_v7(value: &str) -> bool { + uuid::Uuid::parse_str(value).is_ok_and(|parsed| parsed.get_version_num() == 7 && parsed.to_string() == value) +} + +fn is_lower_hex_sha256(value: &str) -> bool { + value.len() == 64 + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn is_bounded_label(value: &str, maximum: usize) -> bool { + !value.is_empty() + && value.len() <= maximum + && value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'_' | b':' | b'-')) +} + +fn is_service_code(value: &str) -> bool { + let mut bytes = value.bytes(); + matches!(bytes.next(), Some(b'A'..=b'Z')) + && value.len() <= 64 + && bytes.all(|byte| byte.is_ascii_uppercase() || byte.is_ascii_digit() || byte == b'_') +} + +fn decode_canonical_base64url( + value: &str, + maximum: usize, + status: LicenseArtifactStatus, + label: &str, +) -> Result, LicenseArtifactError> { + if value.is_empty() + || value.len() > maximum.saturating_mul(2) + || !value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_')) + { + return Err(failure(status, format!("the {label} is not canonical base64url"))); + } + let decoded = URL_SAFE_NO_PAD + .decode_to_vec(value.as_bytes()) + .map_err(|_| failure(status, format!("the {label} is not canonical base64url")))?; + if decoded.len() > maximum || URL_SAFE_NO_PAD.encode_to_string(&decoded) != value { + return Err(failure(status, format!("the {label} is not canonical base64url"))); + } + Ok(decoded) +} + +fn system_now() -> Result { + let seconds = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|_| failure(LicenseArtifactStatus::InvalidConfiguration, "the system clock is before the Unix epoch"))? + .as_secs(); + i64::try_from(seconds).map_err(|_| { + failure( + LicenseArtifactStatus::InvalidConfiguration, + "the system clock is outside the supported range", + ) + }) +} + +fn state_io(path: &Path, source: io::Error) -> LicenseArtifactError { + failure( + LicenseArtifactStatus::StateUnavailable, + format!("license I/O failed at {}: {source}", path.display()), + ) +} + +fn failure(status: LicenseArtifactStatus, message: impl Into) -> LicenseArtifactError { + LicenseArtifactError { + status, + message: message.into(), + license: None, + } +} + +fn failure_with_license( + status: LicenseArtifactStatus, + message: impl Into, + license: LicenseClaims, +) -> LicenseArtifactError { + LicenseArtifactError { + status, + message: message.into(), + license: Some(Box::new(license)), + } +} + +#[cfg(unix)] +fn check_state_mode(path: &Path) -> Result<(), LicenseArtifactError> { + use std::os::unix::fs::PermissionsExt as _; + let mode = fs::metadata(path) + .map_err(|source| state_io(path, source))? + .permissions() + .mode() + & 0o7777; + if mode != STATE_FILE_MODE { + return Err(failure( + LicenseArtifactStatus::StateUnavailable, + format!("the installed service license has mode {mode:o}, expected {STATE_FILE_MODE:o}"), + )); + } + Ok(()) +} + +#[cfg(not(unix))] +fn check_state_mode(_path: &Path) -> Result<(), LicenseArtifactError> { + Ok(()) +} + +fn sync_directory(directory: &Path) -> io::Result<()> { + #[cfg(unix)] + fs::File::open(directory)?.sync_all()?; + #[cfg(not(unix))] + let _ = directory; + Ok(()) +} diff --git a/rustfs/src/connect/license_relay.rs b/rustfs/src/connect/license_relay.rs new file mode 100644 index 000000000..05f85a27d --- /dev/null +++ b/rustfs/src/connect/license_relay.rs @@ -0,0 +1,377 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Offline relay export and destination-side receipt for Connect service licenses. + +use std::collections::BTreeMap; +use std::fs; +use std::io::{self, Write as _}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; + +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use time::OffsetDateTime; +use time::format_description::well_known::Rfc3339; + +use super::license::{ + LicenseArtifactError, LicenseClaims, LicenseReport, LicenseVerificationContext, apply_license_bytes, + read_license_artifact_file, verify_license_bytes, +}; +use super::relay::{ + DestinationReceiptSigner, RelayDirection, RelayEnvelope, RelayError, RelayMaterialKind, RelayParty, RelayReceiptOutcome, + RelayReview, decode_relay_envelope, prepare_approved_artifact, read_protected_relay_artifact, +}; + +const LEDGER_SCHEMA: &str = "rustfs.connect.serviceLicenseRelayLedger/1"; +const LEDGER_FILE: &str = "service-license-relay-ledger.json"; +const LOCK_FILE: &str = ".service-license-relay.lock"; +const MAX_LEDGER_BYTES: u64 = 4 * 1024 * 1024; +const MAX_LEDGER_ENTRIES: usize = 4096; + +#[cfg(unix)] +const STATE_FILE_MODE: u32 = 0o600; + +static STAGING_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +#[derive(Debug, thiserror::Error)] +pub enum ServiceLicenseRelayError { + #[error(transparent)] + Relay(#[from] RelayError), + #[error(transparent)] + License(#[from] LicenseArtifactError), + #[error("the relay producer does not match the verified license issuer")] + ProducerMismatch, + #[error("the relay destination does not match the verified license deployment")] + DestinationMismatch, + #[error("the relay transfer identifier is already bound to different material")] + TransferConflict, + #[error("the service-license relay state is unavailable")] + StateUnavailable, +} + +#[derive(Debug, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ServiceLicenseRelayExport { + pub review: RelayReview, + pub license: LicenseClaims, +} + +#[derive(Debug)] +pub struct ServiceLicenseRelayReceipt { + pub receipt_bytes: Vec, + pub outcome: RelayReceiptOutcome, + pub received_at: String, + pub license: LicenseClaims, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +struct TransferBinding { + material_kind: RelayMaterialKind, + direction: RelayDirection, + artifact_sha256: String, + producer: RelayParty, + destination: RelayParty, +} + +#[derive(Clone, Debug, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +struct ReplayRecord { + received_at: String, +} + +#[derive(Debug, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +struct RelayLedger { + schema: String, + transfers: BTreeMap, + replays: BTreeMap, +} + +impl Default for RelayLedger { + fn default() -> Self { + Self { + schema: LEDGER_SCHEMA.to_owned(), + transfers: BTreeMap::new(), + replays: BTreeMap::new(), + } + } +} + +pub fn export_service_license_relay( + artifact_path: &Path, + envelope_path: &Path, + transfer_uid: &str, + state_directory: &Path, + context: &LicenseVerificationContext, + confirmed: bool, +) -> Result { + let artifact_bytes = read_license_artifact_file(artifact_path)?; + let report = verify_license_bytes(&artifact_bytes, state_directory, context)?; + let license = verified_license(report)?; + let producer = producer(&license); + let destination = destination(&license); + let prepared = prepare_approved_artifact( + transfer_uid, + RelayMaterialKind::ServiceLicense, + &artifact_bytes, + producer, + destination, + |_| confirmed, + )?; + let envelope_bytes = serde_json::to_vec(&prepared.envelope).map_err(|_| RelayError::EnvelopeEncoding)?; + write_new_file(envelope_path, &envelope_bytes)?; + Ok(ServiceLicenseRelayExport { + review: prepared.review, + license, + }) +} + +pub fn receive_service_license_relay( + envelope_path: &Path, + state_directory: &Path, + context: &LicenseVerificationContext, + receipt_signer: &DestinationReceiptSigner, + confirmed: bool, +) -> Result { + if !confirmed { + return Err(RelayError::ApprovalRequired.into()); + } + let envelope_bytes = read_protected_relay_artifact(envelope_path)?; + let (envelope, artifact_bytes) = decode_relay_envelope(&envelope_bytes)?; + if envelope.material_kind != RelayMaterialKind::ServiceLicense || envelope.direction != RelayDirection::ConnectToCluster { + return Err(RelayError::UnsupportedMaterial.into()); + } + + let report = verify_license_bytes(&artifact_bytes, state_directory, context)?; + let license = verified_license(report)?; + let verified_producer = producer(&license); + let verified_destination = destination(&license); + if envelope.asserted_producer != verified_producer { + return Err(ServiceLicenseRelayError::ProducerMismatch); + } + if envelope.destination != verified_destination { + return Err(ServiceLicenseRelayError::DestinationMismatch); + } + + fs::create_dir_all(state_directory).map_err(|_| ServiceLicenseRelayError::StateUnavailable)?; + let _lock = lock_state(state_directory)?; + let ledger_path = state_directory.join(LEDGER_FILE); + let mut ledger = load_ledger(&ledger_path)?; + let binding = TransferBinding { + material_kind: envelope.material_kind, + direction: envelope.direction, + artifact_sha256: envelope.artifact.sha256.clone(), + producer: verified_producer.clone(), + destination: verified_destination, + }; + if ledger + .transfers + .get(&envelope.transfer_uid) + .is_some_and(|existing| existing != &binding) + { + return Err(ServiceLicenseRelayError::TransferConflict); + } + if !ledger.transfers.contains_key(&envelope.transfer_uid) { + if ledger.transfers.len() >= MAX_LEDGER_ENTRIES { + return Err(ServiceLicenseRelayError::StateUnavailable); + } + ledger.transfers.insert(envelope.transfer_uid.clone(), binding); + persist_ledger(&ledger_path, &ledger)?; + } + + let replay_id = replay_id(&envelope); + let (outcome, received_at) = if let Some(existing) = ledger.replays.get(&replay_id) { + (RelayReceiptOutcome::Duplicate, existing.received_at.clone()) + } else { + if ledger.replays.len() >= MAX_LEDGER_ENTRIES { + return Err(ServiceLicenseRelayError::StateUnavailable); + } + let applied = apply_license_bytes(&artifact_bytes, state_directory, context)?; + let received_at = receipt_time(context.now_unix)?; + let outcome = if applied.idempotent { + RelayReceiptOutcome::Duplicate + } else { + RelayReceiptOutcome::Applied + }; + ledger.replays.insert( + replay_id, + ReplayRecord { + received_at: received_at.clone(), + }, + ); + persist_ledger(&ledger_path, &ledger)?; + (outcome, received_at) + }; + let receipt_bytes = receipt_signer.sign(&envelope, verified_producer, outcome, received_at.clone())?; + Ok(ServiceLicenseRelayReceipt { + receipt_bytes, + outcome, + received_at, + license, + }) +} + +fn verified_license(report: LicenseReport) -> Result { + report.license.ok_or(ServiceLicenseRelayError::StateUnavailable) +} + +fn producer(license: &LicenseClaims) -> RelayParty { + RelayParty { + party_type: "CONNECT_LICENSE_ISSUER".to_owned(), + name: license.issuer.clone(), + key_id: Some(license.key_id.clone()), + } +} + +fn destination(license: &LicenseClaims) -> RelayParty { + RelayParty { + party_type: "CLUSTER".to_owned(), + name: license.deployment.clone(), + key_id: None, + } +} + +fn replay_id(envelope: &RelayEnvelope) -> String { + let mut digest = Sha256::new(); + digest.update(b"SERVICE_LICENSE\0"); + digest.update(envelope.artifact.sha256.as_bytes()); + digest.update([0]); + digest.update(envelope.destination.party_type.as_bytes()); + digest.update([0]); + digest.update(envelope.destination.name.as_bytes()); + hex_simd::encode_to_string(digest.finalize(), hex_simd::AsciiCase::Lower) +} + +fn receipt_time(now_unix: i64) -> Result { + let time = OffsetDateTime::from_unix_timestamp(now_unix).map_err(|_| ServiceLicenseRelayError::StateUnavailable)?; + time.format(&Rfc3339).map_err(|_| ServiceLicenseRelayError::StateUnavailable) +} + +fn load_ledger(path: &Path) -> Result { + let bytes = match fs::read(path) { + Ok(bytes) => bytes, + Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(RelayLedger::default()), + Err(_) => return Err(ServiceLicenseRelayError::StateUnavailable), + }; + check_mode(path)?; + if bytes.len() as u64 > MAX_LEDGER_BYTES { + return Err(ServiceLicenseRelayError::StateUnavailable); + } + let ledger: RelayLedger = serde_json::from_slice(&bytes).map_err(|_| ServiceLicenseRelayError::StateUnavailable)?; + if ledger.schema != LEDGER_SCHEMA || ledger.transfers.len() > MAX_LEDGER_ENTRIES || ledger.replays.len() > MAX_LEDGER_ENTRIES + { + return Err(ServiceLicenseRelayError::StateUnavailable); + } + Ok(ledger) +} + +fn persist_ledger(path: &Path, ledger: &RelayLedger) -> Result<(), ServiceLicenseRelayError> { + let bytes = serde_json::to_vec(ledger).map_err(|_| ServiceLicenseRelayError::StateUnavailable)?; + if bytes.len() as u64 > MAX_LEDGER_BYTES { + return Err(ServiceLicenseRelayError::StateUnavailable); + } + let parent = path.parent().ok_or(ServiceLicenseRelayError::StateUnavailable)?; + let temporary = temporary_path(path); + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(STATE_FILE_MODE); + } + let mut file = options + .open(&temporary) + .map_err(|_| ServiceLicenseRelayError::StateUnavailable)?; + let result = file.write_all(&bytes).and_then(|()| file.sync_all()); + drop(file); + if result.is_err() || fs::rename(&temporary, path).is_err() || sync_directory(parent).is_err() { + let _ = fs::remove_file(&temporary); + return Err(ServiceLicenseRelayError::StateUnavailable); + } + Ok(()) +} + +fn write_new_file(path: &Path, bytes: &[u8]) -> Result<(), ServiceLicenseRelayError> { + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(STATE_FILE_MODE); + } + let mut file = options.open(path).map_err(|_| ServiceLicenseRelayError::StateUnavailable)?; + if file.write_all(bytes).and_then(|()| file.sync_all()).is_err() { + drop(file); + let _ = fs::remove_file(path); + return Err(ServiceLicenseRelayError::StateUnavailable); + } + Ok(()) +} + +fn lock_state(directory: &Path) -> Result { + let path = directory.join(LOCK_FILE); + let mut options = fs::OpenOptions::new(); + options.create(true).truncate(false).read(true).write(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(STATE_FILE_MODE).custom_flags(libc::O_NOFOLLOW); + } + let file = options.open(path).map_err(|_| ServiceLicenseRelayError::StateUnavailable)?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt as _; + file.set_permissions(fs::Permissions::from_mode(STATE_FILE_MODE)) + .map_err(|_| ServiceLicenseRelayError::StateUnavailable)?; + } + file.lock().map_err(|_| ServiceLicenseRelayError::StateUnavailable)?; + Ok(file) +} + +fn temporary_path(path: &Path) -> PathBuf { + let file_name = path.file_name().and_then(|name| name.to_str()).unwrap_or(LEDGER_FILE); + path.with_file_name(format!( + ".{file_name}.{}.{}.tmp", + std::process::id(), + STAGING_SEQUENCE.fetch_add(1, Ordering::Relaxed) + )) +} + +#[cfg(unix)] +fn check_mode(path: &Path) -> Result<(), ServiceLicenseRelayError> { + use std::os::unix::fs::PermissionsExt as _; + let metadata = fs::symlink_metadata(path).map_err(|_| ServiceLicenseRelayError::StateUnavailable)?; + if !metadata.is_file() || metadata.permissions().mode() & 0o7777 != STATE_FILE_MODE { + return Err(ServiceLicenseRelayError::StateUnavailable); + } + Ok(()) +} + +#[cfg(not(unix))] +fn check_mode(_path: &Path) -> Result<(), ServiceLicenseRelayError> { + Ok(()) +} + +fn sync_directory(directory: &Path) -> io::Result<()> { + #[cfg(unix)] + fs::File::open(directory)?.sync_all()?; + #[cfg(not(unix))] + let _ = directory; + Ok(()) +} diff --git a/rustfs/src/connect/license_renewal.rs b/rustfs/src/connect/license_renewal.rs new file mode 100644 index 000000000..947321c48 --- /dev/null +++ b/rustfs/src/connect/license_renewal.rs @@ -0,0 +1,636 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Device-side retrieval of operator-approved RustFS Connect service licenses. + +use std::fs; +use std::io::{self, Write as _}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::time::Duration; + +use chrono::Utc; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use uuid::Uuid; + +use super::config::HeartbeatConfig; +use super::license::{ + LICENSE_SCHEMA, LicenseArtifactError, LicenseClaims, LicenseReport, LicenseVerificationContext, apply_license_artifact, + inspect_installed_license, verify_license_artifact, +}; +use super::telemetry::{TelemetryDelivery, TelemetryError, TelemetryTransport, is_exact_utc_seconds}; + +const PROTOCOL_VERSION: &str = "v1"; +const DELIVERY_VERSION: u32 = 1; +const MAX_ATTEMPTS: usize = 3; + +#[cfg(unix)] +const ARTIFACT_FILE_MODE: u32 = 0o600; + +static STAGING_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +/// Result of checking the currently installed license for an approved renewal. +#[derive(Clone, Debug, Eq, PartialEq)] +pub enum LicenseRenewalOutcome { + Requested, + Pending { replacement_license_uid: String }, + Installed(Box), +} + +/// An mTLS client for the read-only Connect license-renewal delivery surface. +pub struct LicenseRenewalClient { + transport: TelemetryTransport, + initial_backoff: Duration, + max_backoff: Duration, +} + +impl LicenseRenewalClient { + /// Reuses the registered device identity, explicit proxy, trust roots, and + /// bounded request timeout from the Connect heartbeat transport. + pub fn new(config: HeartbeatConfig) -> Result { + let initial_backoff = config.schedule.initial_backoff; + let max_backoff = config.schedule.max_backoff; + let transport = TelemetryTransport::new(config).map_err(transport_error)?; + Ok(Self { + transport, + initial_backoff, + max_backoff, + }) + } + + /// Checks the license currently installed in `state_directory` and installs + /// only an operator-approved, signed replacement for the same scope. + pub async fn renew_installed( + &self, + state_directory: &Path, + context: &LicenseVerificationContext, + ) -> Result { + let current = inspect_installed_license(state_directory, context)?; + let current = current.license.ok_or(LicenseRenewalError::InstalledLicense)?; + let status_request = LicenseRenewalRequest::new(¤t)?; + let status_body = self.post_with_retry("licenseRenewal:status", &status_request).await?; + let status = decode_status(&status_body, &status_request)?; + + match status.status { + RenewalStatus::Requested => Ok(LicenseRenewalOutcome::Requested), + RenewalStatus::Pending => Ok(LicenseRenewalOutcome::Pending { + replacement_license_uid: status.replacement_license_uid.ok_or(LicenseRenewalError::Response)?, + }), + RenewalStatus::Ready => { + let expected_replacement = status + .replacement_license_uid + .as_deref() + .ok_or(LicenseRenewalError::Response)?; + let expected_digest = status.artifact_sha256.as_deref().ok_or(LicenseRenewalError::Response)?; + let download_request = LicenseRenewalRequest::new(¤t)?; + let body = self.post_with_retry("licenseRenewal:download", &download_request).await?; + let artifact = + decode_artifact(&body, &download_request, expected_replacement, expected_digest, &status.expire_time)?; + let mut current_context = context.clone(); + current_context.now_unix = Utc::now().timestamp(); + install_downloaded_artifact(state_directory, ¤t_context, artifact) + } + } + } + + async fn post_with_retry(&self, operation: &str, body: &T) -> Result, LicenseRenewalError> { + let mut backoff = self.initial_backoff; + for attempt in 0..MAX_ATTEMPTS { + match self.transport.post(operation, body).await.map_err(transport_error)? { + TelemetryDelivery::Accepted { body, .. } => return Ok(body), + TelemetryDelivery::Retry { retry_after } if attempt + 1 < MAX_ATTEMPTS => { + let delay = retry_after.unwrap_or(backoff).clamp(self.initial_backoff, self.max_backoff); + tokio::time::sleep(delay).await; + backoff = backoff.saturating_mul(2).min(self.max_backoff); + } + TelemetryDelivery::Retry { .. } => return Err(LicenseRenewalError::RetryExhausted), + TelemetryDelivery::AuthenticationStopped { status, .. } => { + return Err(LicenseRenewalError::AuthenticationStopped { status }); + } + TelemetryDelivery::Rejected { status, .. } => return Err(LicenseRenewalError::Rejected { status }), + } + } + Err(LicenseRenewalError::RetryExhausted) + } +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct LicenseRenewalRequest { + protocol_version: &'static str, + current_license_uid: String, + service_code: String, + request_id: String, +} + +impl LicenseRenewalRequest { + fn new(current: &LicenseClaims) -> Result { + if !is_uuid_v7(¤t.license_uid) || !is_service_code(¤t.service_code) { + return Err(LicenseRenewalError::InstalledLicense); + } + Ok(Self { + protocol_version: PROTOCOL_VERSION, + current_license_uid: current.license_uid.clone(), + service_code: current.service_code.clone(), + request_id: Uuid::new_v4().to_string(), + }) + } +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +enum RenewalStatus { + Requested, + Pending, + Ready, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +struct LicenseRenewalResponse { + protocol_version: String, + status: RenewalStatus, + source_license_uid: String, + replacement_license_uid: Option, + service_code: String, + schema: String, + version: u32, + expire_time: String, + artifact_sha256: Option, + manual_approval_required: bool, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +struct DownloadedArtifact { + payload: String, + signature: String, +} + +struct DownloadedLicense { + artifact: DownloadedArtifact, + replacement_license_uid: String, + expire_time: String, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields, rename_all = "camelCase")] +struct LicenseArtifactResponse { + protocol_version: String, + status: RenewalStatus, + source_license_uid: String, + replacement_license_uid: Option, + service_code: String, + schema: String, + version: u32, + expire_time: String, + artifact_sha256: Option, + artifact: DownloadedArtifact, +} + +fn decode_status(body: &[u8], request: &LicenseRenewalRequest) -> Result { + let response: LicenseRenewalResponse = serde_json::from_slice(body).map_err(|_| LicenseRenewalError::Response)?; + validate_common( + &response.protocol_version, + &response.source_license_uid, + response.replacement_license_uid.as_deref(), + &response.service_code, + &response.schema, + response.version, + &response.expire_time, + response.artifact_sha256.as_deref(), + request, + )?; + let valid_state = match response.status { + RenewalStatus::Requested => { + response.replacement_license_uid.is_none() && response.artifact_sha256.is_none() && response.manual_approval_required + } + RenewalStatus::Pending => { + response.replacement_license_uid.is_some() && response.artifact_sha256.is_none() && response.manual_approval_required + } + RenewalStatus::Ready => { + response.replacement_license_uid.is_some() + && response.artifact_sha256.as_deref().is_some_and(is_sha256) + && !response.manual_approval_required + } + }; + if !valid_state { + return Err(LicenseRenewalError::Response); + } + Ok(response) +} + +fn decode_artifact( + body: &[u8], + request: &LicenseRenewalRequest, + expected_replacement: &str, + expected_digest: &str, + expected_expire_time: &str, +) -> Result { + let response: LicenseArtifactResponse = serde_json::from_slice(body).map_err(|_| LicenseRenewalError::Response)?; + validate_common( + &response.protocol_version, + &response.source_license_uid, + response.replacement_license_uid.as_deref(), + &response.service_code, + &response.schema, + response.version, + &response.expire_time, + response.artifact_sha256.as_deref(), + request, + )?; + let replacement = response.replacement_license_uid.as_deref(); + let digest = response.artifact_sha256.as_deref(); + if response.status != RenewalStatus::Ready + || replacement != Some(expected_replacement) + || digest != Some(expected_digest) + || response.expire_time != expected_expire_time + || !is_sha256(expected_digest) + { + return Err(LicenseRenewalError::Response); + } + let canonical = serde_json::to_vec(&response.artifact).map_err(|_| LicenseRenewalError::Response)?; + let actual_digest = hex_simd::encode_to_string(Sha256::digest(&canonical), hex_simd::AsciiCase::Lower); + if !constant_time_eq(actual_digest.as_bytes(), expected_digest.as_bytes()) { + return Err(LicenseRenewalError::DigestMismatch); + } + Ok(DownloadedLicense { + artifact: response.artifact, + replacement_license_uid: expected_replacement.to_owned(), + expire_time: response.expire_time, + }) +} + +#[allow(clippy::too_many_arguments)] +fn validate_common( + protocol_version: &str, + source_license_uid: &str, + replacement_license_uid: Option<&str>, + service_code: &str, + schema: &str, + version: u32, + expire_time: &str, + artifact_sha256: Option<&str>, + request: &LicenseRenewalRequest, +) -> Result<(), LicenseRenewalError> { + if protocol_version != PROTOCOL_VERSION + || source_license_uid != request.current_license_uid + || service_code != request.service_code + || schema != LICENSE_SCHEMA + || version != DELIVERY_VERSION + || !is_exact_utc_seconds(expire_time) + || replacement_license_uid.is_some_and(|uid| !is_uuid_v7(uid)) + || artifact_sha256.is_some_and(|digest| !is_sha256(digest)) + { + return Err(LicenseRenewalError::Response); + } + Ok(()) +} + +fn install_downloaded_artifact( + state_directory: &Path, + context: &LicenseVerificationContext, + download: DownloadedLicense, +) -> Result { + fs::create_dir_all(state_directory).map_err(LicenseRenewalError::InstallState)?; + let bytes = serde_json::to_vec(&download.artifact).map_err(|_| LicenseRenewalError::Response)?; + let path = stage_artifact(state_directory, &bytes)?; + let result = (|| { + let verified = verify_license_artifact(&path, state_directory, context)?; + let claims = verified.license.as_ref().ok_or(LicenseRenewalError::Response)?; + if claims.license_uid != download.replacement_license_uid || claims.expire_time != download.expire_time { + return Err(LicenseRenewalError::Response); + } + apply_license_artifact(&path, state_directory, context) + .map(|report| LicenseRenewalOutcome::Installed(Box::new(report))) + .map_err(LicenseRenewalError::License) + })(); + let _ = fs::remove_file(path); + result +} + +fn stage_artifact(directory: &Path, bytes: &[u8]) -> Result { + loop { + let path = directory.join(format!( + ".license-renewal.{}.{}.tmp", + std::process::id(), + STAGING_SEQUENCE.fetch_add(1, Ordering::Relaxed) + )); + let mut options = fs::OpenOptions::new(); + options.write(true).create_new(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt as _; + options.mode(ARTIFACT_FILE_MODE); + } + let mut file = match options.open(&path) { + Ok(file) => file, + Err(source) if source.kind() == io::ErrorKind::AlreadyExists => continue, + Err(source) => return Err(LicenseRenewalError::InstallState(source)), + }; + if let Err(source) = file.write_all(bytes).and_then(|()| file.sync_all()) { + let _ = fs::remove_file(path); + return Err(LicenseRenewalError::InstallState(source)); + } + return Ok(path); + } +} + +fn is_uuid_v7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| uuid.get_version_num() == 7 && uuid.to_string() == value) +} + +fn is_service_code(value: &str) -> bool { + let mut bytes = value.bytes(); + matches!(bytes.next(), Some(b'A'..=b'Z')) + && value.len() <= 64 + && bytes.all(|byte| byte.is_ascii_uppercase() || byte.is_ascii_digit() || byte == b'_') +} + +fn is_sha256(value: &str) -> bool { + value.len() == 64 + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn constant_time_eq(left: &[u8], right: &[u8]) -> bool { + if left.len() != right.len() { + return false; + } + left.iter() + .zip(right) + .fold(0_u8, |difference, (left, right)| difference | (left ^ right)) + == 0 +} + +fn transport_error(error: TelemetryError) -> LicenseRenewalError { + match error { + TelemetryError::Endpoint => LicenseRenewalError::Endpoint, + TelemetryError::RootCertificate => LicenseRenewalError::RootCertificate, + TelemetryError::ProxyConfiguration => LicenseRenewalError::ProxyConfiguration, + TelemetryError::ProxyAuthentication => LicenseRenewalError::ProxyAuthentication, + TelemetryError::ProxyRejected => LicenseRenewalError::ProxyRejected, + TelemetryError::TlsPeer => LicenseRenewalError::TlsPeer, + TelemetryError::Schedule => LicenseRenewalError::Schedule, + TelemetryError::NotRegistered => LicenseRenewalError::NotRegistered, + TelemetryError::IdentityMissing => LicenseRenewalError::IdentityMissing, + TelemetryError::IdentityCertificate => LicenseRenewalError::IdentityCertificate, + TelemetryError::CredentialName => LicenseRenewalError::CredentialName, + TelemetryError::CredentialExpired => LicenseRenewalError::CredentialExpired, + TelemetryError::StateConflict => LicenseRenewalError::CredentialState, + TelemetryError::ResponseTooLarge => LicenseRenewalError::ResponseTooLarge, + TelemetryError::Url(_) + | TelemetryError::Transport(_) + | TelemetryError::Identity(_) + | TelemetryError::IdentityStore(_) + | TelemetryError::CredentialStore(_) + | TelemetryError::CredentialValidation(_) => LicenseRenewalError::Transport, + } +} + +#[derive(Debug, thiserror::Error)] +pub enum LicenseRenewalError { + #[error("Connect license endpoint must be an HTTPS base URL without credentials, query, or fragment")] + Endpoint, + #[error("Connect license root CA configuration is invalid")] + RootCertificate, + #[error("Connect license proxy configuration is invalid")] + ProxyConfiguration, + #[error("Connect proxy authentication failed; verify the configured proxy credential files")] + ProxyAuthentication, + #[error("Connect proxy connection failed; verify proxy availability, credentials, and the Connect endpoint")] + ProxyRejected, + #[error("Connect TLS peer certificate validation failed; verify the endpoint and configured root CA")] + TlsPeer, + #[error("Connect license retry schedule is invalid")] + Schedule, + #[error("RustFS is not registered with Connect")] + NotRegistered, + #[error("the Connect device private key is missing")] + IdentityMissing, + #[error("the stored Connect certificate and device private key cannot form a TLS identity")] + IdentityCertificate, + #[error("the stored Connect credential name is invalid")] + CredentialName, + #[error("the stored Connect device certificate is not currently valid")] + CredentialExpired, + #[error("the persisted Connect credential transition is invalid")] + CredentialState, + #[error("Connect license response exceeded 64 KiB")] + ResponseTooLarge, + #[error("Connect license transport failed")] + Transport, + #[error("Connect license request failed after three bounded attempts")] + RetryExhausted, + #[error("Connect rejected the device license request with HTTP {status}")] + AuthenticationStopped { status: u16 }, + #[error("Connect rejected the license request with HTTP {status}")] + Rejected { status: u16 }, + #[error("the installed service license cannot identify the renewal scope")] + InstalledLicense, + #[error("the Connect license response is invalid")] + Response, + #[error("the Connect license artifact digest does not match the approved digest")] + DigestMismatch, + #[error("the Connect license artifact staging file could not be written")] + InstallState(#[source] io::Error), + #[error(transparent)] + License(#[from] LicenseArtifactError), +} + +#[cfg(test)] +mod tests { + use super::*; + use base64_simd::URL_SAFE_NO_PAD; + use chrono::DateTime; + use serde::Deserialize; + + const SOURCE_UID: &str = "018cc251-f400-7000-8000-000000000005"; + const REPLACEMENT_UID: &str = "018cc251-f400-7000-8000-000000000002"; + + #[derive(Deserialize)] + #[serde(rename_all = "camelCase")] + struct Vector { + public_key: String, + key_id: String, + connect_artifact: String, + } + + fn vector() -> Vector { + serde_json::from_str(include_str!("../../tests/fixtures/connect-license-ed25519-vector.json")) + .expect("license vector must decode") + } + + fn request() -> LicenseRenewalRequest { + LicenseRenewalRequest { + protocol_version: PROTOCOL_VERSION, + current_license_uid: SOURCE_UID.to_owned(), + service_code: "SUPPORT".to_owned(), + request_id: "018cc251-f400-4000-8000-000000000001".to_owned(), + } + } + + fn artifact_response(digest: &str, artifact: &DownloadedArtifact) -> Vec { + serde_json::to_vec(&serde_json::json!({ + "protocolVersion": "v1", + "status": "READY", + "sourceLicenseUid": SOURCE_UID, + "replacementLicenseUid": REPLACEMENT_UID, + "serviceCode": "SUPPORT", + "schema": LICENSE_SCHEMA, + "version": 1, + "expireTime": "2030-02-01T00:00:00Z", + "artifactSha256": digest, + "artifact": artifact, + })) + .expect("response must encode") + } + + #[test] + fn status_accepts_only_consistent_manual_states() { + let requested = serde_json::json!({ + "protocolVersion": "v1", + "status": "REQUESTED", + "sourceLicenseUid": SOURCE_UID, + "replacementLicenseUid": null, + "serviceCode": "SUPPORT", + "schema": LICENSE_SCHEMA, + "version": 1, + "expireTime": "2030-01-01T00:00:00Z", + "artifactSha256": null, + "manualApprovalRequired": true, + }); + assert_eq!( + decode_status(&serde_json::to_vec(&requested).expect("status"), &request()) + .expect("valid requested state") + .status, + RenewalStatus::Requested + ); + + let mut inconsistent = requested; + inconsistent["manualApprovalRequired"] = false.into(); + assert!(matches!( + decode_status(&serde_json::to_vec(&inconsistent).expect("status"), &request()), + Err(LicenseRenewalError::Response) + )); + } + + #[test] + fn artifact_digest_and_signed_scope_are_verified_before_install() { + let vector = vector(); + let artifact: DownloadedArtifact = serde_json::from_str(&vector.connect_artifact).expect("artifact"); + let bytes = serde_json::to_vec(&artifact).expect("canonical artifact"); + let digest = hex_simd::encode_to_string(Sha256::digest(bytes), hex_simd::AsciiCase::Lower); + let downloaded = decode_artifact( + &artifact_response(&digest, &artifact), + &request(), + REPLACEMENT_UID, + &digest, + "2030-02-01T00:00:00Z", + ) + .expect("matching response"); + let public_key = URL_SAFE_NO_PAD.decode_to_vec(vector.public_key).expect("public key"); + let context = LicenseVerificationContext::new( + public_key.try_into().expect("32-byte public key"), + vector.key_id, + "test-connect-issuer".to_owned(), + "test-rustfs-cluster".to_owned(), + "organizations/018cc251-f400-7000-8000-000000000003".to_owned(), + "organizations/018cc251-f400-7000-8000-000000000003/clusters/018cc251-f400-7000-8000-000000000004".to_owned(), + "SUPPORT".to_owned(), + DateTime::parse_from_rfc3339("2030-01-15T00:00:00Z") + .expect("time") + .timestamp(), + ) + .expect("verification context"); + let state = tempfile::tempdir().expect("state directory"); + + let installed = install_downloaded_artifact(state.path(), &context, downloaded).expect("install"); + let LicenseRenewalOutcome::Installed(report) = installed else { + panic!("expected installed outcome"); + }; + assert_eq!(report.license.expect("claims").license_uid, REPLACEMENT_UID); + assert!( + inspect_installed_license(state.path(), &context) + .expect("installed license") + .is_valid() + ); + } + + #[test] + fn digest_mismatch_and_wrong_audience_never_install() { + let vector = vector(); + let artifact: DownloadedArtifact = serde_json::from_str(&vector.connect_artifact).expect("artifact"); + let digest = "0".repeat(64); + assert!(matches!( + decode_artifact( + &artifact_response(&digest, &artifact), + &request(), + REPLACEMENT_UID, + &digest, + "2030-02-01T00:00:00Z", + ), + Err(LicenseRenewalError::DigestMismatch) + )); + + let public_key = URL_SAFE_NO_PAD.decode_to_vec(vector.public_key).expect("public key"); + let context = LicenseVerificationContext::new( + public_key.try_into().expect("32-byte public key"), + vector.key_id, + "test-connect-issuer".to_owned(), + "another-audience".to_owned(), + "organizations/018cc251-f400-7000-8000-000000000003".to_owned(), + "organizations/018cc251-f400-7000-8000-000000000003/clusters/018cc251-f400-7000-8000-000000000004".to_owned(), + "SUPPORT".to_owned(), + DateTime::parse_from_rfc3339("2030-01-15T00:00:00Z") + .expect("time") + .timestamp(), + ) + .expect("verification context"); + let state = tempfile::tempdir().expect("state directory"); + let result = install_downloaded_artifact( + state.path(), + &context, + DownloadedLicense { + artifact, + replacement_license_uid: REPLACEMENT_UID.to_owned(), + expire_time: "2030-02-01T00:00:00Z".to_owned(), + }, + ); + + assert!(matches!(result, Err(LicenseRenewalError::License(LicenseArtifactError { .. })))); + assert!(matches!( + inspect_installed_license(state.path(), &context), + Err(LicenseArtifactError { .. }) + )); + } + + #[test] + fn transport_errors_do_not_expose_remote_or_proxy_details() { + let errors = [ + LicenseRenewalError::ProxyAuthentication, + LicenseRenewalError::ProxyRejected, + LicenseRenewalError::TlsPeer, + LicenseRenewalError::Transport, + ]; + for error in errors { + let message = error.to_string(); + for secret in ["proxy.example", "proxy-user", "proxy-password", "remote response"] { + assert!(!message.contains(secret)); + } + } + } +} diff --git a/rustfs/src/connect/mod.rs b/rustfs/src/connect/mod.rs index ec3a1f999..35be86729 100644 --- a/rustfs/src/connect/mod.rs +++ b/rustfs/src/connect/mod.rs @@ -28,19 +28,107 @@ pub mod client; pub mod config; pub mod credential_store; +pub mod diagnostics; +pub mod environment; pub mod heartbeat; pub mod identity; pub mod identity_store; pub mod inventory; +pub mod license; +pub mod license_relay; +pub mod license_renewal; pub mod offline; pub mod registration; pub mod registration_bootstrap; +pub mod relay; +mod report_bundle; +pub mod report_upload; pub mod runtime; mod telemetry; pub use client::{ClientError, ConnectClient, ConnectConfig}; -pub use config::{HeartbeatConfig, HeartbeatConfigError, HeartbeatSchedule}; +pub use config::{ + ENV_CONNECT_JOB_SIGNING_KEY_ID, ENV_CONNECT_JOB_SIGNING_PUBLIC_KEY_FILE, ENV_CONNECT_PROXY_BYPASS, + ENV_CONNECT_PROXY_PASSWORD_FILE, ENV_CONNECT_PROXY_URL, ENV_CONNECT_PROXY_USERNAME_FILE, HeartbeatConfig, + HeartbeatConfigError, HeartbeatSchedule, ProxyConfig, ProxyConfigError, +}; pub use credential_store::{CredentialStore, DeviceCredential}; +pub use diagnostics::{ + CLIENT_CAPABILITY, CLIENT_SCHEMA_VERSION, CONNECT_DIAGNOSTIC_CAPABILITIES, CPU_PROFILE_CAPABILITY, CaptureMode, + ClientDiagnosticResult, ClientMeasurement, ClientOperation, ClientOutcome, ClientPerformanceData, ClientPerformanceError, + ClientPerformanceRequest, ClientProbe, ClientProbeError, ClientProbeFuture, ClientProbeMeasurement, ClientProvenance, + ClientReasonCode, ClientTargetParameters, ClientTargetReasonCode, ClientTargetResult, ClientTargetUnits, DRIVE_CAPABILITY, + DRIVE_SCHEMA_VERSION, DiagnosticCollectionPolicy, DiagnosticReceipt, DiagnosticScheduleError, DiagnosticScheduleRuntime, + DiagnosticScheduleStatus, DriveDiagnosticResult, DriveMeasurement, DriveOutcome, DrivePerformanceData, DrivePerformanceError, + DrivePerformanceRequest, DriveProvenance, DriveReadMode, DriveReasonCode, DriveTargetParameters, DriveTargetReasonCode, + DriveTargetResult, DriveTargetUnits, HttpClientProbe, LOGS_CAPABILITY, LOGS_SCHEMA_VERSION, LocalClientConsent, + LocalDriveConsent, LocalLogConsent, LocalOtlpHeaders, LocalProfileConsent, LocalTelemetryConsent, + LocallyReviewedTraceArtifact, LogCaptureError, LogCaptureRequest, LogProvenance, MAX_OTLP_BODY_BYTES, MAX_PROFILE_DURATION, + MAX_SAFE_INTEGER, MAX_TELEMETRY_DURATION, MAX_TELEMETRY_RESULT_BYTES, MAX_TELEMETRY_SPANS, MEMORY_PROFILE_CAPABILITY, + ObservedTelemetrySpan, OperationSummary, OtlpBatch, OtlpForwardError, OtlpReceipt, PROFILE_SCHEMA_VERSION, + ProfileCaptureRequest, ProfileData, ProfileError, ProfileOutcome, ProfileProvenance, ProfileReasonCode, ProfileResult, + ProfileTool, ReceiptOutcome, RecordedTrace, ReplayedTrace, SavedClientExport, SavedDriveExport, SavedLogExport, + SavedProfileExport, SavedTelemetryExport, SignedClientExport, SignedDriveExport, SignedLogExport, SignedProfileExport, + SignedTelemetryExport, TELEMETRY_OTLP_CAPABILITY, TELEMETRY_RECORD_CAPABILITY, TELEMETRY_REPLAY_CAPABILITY, + TELEMETRY_SCHEMA_VERSION, THREAD_PROFILE_CAPABILITY, TelemetryArtifactConsent, TelemetryArtifactError, + TelemetryArtifactRequest, TelemetryCoverage, TelemetryDiagnosticResult, TelemetryOperation, TelemetryOutcome, + TelemetryProducerError, TelemetryProvenance, TelemetryReasonCode, TelemetrySpan, TelemetrySpanStatus, TelemetryTool, + ThreadProfileData, ThreadProfileScope, ThreadState, ThreadStateCount, TraceAnalysis, TraceAnalysisError, TraceRecordCapture, + TraceRecordCompletion, TraceRecordLimits, TraceReplayError, analyze_trace, capture_cpu_profile, capture_thread_profile, + encode_signed_profile_export, encode_signed_telemetry_export, export_cpu_profile, export_logs, export_memory_profile, + export_thread_profile, export_trace_otlp, export_trace_otlp_result, measure_client, measure_drive, + read_protected_client_credential, record_diagnostic_result, record_trace, record_trace_bus, replay_trace, + replay_trace_result, run_local_environment_once, save_signed_client_export, save_signed_drive_export, save_signed_log_export, + save_signed_profile_export, save_signed_telemetry_export, sign_client_export, sign_drive_export, spawn_environment_schedule, + validate_client_limits, validate_drive_limits, +}; +pub use diagnostics::{ + DIAGNOSTIC_JOB_SIGNATURE_DOMAIN, DiagnosticJobEnvelope, DiagnosticJobError, DiagnosticJobExecution, DiagnosticJobLimits, + DiagnosticJobParameters, DiagnosticJobTarget, TrustedDiagnosticJobSigner, VerifiedDiagnosticJob, execute_diagnostic_job, +}; +pub use diagnostics::{ + LocalNetworkConsent, MAX_NETWORK_ARCHIVE_BYTES, MAX_NETWORK_BANDWIDTH_BYTES_PER_SECOND, MAX_NETWORK_DECOMPRESSED_BYTES, + MAX_NETWORK_DURATION, MAX_NETWORK_ENVELOPE_BYTES, MAX_NETWORK_OPERATIONS, MAX_NETWORK_PEERS, MAX_NETWORK_RESULT_BYTES, + MAX_NETWORK_TRAFFIC_BYTES, NETWORK_CAPABILITY, NETWORK_SCHEMA_VERSION, NETWORK_TOOL_ID, NetworkCoverage, + NetworkDiagnosticResult, NetworkMeasurement, NetworkOutcome, NetworkPeerHarness, NetworkPeerResult, NetworkPerformanceData, + NetworkPerformanceError, NetworkPerformanceRequest, NetworkProvenance, NetworkReasonCode, PeerProbeError, PeerProbeFuture, + PeerProbeMeasurement, PeerReasonCode, SavedNetworkExport, SignedNetworkExport, measure_network, measure_network_with_harness, + save_signed_network_export, sign_network_export, +}; +pub use diagnostics::{ + LocalObjectConsent, MAX_OBJECT_BANDWIDTH_BYTES_PER_SECOND, MAX_OBJECT_DURATION, MAX_OBJECT_RESULT_BYTES, + MAX_OBJECT_TRAFFIC_BYTES, OBJECT_CAPABILITY, OBJECT_SCHEMA_VERSION, OBJECT_TOOL_ID, ObjectDiagnosticResult, + ObjectMeasurement, ObjectOperation, ObjectOutcome, ObjectPerformanceData, ObjectPerformanceError, ObjectPerformanceRequest, + ObjectProbe, ObjectProbeError, ObjectProbeFuture, ObjectProbeMeasurement, ObjectProvenance, ObjectReasonCode, + ObjectTargetParameters, ObjectTargetReasonCode, ObjectTargetResult, ObjectTargetUnits, S3ObjectProbe, SavedObjectExport, + SignedObjectExport, measure_object, read_protected_object_credential, save_signed_object_export, sign_object_export, + validate_object_limits, +}; +pub use diagnostics::{ + LocalSiteReplicationConsent, MAX_SITE_REPLICATION_DURATION, MAX_SITE_REPLICATION_TRAFFIC_BYTES, S3SiteReplicationProbe, + SITE_REPLICATION_CAPABILITY, SITE_REPLICATION_SCHEMA_VERSION, SITE_REPLICATION_TOOL_ID, SavedSiteReplicationExport, + SignedSiteReplicationExport, SiteReplicationCredentials, SiteReplicationDiagnosticResult, SiteReplicationEndpoint, + SiteReplicationMeasurement, SiteReplicationOutcome, SiteReplicationPerformanceData, SiteReplicationPerformanceError, + SiteReplicationPerformanceRequest, SiteReplicationProbe, SiteReplicationProbeError, SiteReplicationProbeFuture, + SiteReplicationProbeMeasurement, SiteReplicationProvenance, SiteReplicationReasonCode, SiteReplicationTargetReasonCode, + SiteReplicationTargetResult, measure_site_replication, read_protected_site_replication_credential, + save_signed_site_replication_export, sign_site_replication_export, validate_site_replication_limits, +}; +pub use diagnostics::{ + LocalTopConsent, MAX_TOP_DURATION, MAX_TOP_EXPORT_VALIDITY, NetworkCounterSnapshot, SavedTopExport, SignedTopExport, + TOP_CLASSIFICATION, TOP_SCHEMA_VERSION, TopApiData, TopApiOperation, TopCaptureError, TopCaptureLimits, TopCaptureRequest, + TopCaptureScope, TopCoverage, TopDiskData, TopLocksData, TopNetData, TopOutcome, TopProvenance, TopReasonCode, TopResult, + TopRpcData, capture_top_api, capture_top_disk, capture_top_locks, capture_top_net, capture_top_rpc, evaluate_disk_window, + evaluate_network_window, save_signed_top_export, sign_top_export, +}; +pub(crate) use diagnostics::{ + LocalTraceCaptureError, LocalTraceCaptureRuntime, request_local_trace_capture, spawn_local_trace_capture_runtime, +}; +pub use environment::{ + ENVIRONMENT_CAPABILITY, ENVIRONMENT_SCHEMA_VERSION, EnvironmentCollectionRequest, EnvironmentError, EnvironmentExportRequest, + EnvironmentFilesystemType, EnvironmentInventory, EnvironmentOsFamily, MAX_ENVIRONMENT_DURATION, SavedEnvironmentExport, + SignedEnvironmentExport, collect_environment, save_signed_environment_export, sign_environment_inventory, +}; pub use heartbeat::{CoarseNodeSummary, HeartbeatError, HeartbeatStatus}; pub use identity::{DeviceIdentity, IdentityError, RegistrationProof, RegistrationTranscript}; pub use identity_store::{IdentityStore, StoreError}; @@ -48,7 +136,22 @@ pub use inventory::{ InventoryError, InventoryFlag, InventoryOsVersion, InventorySchedule, InventorySnapshot, InventoryStatus, OperatingSystemFamily, }; +pub use license::{ + LICENSE_DOMAIN_SEPARATION_TAG, LicenseArtifactError, LicenseArtifactStatus, LicenseClaims, LicenseReport, + LicenseVerificationContext, apply_license_artifact, inspect_installed_license, verify_license_artifact, +}; +pub use license_relay::{ + ServiceLicenseRelayError, ServiceLicenseRelayExport, ServiceLicenseRelayReceipt, export_service_license_relay, + receive_service_license_relay, +}; +pub use license_renewal::{LicenseRenewalClient, LicenseRenewalError, LicenseRenewalOutcome}; pub use offline::{EnrollmentError, OfflineEnrollment, OfflineKeyStore, VerifiedChallenge}; pub use registration::{RegistrationToken, TokenError}; pub use registration_bootstrap::{RegistrationBootstrapError, RegistrationBootstrapResult, register_from_protected_input}; +pub use relay::{ + DestinationReceiptSigner, RelayDirection, RelayError, RelayHttpClient, RelayMaterialKind, RelayParty, RelayReceiptOutcome, + RelayReceiptPayload, RelayReview, TrustedReceiptSigner, decode_relay_envelope, prepare_approved_artifact, + read_protected_relay_artifact, read_protected_relay_authentication, +}; +pub use report_upload::{MAX_SUPPORT_BUNDLE_BYTES, ReportUploadClient, ReportUploadError, ReportUploadReceipt}; pub use runtime::{HeartbeatRuntime, InventoryRuntime, spawn_heartbeat_runtime, spawn_inventory_runtime}; diff --git a/rustfs/src/connect/offline/collectors.rs b/rustfs/src/connect/offline/collectors.rs index 4cfdce16a..b6ff40ae8 100644 --- a/rustfs/src/connect/offline/collectors.rs +++ b/rustfs/src/connect/offline/collectors.rs @@ -14,25 +14,21 @@ //! Fixed Q07 L0/L1 collectors for an operator-triggered offline diagnostic. -use std::collections::BTreeSet; use std::path::Path; -use std::sync::{Arc, LazyLock}; use std::time::Duration; use serde::Serialize; use serde_json::{Value, json}; -use sysinfo::{Disks, Networks, RefreshKind, System}; use thiserror::Error; -use tokio::sync::Semaphore; use tokio_util::sync::CancellationToken; +use super::super::environment::{EnvironmentError, HostEnvironment, collect_host_environment}; use super::super::inventory::{InventoryError, InventorySnapshot, InventoryStateStore}; use super::manifest_entry::ManifestEntry; use super::redaction::RedactionError; const COLLECT_TIMEOUT: Duration = Duration::from_secs(2); const MAX_ENTRY_BYTES: usize = 16 * 1024; -static SYSTEM_SCAN_PERMIT: LazyLock> = LazyLock::new(|| Arc::new(Semaphore::new(1))); #[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] pub enum DataClassification { @@ -118,7 +114,7 @@ impl OfflineCollector { self.field_id().split_once('.').expect("collector field ids are frozen").1 } - fn value(self, inventory: &InventorySnapshot, system: &SystemSnapshot) -> Value { + fn value(self, inventory: &InventorySnapshot, system: &HostEnvironment) -> Value { match self { Self::RustfsVersion => json!(inventory.rustfs_version()), Self::NodeCount => json!(inventory.node_count()), @@ -150,6 +146,8 @@ pub enum CollectorError { TimedOut, #[error("offline diagnostic collector task failed")] TaskFailed, + #[error("offline diagnostic source is unavailable")] + SourceUnavailable, #[error("offline diagnostic field {field_id} exceeds its {limit} byte entry budget")] EntryTooLarge { field_id: &'static str, limit: usize }, #[error("offline diagnostic entry is not representable as JSON")] @@ -160,6 +158,24 @@ pub enum CollectorError { Redaction(#[from] RedactionError), } +impl From for CollectorError { + fn from(error: EnvironmentError) -> Self { + match error { + EnvironmentError::Cancelled => Self::Cancelled, + EnvironmentError::TimedOut => Self::TimedOut, + EnvironmentError::TaskFailed => Self::TaskFailed, + EnvironmentError::SourceUnavailable(_) => Self::SourceUnavailable, + EnvironmentError::UnsupportedVersion + | EnvironmentError::UnsupportedCapability + | EnvironmentError::InvalidTimeout + | EnvironmentError::InvalidExport + | EnvironmentError::ExportEncoding + | EnvironmentError::ExportSigning + | EnvironmentError::ExportIo => Self::TaskFailed, + } + } +} + /// The bounded entries plus the capture time of their persisted L0 source. #[derive(Debug, PartialEq)] pub struct OfflineDiagnostics { @@ -168,79 +184,6 @@ pub struct OfflineDiagnostics { pub inventory_age: Duration, } -#[derive(Debug)] -struct SystemSnapshot { - os_summary: String, - kernel_summary: String, - architecture: &'static str, - cores: usize, - total_memory_bytes: u64, - under_memory_pressure: bool, - filesystem_types: Vec, - interface_count: usize, - bond_count: usize, -} - -impl SystemSnapshot { - fn collect() -> Self { - #[cfg(test)] - let _scan = test_support::ScanGuard::start(); - - // Do not enumerate processes: Q07 allows only CPU and memory summaries. - let system = System::new_with_specifics(RefreshKind::everything().without_processes()); - let total_memory_bytes = system.total_memory(); - let available_memory = system.available_memory(); - let filesystem_types = Disks::new_with_refreshed_list() - .iter() - .map(|disk| disk.file_system().to_string_lossy().into_owned()) - .collect::>() - .into_iter() - .collect(); - let networks = Networks::new_with_refreshed_list(); - Self { - os_summary: System::long_os_version().unwrap_or_else(|| "unknown".to_owned()), - kernel_summary: System::kernel_long_version(), - architecture: std::env::consts::ARCH, - cores: system.cpus().len(), - total_memory_bytes, - under_memory_pressure: total_memory_bytes != 0 && available_memory.saturating_mul(10) < total_memory_bytes, - filesystem_types, - interface_count: networks.len(), - bond_count: networks.keys().filter(|name| name.starts_with("bond")).count(), - } - } -} - -async fn collect_system_snapshot(cancel: &CancellationToken) -> Result { - let deadline = tokio::time::Instant::now() + COLLECT_TIMEOUT; - let permit = tokio::select! { - biased; - () = cancel.cancelled() => return Err(CollectorError::Cancelled), - result = tokio::time::timeout_at(deadline, SYSTEM_SCAN_PERMIT.clone().acquire_owned()) => { - match result { - Ok(Ok(permit)) => permit, - Ok(Err(_)) => return Err(CollectorError::TaskFailed), - Err(_) => return Err(CollectorError::TimedOut), - } - } - }; - let task = tokio::task::spawn_blocking(move || { - let _permit = permit; - SystemSnapshot::collect() - }); - tokio::select! { - biased; - () = cancel.cancelled() => Err(CollectorError::Cancelled), - result = tokio::time::timeout_at(deadline, task) => { - match result { - Ok(Ok(snapshot)) => Ok(snapshot), - Ok(Err(_)) => Err(CollectorError::TaskFailed), - Err(_) => Err(CollectorError::TimedOut), - } - } - } -} - /// Collect all and only the Q07 offline L0/L1 fields after acquiring the /// stopped-runtime inventory lock. pub async fn collect_offline_diagnostics( @@ -253,7 +196,7 @@ pub async fn collect_offline_diagnostics( let store = InventoryStateStore::from_state_root(state_root)?; let _lock = store.try_runtime_lock()?; let persisted = store.read_latest(chrono::Utc::now())?; - let system = collect_system_snapshot(cancel).await?; + let system = collect_host_environment(COLLECT_TIMEOUT, cancel).await?; let mut entries = Vec::with_capacity(COLLECTORS.len()); for collector in COLLECTORS { @@ -269,102 +212,3 @@ pub async fn collect_offline_diagnostics( inventory_age: persisted.age, }) } - -#[cfg(test)] -mod test_support { - use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; - use std::time::Duration; - - pub(super) static DELAY_MILLIS: AtomicU64 = AtomicU64::new(0); - pub(super) static ACTIVE: AtomicUsize = AtomicUsize::new(0); - pub(super) static MAX_ACTIVE: AtomicUsize = AtomicUsize::new(0); - - pub(super) struct ScanGuard; - - impl ScanGuard { - pub(super) fn start() -> Self { - let active = ACTIVE.fetch_add(1, Ordering::SeqCst) + 1; - MAX_ACTIVE.fetch_max(active, Ordering::SeqCst); - let delay = DELAY_MILLIS.load(Ordering::SeqCst); - if delay != 0 { - std::thread::sleep(Duration::from_millis(delay)); - } - Self - } - } - - impl Drop for ScanGuard { - fn drop(&mut self) { - ACTIVE.fetch_sub(1, Ordering::SeqCst); - } - } -} - -#[cfg(test)] -mod tests { - use std::sync::atomic::Ordering; - - use super::*; - - async fn wait_for_active(expected: usize) { - tokio::time::timeout(Duration::from_secs(1), async { - while test_support::ACTIVE.load(Ordering::SeqCst) != expected { - tokio::time::sleep(Duration::from_millis(5)).await; - } - }) - .await - .expect("system scan reaches expected state"); - } - - #[tokio::test] - async fn connect_offline_collectors_timeout_and_cancel_never_overlap_system_scans() { - test_support::MAX_ACTIVE.store(0, Ordering::SeqCst); - test_support::DELAY_MILLIS.store((COLLECT_TIMEOUT + Duration::from_millis(200)).as_millis() as u64, Ordering::SeqCst); - - let first_cancel = CancellationToken::new(); - assert!(matches!(collect_system_snapshot(&first_cancel).await, Err(CollectorError::TimedOut))); - assert_eq!(test_support::ACTIVE.load(Ordering::SeqCst), 1, "timed-out blocking scan remains active"); - - let second_cancel = CancellationToken::new(); - let second = tokio::spawn({ - let second_cancel = second_cancel.clone(); - async move { collect_system_snapshot(&second_cancel).await } - }); - tokio::time::sleep(Duration::from_millis(50)).await; - assert_eq!( - test_support::MAX_ACTIVE.load(Ordering::SeqCst), - 1, - "a timed-out scan keeps the single-flight permit" - ); - second_cancel.cancel(); - assert!(matches!(second.await.expect("second collector task"), Err(CollectorError::Cancelled))); - wait_for_active(0).await; - - test_support::MAX_ACTIVE.store(0, Ordering::SeqCst); - test_support::DELAY_MILLIS.store(250, Ordering::SeqCst); - let third_cancel = CancellationToken::new(); - let third = tokio::spawn({ - let third_cancel = third_cancel.clone(); - async move { collect_system_snapshot(&third_cancel).await } - }); - wait_for_active(1).await; - third_cancel.cancel(); - assert!(matches!(third.await.expect("third collector task"), Err(CollectorError::Cancelled))); - - let fourth_cancel = CancellationToken::new(); - let fourth = tokio::spawn({ - let fourth_cancel = fourth_cancel.clone(); - async move { collect_system_snapshot(&fourth_cancel).await } - }); - tokio::time::sleep(Duration::from_millis(50)).await; - assert_eq!( - test_support::MAX_ACTIVE.load(Ordering::SeqCst), - 1, - "a cancelled scan keeps the single-flight permit" - ); - fourth_cancel.cancel(); - assert!(matches!(fourth.await.expect("fourth collector task"), Err(CollectorError::Cancelled))); - wait_for_active(0).await; - test_support::DELAY_MILLIS.store(0, Ordering::SeqCst); - } -} diff --git a/rustfs/src/connect/registration_bootstrap.rs b/rustfs/src/connect/registration_bootstrap.rs index 6231941ac..3c12f820c 100644 --- a/rustfs/src/connect/registration_bootstrap.rs +++ b/rustfs/src/connect/registration_bootstrap.rs @@ -25,7 +25,7 @@ use std::{ use super::TokenError; #[cfg(unix)] -use super::{ConnectClient, ConnectConfig, CredentialStore, IdentityStore, RegistrationToken}; +use super::{ConnectClient, ConnectConfig, CredentialStore, IdentityStore, ProxyConfig, RegistrationToken}; #[cfg(unix)] const REQUEST_TIMEOUT: Duration = Duration::from_secs(15); @@ -58,6 +58,16 @@ pub enum RegistrationBootstrapError { Input(#[source] io::Error), #[error("Connect registration configuration is invalid")] Configuration, + #[error("Connect registration proxy configuration is invalid")] + ProxyConfiguration, + #[error("Connect proxy authentication failed; verify the configured proxy credential files")] + ProxyAuthentication, + #[error( + "Connect proxy connection failed; verify proxy availability, credentials, the proxy allow-list, and the Connect endpoint" + )] + ProxyRejected, + #[error("Connect TLS peer certificate validation failed; verify the endpoint and configured root CA")] + TlsPeer, #[error("Connect registration exchange failed")] Exchange, #[error("Connect registration bootstrap requires Unix owner and permission guarantees")] @@ -85,10 +95,12 @@ pub async fn register_from_protected_input( token_file: Option<&Path>, ) -> Result { let root_ca_pem = read_regular_file(root_ca_file, false)?; + let proxy = ProxyConfig::from_env().map_err(|_| RegistrationBootstrapError::ProxyConfiguration)?; let client = ConnectClient::new(ConnectConfig { endpoint, root_ca_pem: &root_ca_pem, timeout: REQUEST_TIMEOUT, + proxy: proxy.as_ref(), }) .map_err(|_| RegistrationBootstrapError::Configuration)?; @@ -105,7 +117,12 @@ pub async fn register_from_protected_input( &token, ) .await - .map_err(|_| RegistrationBootstrapError::Exchange)?; + .map_err(|error| match error { + super::ClientError::ProxyAuthentication => RegistrationBootstrapError::ProxyAuthentication, + super::ClientError::ProxyRejected => RegistrationBootstrapError::ProxyRejected, + super::ClientError::TlsPeer => RegistrationBootstrapError::TlsPeer, + _ => RegistrationBootstrapError::Exchange, + })?; if credential.name != format!("{cluster_name}/clusterDevices/{}", credential.uid) { return Err(RegistrationBootstrapError::Exchange); } diff --git a/rustfs/src/connect/relay.rs b/rustfs/src/connect/relay.rs new file mode 100644 index 000000000..210912cbb --- /dev/null +++ b/rustfs/src/connect/relay.rs @@ -0,0 +1,728 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Customer-operated transfer of opaque, signed Connect material. +//! +//! The relay has no signing authority. It carries the exact approved bytes and +//! accepts success only from a receipt signed by the preconfigured destination. + +use base64_simd::{STANDARD as BASE64_STANDARD, URL_SAFE_NO_PAD}; +use ed25519_dalek::{Signature, Signer as _, SigningKey, VerifyingKey}; +use reqwest::{Client, StatusCode, Url, header}; +use rustls::pki_types::{CertificateDer, pem::PemObject as _}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +#[cfg(unix)] +use std::fs::OpenOptions; +use std::io::Read as _; +#[cfg(unix)] +use std::os::unix::fs::{MetadataExt as _, OpenOptionsExt as _, PermissionsExt as _}; +use std::path::Path; +use std::time::Duration; +use uuid::{Uuid, Variant, Version}; + +use super::client::build_client; +use super::config::ProxyConfig; + +pub const RELAY_ENVELOPE_FORMAT: &str = "rustfs.connect.relayEnvelope/1"; +pub const RELAY_RECEIPT_FORMAT: &str = "rustfs.connect.relayReceipt/1"; +pub const RELAY_RECEIPT_DOMAIN_SEPARATION_TAG: &str = "rustfs-connect-relay-receipt-v1"; +pub const MAX_RELAY_ARTIFACT_BYTES: usize = 16 * 1024 * 1024; +pub const MAX_DELIVERY_ATTEMPTS: usize = 3; +const MAX_RECEIPT_BYTES: usize = 64 * 1024; +const RETRY_DELAY: Duration = Duration::from_millis(250); +const MAX_AUTHENTICATION_BYTES: u64 = 8 * 1024; +const MAX_PUBLIC_KEY_BYTES: u64 = 256; +const MAX_PRIVATE_KEY_BYTES: u64 = 256; +const MAX_RELAY_ENVELOPE_BYTES: usize = MAX_RELAY_ARTIFACT_BYTES * 2 + 64 * 1024; + +#[derive(Clone, Copy, Debug, Deserialize, Eq, Hash, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum RelayMaterialKind { + OfflineEnrollmentResponse, + DiagnosticBundleManifest, + ServiceLicense, +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum RelayDirection { + ClusterToConnect, + ConnectToCluster, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +pub struct RelayParty { + #[serde(rename = "type")] + pub party_type: String, + pub name: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub key_id: Option, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +pub struct RelayArtifact { + pub encoding: String, + pub bytes: String, + pub sha256: String, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +pub struct RelayEnvelope { + pub format_version: String, + pub protocol_version: String, + pub transfer_uid: String, + pub material_kind: RelayMaterialKind, + pub direction: RelayDirection, + pub artifact: RelayArtifact, + pub asserted_producer: RelayParty, + pub destination: RelayParty, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct RelayReview { + pub transfer_uid: String, + pub material_kind: RelayMaterialKind, + pub direction: RelayDirection, + pub artifact_sha256: String, + pub artifact_size_bytes: usize, + pub asserted_producer: RelayParty, + pub destination: RelayParty, +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum RelayReceiptOutcome { + Applied, + Duplicate, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +pub struct RelayReceiptPayload { + pub format_version: String, + pub protocol_version: String, + pub transfer_uid: String, + pub material_kind: RelayMaterialKind, + pub direction: RelayDirection, + pub artifact_sha256: String, + pub producer: RelayParty, + pub destination: RelayParty, + pub outcome: RelayReceiptOutcome, + pub received_at: String, +} + +#[derive(Debug, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +#[serde(rename_all = "camelCase")] +struct RelayReceiptSignature { + algorithm: String, + key_id: String, + value: String, +} + +#[derive(Debug, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +struct SignedRelayReceipt { + payload: String, + signature: RelayReceiptSignature, +} + +#[derive(Clone, Debug)] +pub struct TrustedReceiptSigner { + key_id: String, + verifying_key: VerifyingKey, +} + +#[derive(Clone)] +pub struct DestinationReceiptSigner { + key_id: String, + signing_key: SigningKey, +} + +impl DestinationReceiptSigner { + pub fn new(key_id: String, seed: [u8; 32]) -> Result { + let signing_key = SigningKey::from_bytes(&seed); + let actual_key_id = hex_lower(&Sha256::digest(signing_key.verifying_key().as_bytes())); + if !is_sha256(&key_id) || key_id != actual_key_id { + return Err(RelayError::ReceiptTrustInvalid); + } + Ok(Self { key_id, signing_key }) + } + + pub fn from_private_key_file(path: &Path, key_id: String) -> Result { + let encoded = read_protected_bytes(path, MAX_PRIVATE_KEY_BYTES)?; + let encoded = std::str::from_utf8(&encoded) + .map_err(|_| RelayError::ReceiptTrustInvalid)? + .trim(); + let seed = decode_canonical_base64url(encoded).ok_or(RelayError::ReceiptTrustInvalid)?; + let seed: [u8; 32] = seed.try_into().map_err(|_| RelayError::ReceiptTrustInvalid)?; + Self::new(key_id, seed) + } + + pub(crate) fn sign( + &self, + envelope: &RelayEnvelope, + producer: RelayParty, + outcome: RelayReceiptOutcome, + received_at: String, + ) -> Result, RelayError> { + let payload = RelayReceiptPayload { + format_version: RELAY_RECEIPT_FORMAT.to_owned(), + protocol_version: envelope.protocol_version.clone(), + transfer_uid: envelope.transfer_uid.clone(), + material_kind: envelope.material_kind, + direction: envelope.direction, + artifact_sha256: envelope.artifact.sha256.clone(), + producer, + destination: envelope.destination.clone(), + outcome, + received_at, + }; + let payload = serde_json::to_vec(&payload).map_err(|_| RelayError::ReceiptEncoding)?; + let mut signed = Vec::with_capacity(RELAY_RECEIPT_DOMAIN_SEPARATION_TAG.len() + 1 + payload.len()); + signed.extend_from_slice(RELAY_RECEIPT_DOMAIN_SEPARATION_TAG.as_bytes()); + signed.push(0); + signed.extend_from_slice(&payload); + let signature = self.signing_key.sign(&signed).to_bytes(); + serde_json::to_vec(&SignedRelayReceipt { + payload: URL_SAFE_NO_PAD.encode_to_string(&payload), + signature: RelayReceiptSignature { + algorithm: "Ed25519".to_owned(), + key_id: self.key_id.clone(), + value: URL_SAFE_NO_PAD.encode_to_string(signature), + }, + }) + .map_err(|_| RelayError::ReceiptEncoding) + } +} + +impl TrustedReceiptSigner { + pub fn new(key_id: String, public_key: [u8; 32]) -> Result { + if !is_sha256(&key_id) || hex_lower(&Sha256::digest(public_key)) != key_id { + return Err(RelayError::ReceiptTrustInvalid); + } + let verifying_key = VerifyingKey::from_bytes(&public_key).map_err(|_| RelayError::ReceiptTrustInvalid)?; + Ok(Self { key_id, verifying_key }) + } + + pub fn from_public_key_file(path: &Path, key_id: String) -> Result { + let encoded = read_protected_bytes(path, MAX_PUBLIC_KEY_BYTES)?; + let encoded = std::str::from_utf8(&encoded) + .map_err(|_| RelayError::ReceiptTrustInvalid)? + .trim(); + let public_key = decode_canonical_base64url(encoded).ok_or(RelayError::ReceiptTrustInvalid)?; + let public_key: [u8; 32] = public_key.try_into().map_err(|_| RelayError::ReceiptTrustInvalid)?; + Self::new(key_id, public_key) + } +} + +#[derive(Clone, Debug, Eq, Hash, PartialEq)] +pub struct RelayReplayKey { + material_kind: RelayMaterialKind, + artifact_sha256: String, + destination_type: String, + destination_name: String, +} + +impl RelayEnvelope { + /// The durable destination uses this key after material-specific signature, + /// freshness, revocation, and scope verification. It must be persisted with + /// the terminal receipt before acknowledging a side effect. + pub fn replay_key(&self) -> RelayReplayKey { + RelayReplayKey { + material_kind: self.material_kind, + artifact_sha256: self.artifact.sha256.clone(), + destination_type: self.destination.party_type.clone(), + destination_name: self.destination.name.clone(), + } + } +} + +#[derive(Debug, PartialEq, Eq, thiserror::Error)] +pub enum RelayError { + #[error("the relay material kind or direction is unsupported")] + UnsupportedMaterial, + #[error("the relay transfer identifier is not a canonical UUIDv7")] + InvalidTransferUid, + #[error("the relay producer or destination identity is invalid")] + InvalidParty, + #[error("the relay artifact is empty or exceeds its size limit")] + InvalidArtifact, + #[error("the customer did not approve this relay transfer")] + ApprovalRequired, + #[error("the relay envelope could not be encoded")] + EnvelopeEncoding, + #[error("the relay envelope is malformed")] + EnvelopeInvalid, + #[error("the relay artifact digest does not match the exact decoded bytes")] + ArtifactDigestMismatch, + #[error("the destination receipt trust configuration is invalid")] + ReceiptTrustInvalid, + #[error("the destination receipt is malformed")] + ReceiptInvalid, + #[error("the destination receipt key is not trusted")] + ReceiptSignerUntrusted, + #[error("the destination receipt signature is invalid")] + ReceiptSignatureInvalid, + #[error("the destination receipt does not bind this transfer")] + ReceiptMismatch, + #[error("the destination receipt could not be encoded")] + ReceiptEncoding, + #[error("delivery produced no verified receipt after three attempts")] + DeliveryUnknown, + #[error("the relay control API rejected delivery with HTTP {0}")] + DeliveryRejected(u16), + #[error("the relay control API response exceeded its size limit")] + ResponseTooLarge, + #[error("the relay HTTP authentication header is invalid")] + AuthenticationInvalid, + #[error("the relay input file is unavailable or exceeds its size limit")] + InputFile, + #[error("the relay credential file is not an owner-only regular file")] + CredentialFileSecurity, + #[error("the relay HTTPS client configuration is invalid")] + ClientConfiguration, + #[error("the relay HTTPS request failed")] + Transport, +} + +pub trait RelayTransport { + fn deliver(&mut self, envelope: &[u8]) -> Result>, RelayError>; +} + +#[derive(Debug)] +pub struct RelayDelivery { + pub review: RelayReview, + pub envelope: RelayEnvelope, + pub receipt: RelayReceiptPayload, + pub receipt_bytes: Vec, + pub attempts: usize, +} + +pub struct PreparedRelay { + pub review: RelayReview, + pub envelope: RelayEnvelope, +} + +pub struct RelayHttpClient { + client: Client, + receive_url: Url, + cookie: header::HeaderValue, + csrf_token: header::HeaderValue, + approval_reference: String, +} + +impl RelayHttpClient { + #[allow(clippy::too_many_arguments)] + pub fn new( + endpoint: &str, + root_ca_pem: &[u8], + organization_uid: &str, + approval_reference: String, + cookie: &str, + csrf_token: &str, + timeout: Duration, + proxy: Option<&ProxyConfig>, + ) -> Result { + let endpoint = Url::parse(endpoint).map_err(|_| RelayError::ClientConfiguration)?; + if endpoint.scheme() != "https" + || endpoint.cannot_be_a_base() + || !endpoint.username().is_empty() + || endpoint.password().is_some() + || endpoint.query().is_some() + || endpoint.fragment().is_some() + || !endpoint.path().ends_with("/api/") + || !is_uuid_v7(organization_uid) + || approval_reference.is_empty() + || approval_reference.len() > 512 + { + return Err(RelayError::ClientConfiguration); + } + let receive_url = endpoint + .join(&format!("organizations/{organization_uid}/relayTransfers:receive")) + .map_err(|_| RelayError::ClientConfiguration)?; + let roots = CertificateDer::pem_slice_iter(root_ca_pem) + .collect::, _>>() + .map_err(|_| RelayError::ClientConfiguration)?; + if roots.is_empty() { + return Err(RelayError::ClientConfiguration); + } + let client = build_client(&roots, timeout, None, proxy).map_err(|_| RelayError::ClientConfiguration)?; + let mut cookie = header::HeaderValue::from_str(cookie).map_err(|_| RelayError::AuthenticationInvalid)?; + let mut csrf_token = header::HeaderValue::from_str(csrf_token).map_err(|_| RelayError::AuthenticationInvalid)?; + cookie.set_sensitive(true); + csrf_token.set_sensitive(true); + Ok(Self { + client, + receive_url, + cookie, + csrf_token, + approval_reference, + }) + } + + pub async fn deliver( + &self, + prepared: PreparedRelay, + trusted_receipt_signer: &TrustedReceiptSigner, + ) -> Result { + #[derive(Serialize)] + #[serde(rename_all = "camelCase")] + struct Request<'a> { + approval_reference: &'a str, + envelope: &'a RelayEnvelope, + } + + for attempts in 1..=MAX_DELIVERY_ATTEMPTS { + let response = self + .client + .post(self.receive_url.clone()) + .header(header::COOKIE, self.cookie.clone()) + .header("X-XSRF-TOKEN", self.csrf_token.clone()) + .header(header::ACCEPT, "application/json") + .json(&Request { + approval_reference: &self.approval_reference, + envelope: &prepared.envelope, + }) + .send() + .await; + let response = match response { + Ok(response) => response, + Err(_) if attempts < MAX_DELIVERY_ATTEMPTS => { + tokio::time::sleep(RETRY_DELAY).await; + continue; + } + Err(_) => return Err(RelayError::Transport), + }; + if retryable_status(response.status()) && attempts < MAX_DELIVERY_ATTEMPTS { + tokio::time::sleep(RETRY_DELAY).await; + continue; + } + if response.status() != StatusCode::OK { + return Err(RelayError::DeliveryRejected(response.status().as_u16())); + } + let receipt_bytes = bounded_body(response).await?; + let receipt = verify_receipt(&receipt_bytes, &prepared.envelope, trusted_receipt_signer)?; + return Ok(RelayDelivery { + review: prepared.review, + envelope: prepared.envelope, + receipt, + receipt_bytes, + attempts, + }); + } + Err(RelayError::DeliveryUnknown) + } +} + +#[allow(clippy::too_many_arguments)] +/// Build, approve, and deliver one relay envelope without changing the signed +/// material bytes. Producer fields are advisory until the destination derives +/// and compares them using its material-specific verifier. +pub fn relay_approved_artifact( + transfer_uid: &str, + material_kind: RelayMaterialKind, + artifact_bytes: &[u8], + asserted_producer: RelayParty, + destination: RelayParty, + trusted_receipt_signer: &TrustedReceiptSigner, + approve: A, + transport: &mut T, +) -> Result +where + T: RelayTransport, + A: FnOnce(&RelayReview) -> bool, +{ + let prepared = + prepare_approved_artifact(transfer_uid, material_kind, artifact_bytes, asserted_producer, destination, approve)?; + let PreparedRelay { review, envelope } = prepared; + let encoded = serde_json::to_vec(&envelope).map_err(|_| RelayError::EnvelopeEncoding)?; + + for attempts in 1..=MAX_DELIVERY_ATTEMPTS { + let Ok(Some(receipt_bytes)) = transport.deliver(&encoded) else { + continue; + }; + let receipt = verify_receipt(&receipt_bytes, &envelope, trusted_receipt_signer)?; + return Ok(RelayDelivery { + review, + envelope, + receipt, + receipt_bytes, + attempts, + }); + } + Err(RelayError::DeliveryUnknown) +} + +pub fn prepare_approved_artifact( + transfer_uid: &str, + material_kind: RelayMaterialKind, + artifact_bytes: &[u8], + asserted_producer: RelayParty, + destination: RelayParty, + approve: A, +) -> Result +where + A: FnOnce(&RelayReview) -> bool, +{ + validate_transfer_uid(transfer_uid)?; + validate_route(material_kind, &asserted_producer, &destination)?; + if artifact_bytes.is_empty() || artifact_bytes.len() > MAX_RELAY_ARTIFACT_BYTES { + return Err(RelayError::InvalidArtifact); + } + + let artifact_sha256 = hex_lower(&Sha256::digest(artifact_bytes)); + let review = RelayReview { + transfer_uid: transfer_uid.to_owned(), + material_kind, + direction: direction_for(material_kind), + artifact_sha256: artifact_sha256.clone(), + artifact_size_bytes: artifact_bytes.len(), + asserted_producer: asserted_producer.clone(), + destination: destination.clone(), + }; + if !approve(&review) { + return Err(RelayError::ApprovalRequired); + } + + let envelope = RelayEnvelope { + format_version: RELAY_ENVELOPE_FORMAT.to_owned(), + protocol_version: "v1".to_owned(), + transfer_uid: transfer_uid.to_owned(), + material_kind, + direction: review.direction, + artifact: RelayArtifact { + encoding: "base64".to_owned(), + bytes: BASE64_STANDARD.encode_to_string(artifact_bytes), + sha256: artifact_sha256, + }, + asserted_producer, + destination, + }; + Ok(PreparedRelay { review, envelope }) +} + +pub fn decode_relay_envelope(envelope_bytes: &[u8]) -> Result<(RelayEnvelope, Vec), RelayError> { + if envelope_bytes.is_empty() || envelope_bytes.len() > MAX_RELAY_ENVELOPE_BYTES { + return Err(RelayError::EnvelopeInvalid); + } + let envelope: RelayEnvelope = serde_json::from_slice(envelope_bytes).map_err(|_| RelayError::EnvelopeInvalid)?; + if envelope.format_version != RELAY_ENVELOPE_FORMAT + || envelope.protocol_version != "v1" + || validate_transfer_uid(&envelope.transfer_uid).is_err() + || validate_route(envelope.material_kind, &envelope.asserted_producer, &envelope.destination).is_err() + || envelope.direction != direction_for(envelope.material_kind) + || envelope.artifact.encoding != "base64" + || !is_sha256(&envelope.artifact.sha256) + { + return Err(RelayError::EnvelopeInvalid); + } + let artifact_bytes = BASE64_STANDARD + .decode_to_vec(envelope.artifact.bytes.as_bytes()) + .map_err(|_| RelayError::EnvelopeInvalid)?; + if artifact_bytes.is_empty() + || artifact_bytes.len() > MAX_RELAY_ARTIFACT_BYTES + || BASE64_STANDARD.encode_to_string(&artifact_bytes) != envelope.artifact.bytes + { + return Err(RelayError::InvalidArtifact); + } + if hex_lower(&Sha256::digest(&artifact_bytes)) != envelope.artifact.sha256 { + return Err(RelayError::ArtifactDigestMismatch); + } + Ok((envelope, artifact_bytes)) +} + +pub fn verify_receipt( + receipt_bytes: &[u8], + envelope: &RelayEnvelope, + trusted_signer: &TrustedReceiptSigner, +) -> Result { + let signed: SignedRelayReceipt = serde_json::from_slice(receipt_bytes).map_err(|_| RelayError::ReceiptInvalid)?; + if signed.signature.algorithm != "Ed25519" || signed.signature.key_id != trusted_signer.key_id { + return Err(RelayError::ReceiptSignerUntrusted); + } + + let payload = decode_canonical_base64url(&signed.payload).ok_or(RelayError::ReceiptInvalid)?; + let signature = decode_canonical_base64url(&signed.signature.value).ok_or(RelayError::ReceiptInvalid)?; + let signature: [u8; 64] = signature.try_into().map_err(|_| RelayError::ReceiptInvalid)?; + let mut signed_bytes = Vec::with_capacity(RELAY_RECEIPT_DOMAIN_SEPARATION_TAG.len() + 1 + payload.len()); + signed_bytes.extend_from_slice(RELAY_RECEIPT_DOMAIN_SEPARATION_TAG.as_bytes()); + signed_bytes.push(0); + signed_bytes.extend_from_slice(&payload); + trusted_signer + .verifying_key + .verify_strict(&signed_bytes, &Signature::from_bytes(&signature)) + .map_err(|_| RelayError::ReceiptSignatureInvalid)?; + + let receipt: RelayReceiptPayload = serde_json::from_slice(&payload).map_err(|_| RelayError::ReceiptInvalid)?; + if receipt.format_version != RELAY_RECEIPT_FORMAT + || receipt.protocol_version != envelope.protocol_version + || receipt.transfer_uid != envelope.transfer_uid + || receipt.material_kind != envelope.material_kind + || receipt.direction != envelope.direction + || receipt.artifact_sha256 != envelope.artifact.sha256 + || receipt.producer != envelope.asserted_producer + || receipt.destination != envelope.destination + { + return Err(RelayError::ReceiptMismatch); + } + Ok(receipt) +} + +fn validate_route( + material_kind: RelayMaterialKind, + asserted_producer: &RelayParty, + destination: &RelayParty, +) -> Result<(), RelayError> { + if !valid_party(asserted_producer) || !valid_party(destination) { + return Err(RelayError::InvalidParty); + } + let valid = match material_kind { + RelayMaterialKind::OfflineEnrollmentResponse | RelayMaterialKind::DiagnosticBundleManifest => { + asserted_producer.party_type == "DEVICE" + && asserted_producer.key_id.is_some() + && destination.party_type == "CONNECT" + && destination.key_id.is_none() + } + RelayMaterialKind::ServiceLicense => { + asserted_producer.party_type == "CONNECT_LICENSE_ISSUER" + && asserted_producer.key_id.is_some() + && destination.party_type == "CLUSTER" + && destination.key_id.is_none() + } + }; + valid.then_some(()).ok_or(RelayError::UnsupportedMaterial) +} + +const fn direction_for(material_kind: RelayMaterialKind) -> RelayDirection { + match material_kind { + RelayMaterialKind::OfflineEnrollmentResponse | RelayMaterialKind::DiagnosticBundleManifest => { + RelayDirection::ClusterToConnect + } + RelayMaterialKind::ServiceLicense => RelayDirection::ConnectToCluster, + } +} + +fn valid_party(party: &RelayParty) -> bool { + !party.party_type.is_empty() + && party.party_type.len() <= 64 + && !party.name.is_empty() + && party.name.len() <= 1024 + && party.key_id.as_deref().is_none_or(is_sha256) +} + +fn validate_transfer_uid(value: &str) -> Result<(), RelayError> { + is_uuid_v7(value).then_some(()).ok_or(RelayError::InvalidTransferUid) +} + +fn is_uuid_v7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn retryable_status(status: StatusCode) -> bool { + matches!(status.as_u16(), 408 | 425 | 429 | 500 | 502 | 503 | 504) +} + +async fn bounded_body(mut response: reqwest::Response) -> Result, RelayError> { + let mut body = Vec::new(); + while let Some(chunk) = response.chunk().await.map_err(|_| RelayError::Transport)? { + if body.len().saturating_add(chunk.len()) > MAX_RECEIPT_BYTES { + return Err(RelayError::ResponseTooLarge); + } + body.extend_from_slice(&chunk); + } + Ok(body) +} + +pub fn read_protected_relay_artifact(path: &Path) -> Result, RelayError> { + read_protected_bytes(path, MAX_RELAY_ARTIFACT_BYTES as u64) +} + +pub fn read_protected_relay_authentication(path: &Path) -> Result { + let bytes = read_protected_bytes(path, MAX_AUTHENTICATION_BYTES)?; + let value = std::str::from_utf8(&bytes) + .map_err(|_| RelayError::AuthenticationInvalid)? + .trim() + .to_owned(); + if value.is_empty() || value.bytes().any(|byte| byte == b'\r' || byte == b'\n') { + return Err(RelayError::AuthenticationInvalid); + } + Ok(value) +} + +#[cfg(unix)] +fn read_protected_bytes(path: &Path, maximum: u64) -> Result, RelayError> { + let file = OpenOptions::new() + .read(true) + .custom_flags(libc::O_NOFOLLOW) + .open(path) + .map_err(|_| RelayError::InputFile)?; + let metadata = file.metadata().map_err(|_| RelayError::InputFile)?; + if !metadata.is_file() || metadata.uid() != process_uid() || metadata.permissions().mode() & 0o077 != 0 { + return Err(RelayError::CredentialFileSecurity); + } + if metadata.len() == 0 || metadata.len() > maximum { + return Err(RelayError::InputFile); + } + let mut bytes = Vec::with_capacity(metadata.len() as usize); + file.take(maximum + 1) + .read_to_end(&mut bytes) + .map_err(|_| RelayError::InputFile)?; + if bytes.is_empty() || bytes.len() as u64 > maximum { + return Err(RelayError::InputFile); + } + Ok(bytes) +} + +#[cfg(unix)] +#[allow(unsafe_code)] +fn process_uid() -> u32 { + // SAFETY: geteuid has no pointer arguments or caller preconditions. + unsafe { libc::geteuid() } +} + +#[cfg(not(unix))] +fn read_protected_bytes(_path: &Path, _maximum: u64) -> Result, RelayError> { + Err(RelayError::CredentialFileSecurity) +} + +fn is_sha256(value: &str) -> bool { + value.len() == 64 + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +fn decode_canonical_base64url(value: &str) -> Option> { + if value.contains('=') { + return None; + } + let decoded = URL_SAFE_NO_PAD.decode_to_vec(value.as_bytes()).ok()?; + (URL_SAFE_NO_PAD.encode_to_string(&decoded) == value).then_some(decoded) +} + +fn hex_lower(bytes: &[u8]) -> String { + hex_simd::encode_to_string(bytes, hex_simd::AsciiCase::Lower) +} diff --git a/rustfs/src/connect/report_bundle.rs b/rustfs/src/connect/report_bundle.rs new file mode 100644 index 000000000..890629ac2 --- /dev/null +++ b/rustfs/src/connect/report_bundle.rs @@ -0,0 +1,603 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Wraps a signed typed diagnostic export in the signed support-bundle envelope Connect imports. + +use std::fs::File; +use std::io::{Read as _, Seek as _, SeekFrom, Write as _}; +use std::path::Path; + +use base64_simd::URL_SAFE_NO_PAD; +use p256::ecdsa::{Signature, SigningKey, VerifyingKey, signature::Signer as _, signature::Verifier as _}; +use p256::pkcs8::{DecodePrivateKey as _, DecodePublicKey as _}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use time::{OffsetDateTime, format_description::well_known::Rfc3339}; +use uuid::{Uuid, Variant, Version}; +use zip::{CompressionMethod, ZipArchive, ZipWriter, write::SimpleFileOptions}; + +use super::identity::DeviceIdentity; +use super::identity_store::IdentityStore; +use super::offline::redaction::{REDACTION_VERSION, RULESET_HASH}; + +const MANIFEST_PATH: &str = "manifest.json"; +const MANIFEST_SIGNATURE_PATH: &str = "manifest.sig"; +const ENVELOPE_PATH: &str = "envelope.json"; +const ENVELOPE_SIGNATURE_PATH: &str = "envelope.sig"; +const RESULT_PATH: &str = "result.json"; +const MANIFEST_DOMAIN: &[u8] = b"rustfs-support-bundle-v1"; +const ENVELOPE_DOMAIN: &[u8] = b"rustfs-diagnostic-envelope-v1"; +const MAX_ENVELOPE_BYTES: u64 = 16 * 1024; +const MAX_SIGNATURE_BYTES: u64 = 4 * 1024; +const MAX_RESULT_BYTES: u64 = 256 * 1024; +const OUTPUT_MODE: u32 = 0o600; + +pub(crate) struct UploadSource { + pub(crate) file: File, + pub(crate) bundle_uid: String, +} + +#[derive(Debug, thiserror::Error)] +pub(crate) enum ReportBundleError { + #[error("the diagnostic report archive is invalid")] + Invalid, + #[error("the diagnostic report archive has expired")] + Expired, + #[error("the Connect device private key is missing")] + IdentityMissing, + #[error("the diagnostic report archive could not be read")] + Io(#[from] std::io::Error), + #[error("the diagnostic report ZIP is invalid")] + Zip(#[from] zip::result::ZipError), +} + +pub(crate) fn upload_source( + path: &Path, + generated_bundle_uid: &str, + identities: &IdentityStore, +) -> Result { + if !is_uuid_v7(generated_bundle_uid) { + return Err(ReportBundleError::Invalid); + } + let mut file = File::open(path)?; + let mut archive = match ZipArchive::new(file) { + Ok(archive) => archive, + Err(_) => return original_source(path, generated_bundle_uid), + }; + let names = (0..archive.len()) + .map(|index| archive.by_index(index).map(|entry| entry.name().to_owned())) + .collect::, _>>()?; + + if names + .iter() + .any(|name| name == MANIFEST_PATH || name == MANIFEST_SIGNATURE_PATH) + { + if !names.iter().any(|name| name == MANIFEST_PATH) || !names.iter().any(|name| name == MANIFEST_SIGNATURE_PATH) { + return Err(ReportBundleError::Invalid); + } + let manifest = read_member(&mut archive, MANIFEST_PATH, 1024 * 1024)?; + let document: ExistingManifest = serde_json::from_slice(&manifest).map_err(|_| ReportBundleError::Invalid)?; + if !is_uuid_v7(&document.bundle_uid) { + return Err(ReportBundleError::Invalid); + } + file = archive.into_inner(); + file.seek(SeekFrom::Start(0))?; + return Ok(UploadSource { + file, + bundle_uid: document.bundle_uid, + }); + } + + let touches_diagnostic = names + .iter() + .any(|name| matches!(name.as_str(), ENVELOPE_PATH | ENVELOPE_SIGNATURE_PATH | RESULT_PATH)); + if !touches_diagnostic { + return original_source(path, generated_bundle_uid); + } + if names.len() != 3 + || ![ENVELOPE_PATH, ENVELOPE_SIGNATURE_PATH, RESULT_PATH] + .iter() + .all(|expected| names.iter().filter(|name| name.as_str() == *expected).count() == 1) + { + return Err(ReportBundleError::Invalid); + } + + let envelope_bytes = read_member(&mut archive, ENVELOPE_PATH, MAX_ENVELOPE_BYTES)?; + let envelope_signature_bytes = read_member(&mut archive, ENVELOPE_SIGNATURE_PATH, MAX_SIGNATURE_BYTES)?; + let result_bytes = read_member(&mut archive, RESULT_PATH, MAX_RESULT_BYTES)?; + let envelope: DiagnosticEnvelope = serde_json::from_slice(&envelope_bytes).map_err(|_| ReportBundleError::Invalid)?; + let signature: DiagnosticSignature = + serde_json::from_slice(&envelope_signature_bytes).map_err(|_| ReportBundleError::Invalid)?; + let now = OffsetDateTime::now_utc(); + let expires_at = OffsetDateTime::parse(&envelope.expires_at, &Rfc3339).map_err(|_| ReportBundleError::Invalid)?; + if expires_at <= now { + return Err(ReportBundleError::Expired); + } + validate_scope(&envelope)?; + validate_payload(&envelope.payload, &result_bytes)?; + if envelope.format_version != "rustfs.connect.diagnosticEnvelope/1" + || envelope.protocol_version != "v1" + || !matches!(envelope.classification.as_str(), "L0" | "L1" | "L2" | "L3") + || !is_nonce(&envelope.nonce) + { + return Err(ReportBundleError::Invalid); + } + + let identity = identities + .load() + .map_err(|_| ReportBundleError::IdentityMissing)? + .ok_or(ReportBundleError::IdentityMissing)?; + let device_key_id = hex_lower(&Sha256::digest(identity.public_key_der())); + if signature.algorithm != "ES256" || signature.key_id != device_key_id || envelope.device_key_id != device_key_id { + return Err(ReportBundleError::Invalid); + } + verify_envelope_signature(&identity, &envelope_bytes, &signature.value)?; + + let produced_at = now + .replace_nanosecond(0) + .map_err(|_| ReportBundleError::Invalid)? + .format(&Rfc3339) + .map_err(|_| ReportBundleError::Invalid)?; + let entries = [ + manifest_entry(ENVELOPE_PATH, &envelope_bytes, &envelope.classification), + manifest_entry(ENVELOPE_SIGNATURE_PATH, &envelope_signature_bytes, &envelope.classification), + manifest_entry(RESULT_PATH, &result_bytes, &envelope.classification), + ]; + let manifest = BundleManifest { + format_version: "rustfs.connect.support.bundleManifest/1", + protocol_version: "v1", + bundle_uid: generated_bundle_uid, + organization_name: &envelope.organization_name, + cluster_name: &envelope.cluster_name, + device_name: &envelope.device_name, + device_key_id: &device_key_id, + nonce: &envelope.nonce, + produced_at: &produced_at, + redaction_version: REDACTION_VERSION, + ruleset_hash: RULESET_HASH, + classification_registry_version: 1, + entries: &entries, + }; + let manifest_bytes = serde_json::to_vec(&manifest).map_err(|_| ReportBundleError::Invalid)?; + let manifest_signature = sign_manifest(&identity, &device_key_id, &manifest_bytes)?; + let mut file = tempfile::tempfile()?; + { + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .system(zip::System::Unix) + .unix_permissions(OUTPUT_MODE); + let mut output = ZipWriter::new(&mut file); + for (name, bytes) in [ + (ENVELOPE_PATH, envelope_bytes.as_slice()), + (ENVELOPE_SIGNATURE_PATH, envelope_signature_bytes.as_slice()), + (RESULT_PATH, result_bytes.as_slice()), + (MANIFEST_PATH, manifest_bytes.as_slice()), + (MANIFEST_SIGNATURE_PATH, manifest_signature.as_slice()), + ] { + output.start_file(name, options)?; + output.write_all(bytes)?; + } + output.finish()?; + } + file.seek(SeekFrom::Start(0))?; + Ok(UploadSource { + file, + bundle_uid: generated_bundle_uid.to_owned(), + }) +} + +fn original_source(path: &Path, bundle_uid: &str) -> Result { + Ok(UploadSource { + file: File::open(path)?, + bundle_uid: bundle_uid.to_owned(), + }) +} + +fn read_member(archive: &mut ZipArchive, name: &str, maximum: u64) -> Result, ReportBundleError> { + let entry = archive.by_name(name)?; + let size = entry.size(); + if size == 0 || size > maximum || !entry.is_file() { + return Err(ReportBundleError::Invalid); + } + let mut bytes = Vec::with_capacity(size as usize); + entry.take(maximum + 1).read_to_end(&mut bytes)?; + if bytes.len() as u64 != size { + return Err(ReportBundleError::Invalid); + } + Ok(bytes) +} + +fn validate_scope(envelope: &DiagnosticEnvelope) -> Result<(), ReportBundleError> { + let organization = envelope.organization_name.split('/').collect::>(); + let cluster = envelope.cluster_name.split('/').collect::>(); + let device = envelope.device_name.split('/').collect::>(); + if organization.len() != 2 + || organization[0] != "organizations" + || !is_uuid_v7(organization[1]) + || cluster.len() != 4 + || cluster[..2] != organization[..] + || cluster[2] != "clusters" + || !is_uuid_v7(cluster[3]) + || device.len() != 6 + || device[..4] != cluster[..] + || device[4] != "clusterDevices" + || !is_uuid_v7(device[5]) + { + return Err(ReportBundleError::Invalid); + } + Ok(()) +} + +fn validate_payload(payload: &DiagnosticPayload, result: &[u8]) -> Result<(), ReportBundleError> { + if payload.path != RESULT_PATH + || payload.media_type != "application/json" + || payload.size_bytes != result.len() as u64 + || payload.sha256 != hex_lower(&Sha256::digest(result)) + { + return Err(ReportBundleError::Invalid); + } + Ok(()) +} + +fn is_uuid_v7(value: &str) -> bool { + Uuid::parse_str(value).is_ok_and(|uuid| { + uuid.get_version() == Some(Version::SortRand) && uuid.get_variant() == Variant::RFC4122 && uuid.to_string() == value + }) +} + +fn is_nonce(value: &str) -> bool { + URL_SAFE_NO_PAD + .decode_to_vec(value) + .is_ok_and(|bytes| bytes.len() == 32 && URL_SAFE_NO_PAD.encode_to_string(&bytes) == value) +} + +fn verify_envelope_signature( + identity: &DeviceIdentity, + envelope: &[u8], + encoded_signature: &str, +) -> Result<(), ReportBundleError> { + let raw = URL_SAFE_NO_PAD + .decode_to_vec(encoded_signature) + .map_err(|_| ReportBundleError::Invalid)?; + if URL_SAFE_NO_PAD.encode_to_string(&raw) != encoded_signature { + return Err(ReportBundleError::Invalid); + } + let signature = Signature::from_slice(&raw).map_err(|_| ReportBundleError::Invalid)?; + if signature.normalize_s() != signature { + return Err(ReportBundleError::Invalid); + } + let key = VerifyingKey::from_public_key_der(&identity.public_key_der()).map_err(|_| ReportBundleError::Invalid)?; + let mut signed = Vec::with_capacity(ENVELOPE_DOMAIN.len() + 1 + envelope.len()); + signed.extend_from_slice(ENVELOPE_DOMAIN); + signed.push(0); + signed.extend_from_slice(envelope); + key.verify(&signed, &signature).map_err(|_| ReportBundleError::Invalid) +} + +fn sign_manifest(identity: &DeviceIdentity, key_id: &str, manifest: &[u8]) -> Result, ReportBundleError> { + let key = identity.to_pkcs8_der().map_err(|_| ReportBundleError::Invalid)?; + let key = SigningKey::from_pkcs8_der(key.as_slice()).map_err(|_| ReportBundleError::Invalid)?; + let mut signed = Vec::with_capacity(MANIFEST_DOMAIN.len() + 1 + manifest.len()); + signed.extend_from_slice(MANIFEST_DOMAIN); + signed.push(0); + signed.extend_from_slice(manifest); + let signature: Signature = key.sign(&signed); + serde_json::to_vec(&BundleSignature { + algorithm: "ES256", + key_id, + value: URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()), + }) + .map_err(|_| ReportBundleError::Invalid) +} + +fn manifest_entry<'a>(path: &'static str, bytes: &[u8], classification: &'a str) -> BundleManifestEntry<'a> { + BundleManifestEntry { + path, + entry_type: "offline-diagnostic", + size_bytes: bytes.len() as u64, + sha256: hex_lower(&Sha256::digest(bytes)), + classification, + } +} + +fn hex_lower(bytes: &[u8]) -> String { + hex_simd::encode_to_string(bytes, hex_simd::AsciiCase::Lower) +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct ExistingManifest { + bundle_uid: String, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct DiagnosticEnvelope { + format_version: String, + protocol_version: String, + organization_name: String, + cluster_name: String, + device_name: String, + classification: String, + expires_at: String, + nonce: String, + device_key_id: String, + payload: DiagnosticPayload, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct DiagnosticPayload { + path: String, + media_type: String, + size_bytes: u64, + sha256: String, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct DiagnosticSignature { + algorithm: String, + key_id: String, + value: String, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct BundleManifest<'a> { + format_version: &'static str, + protocol_version: &'static str, + bundle_uid: &'a str, + organization_name: &'a str, + cluster_name: &'a str, + device_name: &'a str, + device_key_id: &'a str, + nonce: &'a str, + produced_at: &'a str, + redaction_version: &'static str, + ruleset_hash: &'static str, + classification_registry_version: u8, + entries: &'a [BundleManifestEntry<'a>], +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct BundleManifestEntry<'a> { + path: &'static str, + #[serde(rename = "type")] + entry_type: &'static str, + size_bytes: u64, + sha256: String, + classification: &'a str, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct BundleSignature<'a> { + algorithm: &'static str, + key_id: &'a str, + value: String, +} + +#[cfg(test)] +mod tests { + use std::io::{Read as _, Seek as _, SeekFrom, Write as _}; + + use base64_simd::URL_SAFE_NO_PAD; + use p256::ecdsa::{Signature, SigningKey, VerifyingKey, signature::Signer as _, signature::Verifier as _}; + use p256::pkcs8::{DecodePrivateKey as _, DecodePublicKey as _}; + use serde_json::{Value, json}; + use sha2::{Digest as _, Sha256}; + use time::{Duration, OffsetDateTime, format_description::well_known::Rfc3339}; + use uuid::Uuid; + use zip::{CompressionMethod, ZipArchive, ZipWriter, write::SimpleFileOptions}; + + use super::{ENVELOPE_DOMAIN, IdentityStore, MANIFEST_DOMAIN, ReportBundleError, hex_lower, upload_source}; + + #[test] + fn wraps_signed_diagnostic_without_changing_original_members() { + let fixture = Fixture::new(Duration::minutes(5)); + let bundle_uid = Uuid::now_v7().to_string(); + let mut source = upload_source(&fixture.path, &bundle_uid, &fixture.identities).expect("wrap diagnostic"); + source.file.seek(SeekFrom::Start(0)).expect("seek wrapped archive"); + let mut archive = ZipArchive::new(source.file).expect("open wrapped archive"); + + assert_eq!(archive.len(), 5); + assert_eq!(member(&mut archive, "envelope.json"), fixture.envelope); + assert_eq!(member(&mut archive, "envelope.sig"), fixture.signature); + assert_eq!(member(&mut archive, "result.json"), fixture.result); + + let manifest_bytes = member(&mut archive, "manifest.json"); + let manifest: Value = serde_json::from_slice(&manifest_bytes).expect("manifest JSON"); + assert_eq!(manifest["bundleUid"], bundle_uid); + assert_eq!(manifest["nonce"], fixture.nonce); + assert_eq!(manifest["producedAt"].as_str().expect("producedAt").len(), 20); + assert_eq!(manifest["entries"].as_array().expect("entries").len(), 3); + for (index, (path, bytes)) in [ + ("envelope.json", fixture.envelope.as_slice()), + ("envelope.sig", fixture.signature.as_slice()), + ("result.json", fixture.result.as_slice()), + ] + .into_iter() + .enumerate() + { + let entry = &manifest["entries"][index]; + assert_eq!(entry["path"], path); + assert_eq!(entry["type"], "offline-diagnostic"); + assert_eq!(entry["classification"], "L3"); + assert_eq!(entry["sizeBytes"], bytes.len()); + assert_eq!(entry["sha256"], hex_lower(&Sha256::digest(bytes))); + } + + let signature: Value = serde_json::from_slice(&member(&mut archive, "manifest.sig")).expect("signature JSON"); + let mut keys = signature.as_object().expect("signature object").keys().collect::>(); + keys.sort(); + assert_eq!(keys, ["algorithm", "keyId", "value"]); + let encoded = signature["value"].as_str().expect("signature value"); + let signature = + Signature::from_slice(&URL_SAFE_NO_PAD.decode_to_vec(encoded).expect("decode signature")).expect("parse signature"); + let identity = fixture.identities.load().expect("load identity").expect("identity exists"); + let key = VerifyingKey::from_public_key_der(&identity.public_key_der()).expect("public key"); + let mut input = Vec::from(MANIFEST_DOMAIN); + input.push(0); + input.extend_from_slice(&manifest_bytes); + key.verify(&input, &signature).expect("valid manifest signature"); + } + + #[test] + fn refuses_expired_or_tampered_diagnostics() { + let expired = Fixture::new(Duration::seconds(-1)); + assert!(matches!( + upload_source(&expired.path, &Uuid::now_v7().to_string(), &expired.identities), + Err(ReportBundleError::Expired) + )); + + let tampered = Fixture::new(Duration::minutes(5)); + tamper_envelope(&tampered.path); + assert!(matches!( + upload_source(&tampered.path, &Uuid::now_v7().to_string(), &tampered.identities), + Err(ReportBundleError::Invalid) + )); + } + + struct Fixture { + _directory: tempfile::TempDir, + identities: IdentityStore, + path: std::path::PathBuf, + envelope: Vec, + signature: Vec, + result: Vec, + nonce: String, + } + + impl Fixture { + fn new(validity: Duration) -> Self { + let directory = tempfile::tempdir().expect("temporary directory"); + let identities = IdentityStore::new(directory.path().join("identity")); + let identity = identities.load_or_create().expect("device identity"); + let key_id = hex_lower(&Sha256::digest(identity.public_key_der())); + let now = OffsetDateTime::now_utc().replace_nanosecond(0).expect("whole second"); + let nonce = URL_SAFE_NO_PAD.encode_to_string([7u8; 32]); + let run_uid = Uuid::now_v7().to_string(); + let result = serde_json::to_vec(&json!({ + "schemaVersion": 1, + "runUid": run_uid, + "toolId": "top.net", + "capability": "top.net@1", + "outcome": "SUCCEEDED", + "reasonCode": "COMPLETE", + "durationMillis": 1000, + "provenance": { + "repository": "rustfs/rustfs", + "sourceCommit": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "executableSha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "rustfsVersion": "1.0.0", + "osFamily": "LINUX", + "architecture": "x86_64", + "buildFeatures": [] + }, + "coverage": {"requestedUnits": 1, "completedUnits": 1, "unit": "WINDOW"}, + "data": {"receivedBytes": 4096, "sentBytes": 4096, "windowMillis": 1000} + })) + .expect("result JSON"); + let organization = format!("organizations/{}", Uuid::now_v7()); + let cluster = format!("{organization}/clusters/{}", Uuid::now_v7()); + let device = format!("{cluster}/clusterDevices/{}", Uuid::now_v7()); + let envelope = serde_json::to_vec(&json!({ + "formatVersion": "rustfs.connect.diagnosticEnvelope/1", + "protocolVersion": "v1", + "organizationName": organization, + "clusterName": cluster, + "deviceName": device, + "runUid": run_uid, + "artifactUid": Uuid::now_v7().to_string(), + "toolId": "top.net", + "schemaVersion": 1, + "classification": "L3", + "consentUid": Uuid::now_v7().to_string(), + "policyRevision": 1, + "producedAt": now.format(&Rfc3339).expect("producedAt"), + "expiresAt": (now + validity).format(&Rfc3339).expect("expiresAt"), + "nonce": nonce, + "deviceKeyId": key_id, + "payload": { + "path": "result.json", + "mediaType": "application/json", + "sizeBytes": result.len(), + "sha256": hex_lower(&Sha256::digest(&result)) + } + })) + .expect("envelope JSON"); + let key = identity.to_pkcs8_der().expect("private key"); + let key = SigningKey::from_pkcs8_der(key.as_slice()).expect("signing key"); + let mut input = Vec::from(ENVELOPE_DOMAIN); + input.push(0); + input.extend_from_slice(&envelope); + let signature: Signature = key.sign(&input); + let signature = serde_json::to_vec(&json!({ + "algorithm": "ES256", + "keyId": key_id, + "value": URL_SAFE_NO_PAD.encode_to_string(signature.normalize_s().to_bytes()) + })) + .expect("signature JSON"); + let path = directory.path().join("diagnostic.zip"); + write_archive(&path, &envelope, &signature, &result); + Self { + _directory: directory, + identities, + path, + envelope, + signature, + result, + nonce, + } + } + } + + fn write_archive(path: &std::path::Path, envelope: &[u8], signature: &[u8], result: &[u8]) { + let file = std::fs::File::create(path).expect("create archive"); + let options = SimpleFileOptions::DEFAULT + .compression_method(CompressionMethod::Stored) + .system(zip::System::Unix) + .unix_permissions(0o600); + let mut archive = ZipWriter::new(file); + for (name, bytes) in [ + ("envelope.json", envelope), + ("envelope.sig", signature), + ("result.json", result), + ] { + archive.start_file(name, options).expect("start member"); + archive.write_all(bytes).expect("write member"); + } + archive.finish().expect("finish archive"); + } + + fn member(archive: &mut ZipArchive, name: &str) -> Vec { + let mut member = archive.by_name(name).expect("archive member"); + let mut bytes = Vec::new(); + member.read_to_end(&mut bytes).expect("read member"); + bytes + } + + fn tamper_envelope(path: &std::path::Path) { + let file = std::fs::File::open(path).expect("open archive"); + let mut archive = ZipArchive::new(file).expect("read archive"); + let mut envelope = member(&mut archive, "envelope.json"); + let signature = member(&mut archive, "envelope.sig"); + let result = member(&mut archive, "result.json"); + drop(archive); + envelope.push(b' '); + write_archive(path, &envelope, &signature, &result); + } +} diff --git a/rustfs/src/connect/report_upload.rs b/rustfs/src/connect/report_upload.rs new file mode 100644 index 000000000..a68d4fe42 --- /dev/null +++ b/rustfs/src/connect/report_upload.rs @@ -0,0 +1,713 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Device-authenticated upload of bounded support bundle archives. + +use std::collections::{HashMap, HashSet}; +use std::io::SeekFrom; +use std::path::Path; +use std::time::Duration; + +use chrono::{DateTime, Utc}; +use reqwest::header::{self, HeaderMap, HeaderName, HeaderValue}; +use reqwest::{Client, StatusCode, Url}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest as _, Sha256}; +use tokio::fs::File; +use tokio::io::{AsyncReadExt as _, AsyncSeekExt as _}; +use tokio_util::io::ReaderStream; +use tokio_util::sync::CancellationToken; +use uuid::Uuid; + +use super::client::{TransportFailure, classify_transport_failure}; +use super::config::HeartbeatConfig; +use super::telemetry::{TelemetryDelivery, TelemetryError, TelemetryTransport, is_exact_utc_seconds}; + +const PROTOCOL_VERSION: &str = "v1"; +const CONTENT_TYPE: &str = "application/octet-stream"; +const MAX_ATTEMPTS: usize = 3; +const MAX_UPLOAD_TIMEOUT: Duration = Duration::from_secs(15 * 60); +const UPLOAD_BUFFER_BYTES: usize = 64 * 1024; + +/// Maximum archive size accepted by the Connect agent API. +pub const MAX_SUPPORT_BUNDLE_BYTES: u64 = 256 * 1024 * 1024; + +/// Verified outcome returned after Connect pins the uploaded object version. +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ReportUploadReceipt { + pub name: String, + pub uid: String, + pub state: String, + pub declared_size_bytes: u64, + pub declared_sha256: String, +} + +/// Uploads one local archive through the registered device identity. +pub struct ReportUploadClient { + transport: TelemetryTransport, + upload_client: Client, + identity_store: super::IdentityStore, + initial_backoff: Duration, + max_backoff: Duration, + proxy_configured: bool, +} + +impl ReportUploadClient { + /// Reuses the Connect mTLS identity, root bundle, explicit proxy, and + /// disabled redirect/environment-proxy policy. + pub fn new(config: HeartbeatConfig, upload_timeout: Duration) -> Result { + if upload_timeout.is_zero() || upload_timeout > MAX_UPLOAD_TIMEOUT { + return Err(ReportUploadError::UploadTimeout); + } + let initial_backoff = config.schedule.initial_backoff; + let max_backoff = config.schedule.max_backoff; + let proxy_configured = config.proxy.is_some(); + let identity_store = config.identity_store.clone(); + let transport = TelemetryTransport::new(config).map_err(transport_error)?; + let upload_client = transport.presigned_client(upload_timeout).map_err(transport_error)?; + Ok(Self { + transport, + upload_client, + identity_store, + initial_backoff, + max_backoff, + proxy_configured, + }) + } + + /// Wraps a typed diagnostic in its signed upload manifest when needed, + /// hashes the exact upload bytes, reserves one object, and completes the + /// reservation only after the object store accepts the archive. + pub async fn upload( + &self, + archive: &Path, + cancellation: &CancellationToken, + ) -> Result { + if cancellation.is_cancelled() { + return Err(ReportUploadError::Cancelled); + } + let generated_bundle_uid = Uuid::now_v7().to_string(); + let source = super::report_bundle::upload_source(archive, &generated_bundle_uid, &self.identity_store) + .map_err(report_bundle_error)?; + let prepared = prepare_file(File::from_std(source.file), cancellation).await?; + let request_id = Uuid::new_v4().to_string(); + let bundle_uid = source.bundle_uid; + let reserve = ReserveRequest { + protocol_version: PROTOCOL_VERSION, + request_id: &request_id, + bundle_uid: &bundle_uid, + content_type: CONTENT_TYPE, + declared_size_bytes: prepared.size, + declared_sha256: &prepared.sha256, + }; + + let mut backoff = self.initial_backoff; + let mut expected_name = None; + for attempt in 0..MAX_ATTEMPTS { + let (cluster_name, body) = self + .post_control("supportBundles", &reserve, StatusCode::CREATED, cancellation) + .await?; + let name = format!("{cluster_name}/supportBundles/{bundle_uid}"); + if expected_name.as_ref().is_some_and(|expected| expected != &name) { + return Err(ReportUploadError::Response); + } + expected_name = Some(name.clone()); + let reservation = decode_reservation(&body, &name, &bundle_uid, &prepared)?; + + match self.put(&prepared.file, &reservation.authorization, cancellation).await? { + UploadDelivery::Accepted | UploadDelivery::AlreadyPresent => break, + UploadDelivery::Retry { retry_after } if attempt + 1 < MAX_ATTEMPTS => { + let delay = retry_after.unwrap_or(backoff).clamp(self.initial_backoff, self.max_backoff); + sleep_or_cancel(cancellation, delay).await?; + backoff = backoff.saturating_mul(2).min(self.max_backoff); + } + UploadDelivery::Retry { .. } => return Err(ReportUploadError::RetryExhausted), + } + } + + let complete_request_id = Uuid::new_v4().to_string(); + let complete = CompleteRequest { + protocol_version: PROTOCOL_VERSION, + request_id: &complete_request_id, + }; + let path = format!("supportBundles/{bundle_uid}:completeUpload"); + let (cluster_name, body) = self.post_control(&path, &complete, StatusCode::OK, cancellation).await?; + let expected_name = expected_name.ok_or(ReportUploadError::Response)?; + if expected_name != format!("{cluster_name}/supportBundles/{bundle_uid}") { + return Err(ReportUploadError::Response); + } + decode_receipt(&body, &expected_name, &bundle_uid, &prepared) + } + + async fn post_control( + &self, + operation: &str, + body: &T, + expected_status: StatusCode, + cancellation: &CancellationToken, + ) -> Result<(String, Vec), ReportUploadError> { + let mut backoff = self.initial_backoff; + for attempt in 0..MAX_ATTEMPTS { + let delivery = tokio::select! { + biased; + () = cancellation.cancelled() => return Err(ReportUploadError::Cancelled), + result = self.transport.post_expect(operation, body, expected_status) => result.map_err(transport_error)?, + }; + match delivery { + TelemetryDelivery::Accepted { cluster_name, body } => return Ok((cluster_name, body)), + TelemetryDelivery::Retry { retry_after } if attempt + 1 < MAX_ATTEMPTS => { + let delay = retry_after.unwrap_or(backoff).clamp(self.initial_backoff, self.max_backoff); + sleep_or_cancel(cancellation, delay).await?; + backoff = backoff.saturating_mul(2).min(self.max_backoff); + } + TelemetryDelivery::Retry { .. } => return Err(ReportUploadError::RetryExhausted), + TelemetryDelivery::AuthenticationStopped { status, .. } => { + return Err(ReportUploadError::AuthenticationStopped { status }); + } + TelemetryDelivery::Rejected { status, .. } => { + return Err(ReportUploadError::ControlRejected { status }); + } + } + } + Err(ReportUploadError::RetryExhausted) + } + + async fn put( + &self, + file: &File, + authorization: &UploadAuthorization, + cancellation: &CancellationToken, + ) -> Result { + let mut source = file.try_clone().await.map_err(ReportUploadError::ArchiveRead)?; + source + .seek(SeekFrom::Start(0)) + .await + .map_err(ReportUploadError::ArchiveRead)?; + let body = reqwest::Body::wrap_stream(ReaderStream::with_capacity(source, UPLOAD_BUFFER_BYTES)); + let response = tokio::select! { + biased; + () = cancellation.cancelled() => return Err(ReportUploadError::Cancelled), + result = self.upload_client.put(authorization.url.clone()).headers(authorization.headers.clone()).body(body).send() => result, + }; + let response = match response { + Ok(response) => response, + Err(error) if error.is_body() => return Ok(UploadDelivery::Retry { retry_after: None }), + Err(error) if error.is_timeout() || error.is_connect() || error.is_request() => { + if let Some(failure) = classify_transport_failure(&error, self.proxy_configured) { + return Err(match failure { + TransportFailure::ProxyAuthentication => ReportUploadError::ProxyAuthentication, + TransportFailure::ProxyRejected => ReportUploadError::ProxyRejected, + TransportFailure::TlsPeer => ReportUploadError::TlsPeer, + }); + } + return Ok(UploadDelivery::Retry { retry_after: None }); + } + Err(_) => return Err(ReportUploadError::UploadTransport), + }; + let status = response.status(); + if status.is_success() { + return Ok(UploadDelivery::Accepted); + } + if status == StatusCode::PRECONDITION_FAILED { + // A lost successful PUT response can make the create-only replay + // fail. Connect still verifies size and digest before completion. + return Ok(UploadDelivery::AlreadyPresent); + } + if status == StatusCode::TOO_MANY_REQUESTS || status == StatusCode::REQUEST_TIMEOUT || status.is_server_error() { + return Ok(UploadDelivery::Retry { + retry_after: retry_after(response.headers(), Utc::now(), self.max_backoff), + }); + } + Err(ReportUploadError::UploadRejected { status: status.as_u16() }) + } +} + +struct PreparedArchive { + file: File, + size: u64, + sha256: String, + checksum_base64: String, +} + +#[cfg(test)] +async fn prepare_archive(path: &Path, cancellation: &CancellationToken) -> Result { + let file = File::open(path).await.map_err(ReportUploadError::ArchiveOpen)?; + prepare_file(file, cancellation).await +} + +async fn prepare_file(mut file: File, cancellation: &CancellationToken) -> Result { + let metadata = file.metadata().await.map_err(ReportUploadError::ArchiveRead)?; + if !metadata.is_file() { + return Err(ReportUploadError::ArchiveType); + } + if metadata.len() == 0 || metadata.len() > MAX_SUPPORT_BUNDLE_BYTES { + return Err(ReportUploadError::ArchiveSize); + } + + let mut hasher = Sha256::new(); + let mut size = 0u64; + let mut buffer = vec![0u8; UPLOAD_BUFFER_BYTES]; + loop { + let read = tokio::select! { + biased; + () = cancellation.cancelled() => return Err(ReportUploadError::Cancelled), + result = file.read(&mut buffer) => result.map_err(ReportUploadError::ArchiveRead)?, + }; + if read == 0 { + break; + } + size = size.checked_add(read as u64).ok_or(ReportUploadError::ArchiveSize)?; + if size > MAX_SUPPORT_BUNDLE_BYTES { + return Err(ReportUploadError::ArchiveSize); + } + hasher.update(&buffer[..read]); + } + if size != metadata.len() { + return Err(ReportUploadError::ArchiveChanged); + } + let current = file.metadata().await.map_err(ReportUploadError::ArchiveRead)?; + if current.len() != size { + return Err(ReportUploadError::ArchiveChanged); + } + let digest = hasher.finalize(); + Ok(PreparedArchive { + file, + size, + sha256: faster_hex::hex_string(&digest), + checksum_base64: base64_simd::STANDARD.encode_to_string(digest), + }) +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct ReserveRequest<'a> { + protocol_version: &'static str, + request_id: &'a str, + bundle_uid: &'a str, + content_type: &'static str, + declared_size_bytes: u64, + declared_sha256: &'a str, +} + +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +struct CompleteRequest<'a> { + protocol_version: &'static str, + request_id: &'a str, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct ReservationResponse { + support_bundle: BundleResource, + upload_authorization: RawUploadAuthorization, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct RawUploadAuthorization { + method: String, + url: String, + headers: HashMap, + expire_time: String, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase")] +struct BundleResource { + name: String, + uid: String, + state: String, + declared_size_bytes: u64, + declared_sha256: String, + expire_time: String, + create_time: String, + update_time: String, +} + +struct Reservation { + authorization: UploadAuthorization, +} + +struct UploadAuthorization { + url: Url, + headers: HeaderMap, +} + +enum UploadDelivery { + Accepted, + AlreadyPresent, + Retry { retry_after: Option }, +} + +fn decode_reservation( + body: &[u8], + expected_name: &str, + expected_uid: &str, + archive: &PreparedArchive, +) -> Result { + let response: ReservationResponse = serde_json::from_slice(body).map_err(|_| ReportUploadError::Response)?; + validate_resource(&response.support_bundle, "PENDING", expected_name, expected_uid, archive)?; + if response.upload_authorization.method != "PUT" || !is_short_lived_future_instant(&response.upload_authorization.expire_time) + { + return Err(ReportUploadError::UploadAuthorization); + } + let url = Url::parse(&response.upload_authorization.url).map_err(|_| ReportUploadError::UploadAuthorization)?; + if url.scheme() != "https" + || url.host_str().is_none() + || url.cannot_be_a_base() + || !url.username().is_empty() + || url.password().is_some() + || url.fragment().is_some() + { + return Err(ReportUploadError::UploadAuthorization); + } + let headers = validate_headers(response.upload_authorization.headers, archive)?; + Ok(Reservation { + authorization: UploadAuthorization { url, headers }, + }) +} + +fn decode_receipt( + body: &[u8], + expected_name: &str, + expected_uid: &str, + archive: &PreparedArchive, +) -> Result { + let response: BundleResource = serde_json::from_slice(body).map_err(|_| ReportUploadError::Response)?; + validate_resource(&response, "UPLOADED", expected_name, expected_uid, archive)?; + Ok(ReportUploadReceipt { + name: response.name, + uid: response.uid, + state: response.state, + declared_size_bytes: response.declared_size_bytes, + declared_sha256: response.declared_sha256, + }) +} + +fn validate_resource( + resource: &BundleResource, + expected_state: &str, + expected_name: &str, + expected_uid: &str, + archive: &PreparedArchive, +) -> Result<(), ReportUploadError> { + if resource.name != expected_name + || resource.uid != expected_uid + || resource.state != expected_state + || resource.declared_size_bytes != archive.size + || resource.declared_sha256 != archive.sha256 + || !is_exact_utc_seconds(&resource.expire_time) + || !is_exact_utc_seconds(&resource.create_time) + || !is_exact_utc_seconds(&resource.update_time) + { + return Err(ReportUploadError::Response); + } + Ok(()) +} + +fn validate_headers(raw: HashMap, archive: &PreparedArchive) -> Result { + let mut headers = HeaderMap::new(); + let mut names = HashSet::new(); + for (name, value) in raw { + let normalized = name.to_ascii_lowercase(); + if !names.insert(normalized.clone()) || forbidden_header(&normalized) { + return Err(ReportUploadError::UploadAuthorization); + } + let name = HeaderName::from_bytes(name.as_bytes()).map_err(|_| ReportUploadError::UploadAuthorization)?; + let value = HeaderValue::from_str(&value).map_err(|_| ReportUploadError::UploadAuthorization)?; + headers.insert(name, value); + } + let expected_length = archive.size.to_string(); + let encryption_authorized = match headers + .get("x-amz-server-side-encryption") + .and_then(|value| value.to_str().ok()) + { + Some("AES256") => true, + Some("aws:kms") => headers + .get("x-amz-server-side-encryption-aws-kms-key-id") + .and_then(|value| value.to_str().ok()) + .is_some_and(|value| !value.trim().is_empty()), + _ => false, + }; + if headers.get(header::CONTENT_TYPE).and_then(|value| value.to_str().ok()) != Some(CONTENT_TYPE) + || headers.get(header::CONTENT_LENGTH).and_then(|value| value.to_str().ok()) != Some(expected_length.as_str()) + || headers.get(header::IF_NONE_MATCH).and_then(|value| value.to_str().ok()) != Some("*") + || headers.get("x-amz-checksum-sha256").and_then(|value| value.to_str().ok()) != Some(archive.checksum_base64.as_str()) + || !encryption_authorized + { + return Err(ReportUploadError::UploadAuthorization); + } + Ok(headers) +} + +fn forbidden_header(name: &str) -> bool { + matches!( + name, + "authorization" + | "proxy-authorization" + | "cookie" + | "set-cookie" + | "host" + | "connection" + | "transfer-encoding" + | "upgrade" + | "te" + | "trailer" + ) +} + +fn is_short_lived_future_instant(value: &str) -> bool { + if !is_exact_utc_seconds(value) { + return false; + } + let now = Utc::now(); + DateTime::parse_from_rfc3339(value).is_ok_and(|instant| { + let instant = instant.with_timezone(&Utc); + instant > now && (instant - now).to_std().is_ok_and(|duration| duration <= MAX_UPLOAD_TIMEOUT) + }) +} + +fn retry_after(headers: &HeaderMap, now: DateTime, maximum: Duration) -> Option { + let value = headers.get(header::RETRY_AFTER)?.to_str().ok()?; + let delay = value.parse::().ok().map(Duration::from_secs).or_else(|| { + DateTime::parse_from_rfc2822(value) + .ok() + .and_then(|at| (at.with_timezone(&Utc) - now).to_std().ok()) + })?; + Some(delay.min(maximum)) +} + +async fn sleep_or_cancel(cancellation: &CancellationToken, delay: Duration) -> Result<(), ReportUploadError> { + tokio::select! { + biased; + () = cancellation.cancelled() => Err(ReportUploadError::Cancelled), + () = tokio::time::sleep(delay) => Ok(()), + } +} + +fn transport_error(error: TelemetryError) -> ReportUploadError { + match error { + TelemetryError::Endpoint => ReportUploadError::Endpoint, + TelemetryError::RootCertificate => ReportUploadError::RootCertificate, + TelemetryError::ProxyConfiguration => ReportUploadError::ProxyConfiguration, + TelemetryError::ProxyAuthentication => ReportUploadError::ProxyAuthentication, + TelemetryError::ProxyRejected => ReportUploadError::ProxyRejected, + TelemetryError::TlsPeer => ReportUploadError::TlsPeer, + TelemetryError::Schedule => ReportUploadError::Schedule, + TelemetryError::NotRegistered => ReportUploadError::NotRegistered, + TelemetryError::IdentityMissing => ReportUploadError::IdentityMissing, + TelemetryError::IdentityCertificate => ReportUploadError::IdentityCertificate, + TelemetryError::CredentialName => ReportUploadError::CredentialName, + TelemetryError::CredentialExpired => ReportUploadError::CredentialExpired, + TelemetryError::StateConflict => ReportUploadError::CredentialState, + TelemetryError::ResponseTooLarge => ReportUploadError::ResponseTooLarge, + TelemetryError::Url(_) + | TelemetryError::Transport(_) + | TelemetryError::Identity(_) + | TelemetryError::IdentityStore(_) + | TelemetryError::CredentialStore(_) + | TelemetryError::CredentialValidation(_) => ReportUploadError::UploadTransport, + } +} + +fn report_bundle_error(error: super::report_bundle::ReportBundleError) -> ReportUploadError { + use super::report_bundle::ReportBundleError; + + match error { + ReportBundleError::Expired => ReportUploadError::DiagnosticExpired, + ReportBundleError::IdentityMissing => ReportUploadError::IdentityMissing, + ReportBundleError::Io(error) => ReportUploadError::ArchiveRead(error), + ReportBundleError::Invalid | ReportBundleError::Zip(_) => ReportUploadError::DiagnosticArchive, + } +} + +/// Safe, credential-redacted report upload failures. +#[derive(Debug, thiserror::Error)] +pub enum ReportUploadError { + #[error("Connect report upload timeout must be from one second through fifteen minutes")] + UploadTimeout, + #[error("Connect report upload endpoint must be an HTTPS base URL without credentials, query, or fragment")] + Endpoint, + #[error("Connect report upload root CA configuration is invalid")] + RootCertificate, + #[error("Connect report upload proxy configuration is invalid")] + ProxyConfiguration, + #[error("Connect proxy authentication failed; verify the configured proxy credential files")] + ProxyAuthentication, + #[error("Connect proxy connection failed; verify proxy availability, credentials, and the approved targets")] + ProxyRejected, + #[error("Connect TLS peer certificate validation failed; verify the endpoint and configured root CA")] + TlsPeer, + #[error("Connect report upload retry schedule is invalid")] + Schedule, + #[error("RustFS is not registered with Connect")] + NotRegistered, + #[error("the Connect device private key is missing")] + IdentityMissing, + #[error("the stored Connect certificate and device private key cannot form a TLS identity")] + IdentityCertificate, + #[error("the stored Connect credential name is invalid")] + CredentialName, + #[error("the stored Connect device certificate is not currently valid")] + CredentialExpired, + #[error("the persisted Connect credential transition is invalid")] + CredentialState, + #[error("Connect report upload response exceeded 64 KiB")] + ResponseTooLarge, + #[error("the report archive could not be opened")] + ArchiveOpen(#[source] std::io::Error), + #[error("the report archive could not be read")] + ArchiveRead(#[source] std::io::Error), + #[error("the report archive must be a regular file")] + ArchiveType, + #[error("the report archive must contain 1 byte through 256 MiB")] + ArchiveSize, + #[error("the report archive changed while its digest was calculated")] + ArchiveChanged, + #[error("the signed diagnostic archive is invalid")] + DiagnosticArchive, + #[error("the signed diagnostic archive has expired")] + DiagnosticExpired, + #[error("Connect returned an invalid report upload response")] + Response, + #[error("Connect returned an invalid or expired report upload authorization")] + UploadAuthorization, + #[error("Connect authentication stopped report upload with HTTP {status}")] + AuthenticationStopped { status: u16 }, + #[error("Connect rejected report upload control request with HTTP {status}")] + ControlRejected { status: u16 }, + #[error("the report object upload was rejected with HTTP {status}")] + UploadRejected { status: u16 }, + #[error("the report object upload transport failed")] + UploadTransport, + #[error("Connect report upload exhausted its bounded retries")] + RetryExhausted, + #[error("Connect report upload was cancelled")] + Cancelled, +} + +#[cfg(test)] +mod tests { + use chrono::SecondsFormat; + use serde_json::json; + + use super::*; + + #[tokio::test] + async fn prepares_the_exact_bounded_archive() { + let temp = tempfile::tempdir().expect("tempdir"); + let path = temp.path().join("bundle.tar.zst"); + tokio::fs::write(&path, b"redacted support bundle") + .await + .expect("write bundle"); + + let archive = prepare_archive(&path, &CancellationToken::new()) + .await + .expect("prepare archive"); + + assert_eq!(archive.size, 23); + let digest = Sha256::digest(b"redacted support bundle"); + assert_eq!(archive.sha256, faster_hex::hex_string(&digest)); + assert_eq!(archive.checksum_base64, base64_simd::STANDARD.encode_to_string(digest)); + } + + #[tokio::test] + async fn cancellation_stops_archive_preparation() { + let temp = tempfile::tempdir().expect("tempdir"); + let path = temp.path().join("bundle.tar.zst"); + tokio::fs::write(&path, b"bundle").await.expect("write bundle"); + let cancellation = CancellationToken::new(); + cancellation.cancel(); + + assert!(matches!(prepare_archive(&path, &cancellation).await, Err(ReportUploadError::Cancelled))); + } + + #[tokio::test] + async fn reservation_is_bound_to_archive_and_safe_headers() { + let temp = tempfile::tempdir().expect("tempdir"); + let path = temp.path().join("bundle.tar.zst"); + tokio::fs::write(&path, b"bundle").await.expect("write bundle"); + let archive = prepare_archive(&path, &CancellationToken::new()) + .await + .expect("prepare archive"); + let bundle_uid = Uuid::now_v7().to_string(); + let name = format!("organizations/o/clusters/c/supportBundles/{bundle_uid}"); + let now = Utc::now(); + let instant = |minutes| (now + chrono::Duration::minutes(minutes)).to_rfc3339_opts(SecondsFormat::Secs, true); + let response = json!({ + "supportBundle": { + "name": &name, + "uid": &bundle_uid, + "state": "PENDING", + "declaredSizeBytes": archive.size, + "declaredSha256": &archive.sha256, + "expireTime": instant(60), + "createTime": now.to_rfc3339_opts(SecondsFormat::Secs, true), + "updateTime": now.to_rfc3339_opts(SecondsFormat::Secs, true) + }, + "uploadAuthorization": { + "method": "PUT", + "url": "https://objects.example.test/upload?signature=hidden", + "headers": { + "Content-Type": CONTENT_TYPE, + "Content-Length": archive.size.to_string(), + "If-None-Match": "*", + "x-amz-checksum-sha256": &archive.checksum_base64, + "x-amz-server-side-encryption": "AES256" + }, + "expireTime": instant(5) + } + }); + + decode_reservation(&serde_json::to_vec(&response).expect("response JSON"), &name, &bundle_uid, &archive) + .expect("valid reservation"); + + let mut kms_response = response.clone(); + kms_response["uploadAuthorization"]["headers"]["x-amz-server-side-encryption"] = json!("aws:kms"); + kms_response["uploadAuthorization"]["headers"]["x-amz-server-side-encryption-aws-kms-key-id"] = + json!("connect-support-bundles"); + decode_reservation(&serde_json::to_vec(&kms_response).expect("response JSON"), &name, &bundle_uid, &archive) + .expect("valid KMS reservation"); + + kms_response["uploadAuthorization"]["headers"] + .as_object_mut() + .expect("headers object") + .remove("x-amz-server-side-encryption-aws-kms-key-id"); + assert!(matches!( + decode_reservation(&serde_json::to_vec(&kms_response).expect("response JSON"), &name, &bundle_uid, &archive,), + Err(ReportUploadError::UploadAuthorization) + )); + + let mut wrong_target = response.clone(); + wrong_target["supportBundle"]["name"] = json!(format!("organizations/o/clusters/other/supportBundles/{bundle_uid}")); + assert!(matches!( + decode_reservation(&serde_json::to_vec(&wrong_target).expect("response JSON"), &name, &bundle_uid, &archive,), + Err(ReportUploadError::Response) + )); + + let mut unsafe_response = response; + unsafe_response["uploadAuthorization"]["headers"]["Authorization"] = json!("secret"); + assert!(matches!( + decode_reservation( + &serde_json::to_vec(&unsafe_response).expect("response JSON"), + &name, + &bundle_uid, + &archive, + ), + Err(ReportUploadError::UploadAuthorization) + )); + } +} diff --git a/rustfs/src/connect/runtime.rs b/rustfs/src/connect/runtime.rs index a866e7684..60d775eff 100644 --- a/rustfs/src/connect/runtime.rs +++ b/rustfs/src/connect/runtime.rs @@ -25,6 +25,11 @@ use tokio_util::sync::CancellationToken; use super::client::{ClientError, ConnectClient, ConnectConfig, RotationAttempt}; use super::config::HeartbeatConfig; +use super::diagnostics::job_delivery::DiagnosticJobRuntime; +use super::diagnostics::{ + DiagnosticCollectionPolicy, DiagnosticReceipt, DiagnosticReceiptDelivery, DiagnosticReceiptSender, DiagnosticScheduleRuntime, + DiagnosticScheduleStatus, spawn_environment_schedule, +}; use super::heartbeat::{CoarseNodeSummary, Delivery, HeartbeatError, HeartbeatSender, HeartbeatStateStore, HeartbeatStatus}; use super::inventory::{ InventoryDelivery, InventoryError, InventorySchedule, InventorySender, InventorySnapshot, InventoryStateStore, @@ -35,6 +40,9 @@ pub struct HeartbeatRuntime { shutdown: CancellationToken, status: watch::Receiver, task: Option>, + diagnostic_status: watch::Receiver, + diagnostic_task: Option, + diagnostic_receipt_task: Option>, } impl HeartbeatRuntime { @@ -42,11 +50,21 @@ impl HeartbeatRuntime { self.status.clone() } + pub fn diagnostic_status(&self) -> watch::Receiver { + self.diagnostic_status.clone() + } + pub async fn shutdown(mut self) { self.shutdown.cancel(); if let Some(task) = self.task.take() { let _ = task.await; } + if let Some(task) = self.diagnostic_task.take() { + task.shutdown().await; + } + if let Some(task) = self.diagnostic_receipt_task.take() { + let _ = task.await; + } } } @@ -111,20 +129,41 @@ where return Ok(None); } let sender = HeartbeatSender::new(config.clone())?; + let receipt_sender = DiagnosticReceiptSender::new(config.clone())?; let rotation = ConnectClient::new(ConnectConfig { endpoint: &config.endpoint, root_ca_pem: &config.root_ca_pem, timeout: config.schedule.timeout, + proxy: config.proxy.as_ref(), }) .map_err(rotation_failure)?; let identity_store = config.identity_store.clone(); let credential_store = config.credential_store.clone(); - let store = HeartbeatStateStore::new(config.state_path.clone()); + let diagnostic_job_runtime = DiagnosticJobRuntime::from_config(&config); + let store = HeartbeatStateStore::new(config.state_path.clone(), diagnostic_job_runtime.is_some()); let lock = store.try_runtime_lock()?; let schedule = config.schedule; + let state_root = config.state_root().ok_or(HeartbeatError::StateConflict)?.to_path_buf(); let shutdown = parent_shutdown.child_token(); let task_shutdown = shutdown.clone(); let (status_tx, status_rx) = watch::channel(HeartbeatStatus::Starting); + let (policy_tx, policy_rx) = watch::channel(DiagnosticCollectionPolicy::stopped()); + let diagnostic_task = + spawn_environment_schedule(&state_root, policy_rx, shutdown.clone()).map_err(|_| HeartbeatError::StateConflict)?; + let diagnostic_status = diagnostic_task.status(); + let receipt_status = diagnostic_task.receipts(); + let receipt_shutdown = shutdown.clone(); + let retry_schedule = schedule; + let diagnostic_receipt_task = tokio::spawn(async move { + run_diagnostic_receipt_delivery( + receipt_sender, + receipt_status, + retry_schedule.initial_backoff, + retry_schedule.max_backoff, + receipt_shutdown, + ) + .await; + }); let task = tokio::spawn(async move { let _lock = lock; let mut backoff = schedule.initial_backoff; @@ -178,11 +217,19 @@ where None => break, }; let delay = match delivery { - Delivery::Accepted { server_time } => { + Delivery::Accepted { + server_time, + diagnostic_collection_policy, + diagnostic_job, + } => { if let Err(error) = store.mark_accepted(&pending).await { return failed(&status_tx, error); } backoff = schedule.initial_backoff; + let _ = policy_tx.send(diagnostic_collection_policy); + if let (Some(runtime), Some(job)) = (&diagnostic_job_runtime, diagnostic_job) { + runtime.offer(*job, &task_shutdown); + } let _ = status_tx.send(HeartbeatStatus::Online { server_time }); schedule.cadence.saturating_add(jitter(schedule.jitter)) } @@ -216,9 +263,60 @@ where shutdown, status: status_rx, task: Some(task), + diagnostic_status, + diagnostic_task: Some(diagnostic_task), + diagnostic_receipt_task: Some(diagnostic_receipt_task), })) } +async fn run_diagnostic_receipt_delivery( + sender: DiagnosticReceiptSender, + mut receipts: watch::Receiver>, + initial_backoff: Duration, + max_backoff: Duration, + shutdown: CancellationToken, +) { + let mut last_accepted = None; + let mut backoff = initial_backoff; + loop { + let Some(receipt) = receipts.borrow_and_update().clone() else { + tokio::select! { + biased; + () = shutdown.cancelled() => break, + changed = receipts.changed() => if changed.is_err() { break; }, + } + continue; + }; + if last_accepted.as_deref() == Some(receipt.receipt_id.as_str()) { + tokio::select! { + biased; + () = shutdown.cancelled() => break, + changed = receipts.changed() => if changed.is_err() { break; }, + } + continue; + } + let delivery = tokio::select! { + biased; + () = shutdown.cancelled() => break, + delivery = sender.send(&receipt) => delivery, + }; + match delivery { + Ok(DiagnosticReceiptDelivery::Accepted) => { + last_accepted = Some(receipt.receipt_id.clone()); + backoff = initial_backoff; + } + Ok(DiagnosticReceiptDelivery::Retry { retry_after }) => { + let delay = retry_after.unwrap_or(backoff).clamp(initial_backoff, max_backoff); + backoff = backoff.saturating_mul(2).min(max_backoff); + if sleep_or_cancel(&shutdown, delay).await { + break; + } + } + Ok(DiagnosticReceiptDelivery::AuthenticationStopped | DiagnosticReceiptDelivery::Rejected) | Err(_) => break, + } + } +} + pub fn spawn_inventory_runtime( config: Option, schedule: InventorySchedule, @@ -394,6 +492,10 @@ fn rotation_failure(error: ClientError) -> HeartbeatError { match error { ClientError::Endpoint => HeartbeatError::Endpoint, ClientError::RootCertificate => HeartbeatError::RootCertificate, + ClientError::ProxyConfiguration(_) => HeartbeatError::ProxyConfiguration, + ClientError::ProxyAuthentication => HeartbeatError::ProxyAuthentication, + ClientError::ProxyRejected => HeartbeatError::ProxyRejected, + ClientError::TlsPeer => HeartbeatError::TlsPeer, ClientError::NotRegistered => HeartbeatError::NotRegistered, ClientError::IdentityMissing => HeartbeatError::IdentityMissing, ClientError::CredentialExpired | ClientError::CredentialNotYetValid => HeartbeatError::CredentialExpired, @@ -418,6 +520,10 @@ pub(crate) fn heartbeat_failure_reason(error: &HeartbeatError) -> &'static str { match error { HeartbeatError::Endpoint => "connect_heartbeat_endpoint", HeartbeatError::RootCertificate => "connect_heartbeat_root_certificate", + HeartbeatError::ProxyConfiguration => "connect_heartbeat_proxy_configuration", + HeartbeatError::ProxyAuthentication => "connect_heartbeat_proxy_authentication", + HeartbeatError::ProxyRejected => "connect_heartbeat_proxy_rejected", + HeartbeatError::TlsPeer => "connect_heartbeat_tls_peer", HeartbeatError::Schedule => "connect_heartbeat_schedule", HeartbeatError::NotRegistered => "connect_heartbeat_not_registered", HeartbeatError::IdentityMissing => "connect_heartbeat_identity_missing", @@ -547,6 +653,7 @@ mod tests { let (release_heartbeat, wait_for_release) = tokio::sync::oneshot::channel(); let (inventory_stopped, stopped) = tokio::sync::oneshot::channel(); let (_, heartbeat_status) = watch::channel(HeartbeatStatus::Starting); + let (_, diagnostic_status) = watch::channel(DiagnosticScheduleStatus::Waiting); let (_, inventory_status) = watch::channel(InventoryStatus::Starting); let heartbeat_task = tokio::spawn(async move { let _ = wait_for_release.await; @@ -560,6 +667,9 @@ mod tests { shutdown: heartbeat_shutdown, status: heartbeat_status, task: Some(heartbeat_task), + diagnostic_status, + diagnostic_task: None, + diagnostic_receipt_task: None, }; let inventory = InventoryRuntime { shutdown: inventory_shutdown, diff --git a/rustfs/src/connect/telemetry.rs b/rustfs/src/connect/telemetry.rs index 3baef8cc2..828f07592 100644 --- a/rustfs/src/connect/telemetry.rs +++ b/rustfs/src/connect/telemetry.rs @@ -21,7 +21,7 @@ use rustls::pki_types::{CertificateDer, pem::PemObject as _}; use serde::{Deserialize, Serialize}; use zeroize::Zeroizing; -use super::client::{ClientError, ConnectClient}; +use super::client::{ClientError, ConnectClient, TransportFailure, build_client, classify_transport_failure}; use super::config::HeartbeatConfig; use super::credential_store::{CredentialStoreError, DeviceCredential}; use super::identity::IdentityError; @@ -95,6 +95,15 @@ impl TelemetryTransport { } pub(crate) async fn post(&self, collection: &str, value: &T) -> Result { + self.post_expect(collection, value, StatusCode::OK).await + } + + pub(crate) async fn post_expect( + &self, + collection: &str, + value: &T, + expected_status: StatusCode, + ) -> Result { let mut authenticated = self.authenticated_client().await?; let mut refreshed = false; loop { @@ -104,6 +113,13 @@ impl TelemetryTransport { let response = match authenticated.client.post(url).json(value).send().await { Ok(response) => response, Err(error) if error.is_timeout() || error.is_connect() || error.is_request() => { + if let Some(failure) = classify_transport_failure(&error, self.config.proxy.is_some()) { + return Err(match failure { + TransportFailure::ProxyAuthentication => TelemetryError::ProxyAuthentication, + TransportFailure::ProxyRejected => TelemetryError::ProxyRejected, + TransportFailure::TlsPeer => TelemetryError::TlsPeer, + }); + } return Ok(TelemetryDelivery::Retry { retry_after: None }); } Err(error) => return Err(error.into()), @@ -131,7 +147,7 @@ impl TelemetryTransport { reason: response_reason(response).await, }); } - if status != StatusCode::OK { + if status != expected_status { return Ok(TelemetryDelivery::Rejected { status: status.as_u16(), reason: response_reason(response).await, @@ -144,6 +160,10 @@ impl TelemetryTransport { } } + pub(crate) fn presigned_client(&self, timeout: Duration) -> Result { + build_client(&self.roots, timeout, None, self.config.proxy.as_ref()).map_err(credential_recovery_error) + } + async fn authenticated_client(&self) -> Result { let lock = self.config.credential_store.lock().await?; let (credential, identity, _) = ConnectClient::recover_valid_credential_locked( @@ -177,19 +197,8 @@ impl TelemetryTransport { pem.push(b'\n'); pem.extend_from_slice(key.as_bytes()); let identity = reqwest::Identity::from_pem(&pem).map_err(|_| TelemetryError::IdentityCertificate)?; - let roots = self - .roots - .iter() - .map(|root| reqwest::Certificate::from_der(root.as_ref())) - .collect::, _>>()?; - Client::builder() - .https_only(true) - .redirect(reqwest::redirect::Policy::none()) - .timeout(self.config.schedule.timeout) - .tls_certs_only(roots) - .identity(identity) - .build() - .map_err(Into::into) + build_client(&self.roots, self.config.schedule.timeout, Some(identity), self.config.proxy.as_ref()) + .map_err(credential_recovery_error) } } @@ -197,6 +206,10 @@ fn credential_recovery_error(error: ClientError) -> TelemetryError { match error { ClientError::Endpoint => TelemetryError::Endpoint, ClientError::RootCertificate => TelemetryError::RootCertificate, + ClientError::ProxyConfiguration(_) => TelemetryError::ProxyConfiguration, + ClientError::ProxyAuthentication => TelemetryError::ProxyAuthentication, + ClientError::ProxyRejected => TelemetryError::ProxyRejected, + ClientError::TlsPeer => TelemetryError::TlsPeer, ClientError::PendingRegistration | ClientError::PendingRotation => TelemetryError::StateConflict, ClientError::NotRegistered => TelemetryError::NotRegistered, ClientError::IdentityMissing => TelemetryError::IdentityMissing, @@ -291,6 +304,16 @@ pub(crate) enum TelemetryError { Endpoint, #[error("Connect telemetry root CA configuration is invalid")] RootCertificate, + #[error("Connect telemetry proxy configuration is invalid")] + ProxyConfiguration, + #[error("Connect proxy authentication failed; verify the configured proxy credential files")] + ProxyAuthentication, + #[error( + "Connect proxy connection failed; verify proxy availability, credentials, the proxy allow-list, and the Connect endpoint" + )] + ProxyRejected, + #[error("Connect TLS peer certificate validation failed; verify the endpoint and configured root CA")] + TlsPeer, #[error("Connect telemetry retry schedule is invalid")] Schedule, #[error("RustFS is not registered with Connect")] diff --git a/rustfs/src/server/compress.rs b/rustfs/src/server/compress.rs index b678c1277..f43c68ffb 100644 --- a/rustfs/src/server/compress.rs +++ b/rustfs/src/server/compress.rs @@ -418,7 +418,7 @@ impl PathCategory { PathCategory::InternodeRpc } else if path.starts_with("/rustfs/admin/") || path.starts_with("/minio/admin/") { PathCategory::AdminApi - } else if path.starts_with("/rustfs/console") { + } else if crate::server::has_path_prefix(path, crate::server::console_prefix()) { PathCategory::Console } else if path == "/health" || path.starts_with("/health/") @@ -766,8 +766,10 @@ mod tests { #[test] fn test_path_category_classify_console() { - assert_eq!(PathCategory::classify("/rustfs/console/index.html"), PathCategory::Console); - assert_eq!(PathCategory::classify("/rustfs/console"), PathCategory::Console); + let prefix = crate::server::console_prefix(); + assert_eq!(PathCategory::classify(&format!("{prefix}/index.html")), PathCategory::Console); + assert_eq!(PathCategory::classify(prefix), PathCategory::Console); + assert_eq!(PathCategory::classify(&format!("{prefix}-other/index.html")), PathCategory::S3DataPlane); } #[test] diff --git a/rustfs/src/server/http.rs b/rustfs/src/server/http.rs index 2f115b939..80c92fd8f 100644 --- a/rustfs/src/server/http.rs +++ b/rustfs/src/server/http.rs @@ -26,7 +26,7 @@ use crate::server::{ BodylessStatusFixLayer, ConditionalCorsLayer, DoubleSlashListBucketsCompatLayer, EmptyBodyContentLengthCompatLayer, ExternalRequestContextLayer, HeadRequestBodyFixLayer, IcebergRestErrorCompatLayer, ObjectAttributesEtagFixLayer, PublicHealthEndpointLayer, RedirectLayer, RequestContextLayer, RequestLoggingLayer, S3ErrorMessageCompatLayer, - StsQueryApiCompatLayer, VirtualHostStyleHintLayer, redact_sensitive_uri_query, + SigV4HeaderGuardLayer, StsQueryApiCompatLayer, VirtualHostStyleHintLayer, redact_sensitive_uri_query, }, rate_limit::{RateLimitLayer, api_rate_limit_layer_from_env}, ssec_transport::SsecTransportLayer, @@ -57,7 +57,10 @@ use hyper_util::{ use metrics::{counter, gauge, histogram}; use opentelemetry::global; use opentelemetry::trace::TraceContextExt; -use rustfs_common::GlobalReadiness; +use rustfs_common::{ + GlobalReadiness, + trace_bus::{TelemetryTraceEvent, TelemetryTraceOperation, TelemetryTraceStatus, telemetry_trace_emit}, +}; use rustfs_io_metrics::internode_metrics::{ INTERNODE_OPERATION_GRPC_COMPARE_AND_UPDATE_FILE, INTERNODE_OPERATION_GRPC_OTHER, INTERNODE_OPERATION_GRPC_READ_ALL, INTERNODE_OPERATION_GRPC_READ_MULTIPLE, INTERNODE_OPERATION_GRPC_WRITE_ALL, INTERNODE_TRANSPORT_BACKEND_GRPC, @@ -85,7 +88,7 @@ use std::pin::Pin; use std::sync::Arc; use std::sync::atomic::{AtomicU64, Ordering}; use std::task::{Context, Poll}; -use std::time::Duration; +use std::time::{Duration, Instant}; use tokio::net::{TcpListener, TcpStream}; use tokio::sync::{OwnedSemaphorePermit, Semaphore}; use tonic::service::Routes; @@ -259,6 +262,93 @@ struct RpcRequestPathService { inner: S, } +struct RpcCompletionBody { + inner: B, + started_at: Instant, + header_status: Option, + complete: bool, +} + +impl RpcCompletionBody { + fn new(inner: B, started_at: Instant, header_status: Option) -> Self { + Self { + inner, + started_at, + header_status, + complete: false, + } + } + + fn complete(&mut self, status: TelemetryTraceStatus) { + if self.complete { + return; + } + self.complete = true; + telemetry_trace_emit(|| { + TelemetryTraceEvent::new(TelemetryTraceOperation::InternalRpc, self.started_at.elapsed(), status) + }); + } +} + +impl http_body::Body for RpcCompletionBody +where + B: http_body::Body + Unpin, +{ + type Data = Bytes; + type Error = B::Error; + + fn is_end_stream(&self) -> bool { + self.complete || self.inner.is_end_stream() + } + + fn poll_frame( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + ) -> Poll, Self::Error>>> { + match Pin::new(&mut self.inner).poll_frame(cx) { + Poll::Ready(Some(Ok(frame))) => { + if let Some(trailers) = frame.trailers_ref() { + let status = grpc_telemetry_status(trailers).unwrap_or(TelemetryTraceStatus::Error); + self.complete(status); + } + Poll::Ready(Some(Ok(frame))) + } + Poll::Ready(Some(Err(error))) => { + self.complete(TelemetryTraceStatus::Error); + Poll::Ready(Some(Err(error))) + } + Poll::Ready(None) => { + let status = self.header_status.unwrap_or(TelemetryTraceStatus::Error); + self.complete(status); + Poll::Ready(None) + } + Poll::Pending => Poll::Pending, + } + } + + fn size_hint(&self) -> http_body::SizeHint { + self.inner.size_hint() + } +} + +impl Drop for RpcCompletionBody { + fn drop(&mut self) { + if !self.complete { + self.complete(self.header_status.unwrap_or(TelemetryTraceStatus::Error)); + } + } +} + +fn grpc_telemetry_status(headers: &HeaderMap) -> Option { + headers.get("grpc-status").map(|status| { + if status == "0" { + TelemetryTraceStatus::Ok + } else { + TelemetryTraceStatus::Error + } + }) +} + impl RpcRequestPathService { fn new(inner: S) -> Self { Self { inner } @@ -271,9 +361,9 @@ where S::Error: Send + 'static, S::Future: Send + 'static, B: Send + 'static, - ResBody: Send + 'static, + ResBody: http_body::Body + Unpin + Send + 'static, { - type Response = Response; + type Response = Response>; type Error = S::Error; type Future = Pin> + Send>>; @@ -282,6 +372,7 @@ where } fn call(&mut self, mut req: HttpRequest) -> Self::Future { + let started_at = Instant::now(); let target = RpcRequestTarget { uri: req.uri().clone(), method: req.method().clone(), @@ -301,7 +392,10 @@ where if let Some(headers) = response_headers { response.headers_mut().extend(headers); } - Ok(response) + let header_status = grpc_telemetry_status(response.headers()); + let (parts, body) = response.into_parts(); + let tracked = RpcCompletionBody::new(body, started_at, header_status); + Ok(Response::from_parts(parts, tracked)) }) } } @@ -964,6 +1058,7 @@ pub async fn start_http_server( readiness: Arc, server_ctx: Arc, ) -> Result<(ShutdownHandle, SocketAddr)> { + crate::server::init_console_prefix()?; let server_addr = parse_and_resolve_address(config.address.as_str()).map_err(Error::other)?; // The listening address and port are obtained from the parameters @@ -1211,6 +1306,7 @@ pub async fn start_http_server( let now_time = jiff::Zoned::now().strftime("%Y-%m-%d %H:%M:%S").to_string(); if config.console_enable { admin::console::init_console_cfg(local_ip, local_port); + let console_prefix = crate::server::console_prefix(); info!( target: "rustfs::console::startup", @@ -1218,7 +1314,7 @@ pub async fn start_http_server( component = LOG_COMPONENT_SERVER, subsystem = LOG_SUBSYSTEM_STARTUP, service = "console", - endpoint = %format!("{protocol}://{local_ip_str}:{local_port}/rustfs/console/index.html"), + endpoint = %format!("{protocol}://{local_ip_str}:{local_port}{console_prefix}/index.html"), "Startup endpoint available" ); info!( @@ -1227,7 +1323,7 @@ pub async fn start_http_server( component = LOG_COMPONENT_SERVER, subsystem = LOG_SUBSYSTEM_STARTUP, service = "console_localhost", - endpoint = %format!("{protocol}://127.0.0.1:{local_port}/rustfs/console/index.html"), + endpoint = %format!("{protocol}://127.0.0.1:{local_port}{console_prefix}/index.html"), "Startup endpoint available" ); } else { @@ -1941,6 +2037,7 @@ fn process_connection( // 22. PublicHealthEndpointLayer — handles public health before s3s host parsing // 23. VirtualHostStyleHintLayer — actionable error for unroutable virtual-hosted-style (conditional) // 24. DoubleSlashListBucketsCompatLayer — rewrites `GET //` to `GET /` for ListBuckets (MinIO browser compat) + // 25. SigV4HeaderGuardLayer — GHSA-xm99/-g8w9 unsigned x-amz-* rules, ahead of s3s signature dispatch // The internode lane below intentionally keeps only the shared // transport/auth/observability subset needed by `/rustfs/rpc/...`. // ───────────────────────────────────────────────────────────── @@ -2060,6 +2157,7 @@ fn process_connection( )) .option_layer((!server_domains_configured && !is_console).then_some(VirtualHostStyleHintLayer)) .layer(DoubleSlashListBucketsCompatLayer) + .layer(SigV4HeaderGuardLayer) .service(service) }; let build_internode_stack = |service| { @@ -3438,6 +3536,49 @@ mod tests { assert_eq!(captured.method, Method::POST); } + #[tokio::test] + #[serial_test::serial] + async fn rpc_completion_waits_for_the_grpc_stream_trailer() { + use http_body_util::StreamBody; + use rustfs_common::trace_bus::subscribe_telemetry_trace_events; + use tokio_stream::wrappers::ReceiverStream; + + let mut subscription = subscribe_telemetry_trace_events(); + let (tx, rx) = mpsc::channel::, Infallible>>(2); + let mut body = RpcCompletionBody::new(StreamBody::new(ReceiverStream::new(rx)), Instant::now(), None); + + tx.send(Ok(Frame::data(Bytes::from_static(b"rpc-data")))) + .await + .expect("response data frame should send"); + let frame = body + .frame() + .await + .expect("response data frame") + .expect("response body should remain valid"); + assert!(frame.is_data()); + assert!(matches!(subscription.try_recv(), Err(tokio::sync::broadcast::error::TryRecvError::Empty))); + + let mut trailers = HeaderMap::new(); + trailers.insert("grpc-status", HeaderValue::from_static("0")); + tx.send(Ok(Frame::trailers(trailers))) + .await + .expect("response trailer should send"); + let frame = body + .frame() + .await + .expect("response trailer frame") + .expect("response body should remain valid"); + assert!(frame.is_trailers()); + + let event = tokio::time::timeout(Duration::from_secs(1), subscription.recv()) + .await + .expect("RPC completion event should arrive") + .expect("telemetry source should remain open"); + assert_eq!(event.operation, TelemetryTraceOperation::InternalRpc); + assert_eq!(event.status, TelemetryTraceStatus::Ok); + assert!(event.duration > Duration::ZERO); + } + #[tokio::test] #[serial_test::serial] async fn rpc_auth_binds_post_method_authority_and_exact_path() { diff --git a/rustfs/src/server/layer.rs b/rustfs/src/server/layer.rs index 78684e22b..f0e9f484e 100644 --- a/rustfs/src/server/layer.rs +++ b/rustfs/src/server/layer.rs @@ -20,10 +20,10 @@ use crate::server::RemoteAddr; use crate::server::cors; use crate::server::hybrid::{HybridBody, is_grpc_request}; use crate::server::{ - ADMIN_PREFIX, CONSOLE_PREFIX, HEALTH_COMPAT_LIVE_PATH, HEALTH_PREFIX, HEALTH_READY_PATH, HealthProbe, MINIO_ADMIN_PREFIX, + ADMIN_PREFIX, HEALTH_COMPAT_LIVE_PATH, HEALTH_PREFIX, HEALTH_READY_PATH, HealthProbe, MINIO_ADMIN_PREFIX, MINIO_ADMIN_V3_PREFIX, MINIO_HEALTH_CLUSTER_PATH, MINIO_HEALTH_CLUSTER_READ_PATH, MINIO_HEALTH_LIVE_PATH, MINIO_HEALTH_READY_PATH, PROFILE_CPU_PATH, PROFILE_MEMORY_PATH, RPC_PREFIX, RUSTFS_ADMIN_PREFIX, active_http_requests, - build_health_response_parts, collect_probe_readiness, has_path_prefix, is_admin_path, is_table_catalog_path, + build_health_response_parts, collect_probe_readiness, console_prefix, has_path_prefix, is_admin_path, is_table_catalog_path, kms_probe_staleness_limit, kms_ready_from_probe, }; use crate::shared_types::ReadinessDegradedReason; @@ -38,6 +38,9 @@ use hyper::body::Incoming; use pin_project_lite::pin_project; use quick_xml::events::Event; use rustfs_common::GlobalReadiness; +use rustfs_common::trace_bus::{ + TelemetryTraceEvent, TelemetryTraceOperation, TelemetryTraceStatus, telemetry_trace_emit, telemetry_trace_subscriber_count, +}; use rustfs_io_metrics::s3_http_metrics::S3HttpRequestGuard; use rustfs_obs::HTTP_SERVER_LOG_TARGET; #[cfg(feature = "swift")] @@ -45,6 +48,7 @@ use rustfs_protocols::swift::SwiftRouter; use rustfs_trusted_proxies::ClientInfo; use rustfs_utils::get_env_opt_str; use rustfs_utils::http::headers::{AMZ_REQUEST_ID, REQUEST_ID_HEADER}; +use s3s::S3Error; use s3s::S3ErrorCode; use serde::{Deserialize, Serialize}; use std::borrow::Cow; @@ -273,7 +277,7 @@ where // This outer boundary includes readiness, rate-limit and auth // rejections. Metric attribution never depends on an enabled span. - let mut metrics = is_s3.then(|| S3HttpRequestGuard::new(req.method().as_str())); + let mut metrics = is_s3.then(|| s3_http_request_guard(req.method().as_str())); let inner = match metrics.as_mut() { Some(metrics) => metrics.in_scope(|| self.inner.call(req)), None => self.inner.call(req), @@ -287,6 +291,40 @@ where } } +/// Start accounting for an external S3 request. While a typed telemetry trace +/// is being recorded, the finished request is also published to the trace bus +/// as a pre-classified event; otherwise no clock is read. +pub fn s3_http_request_guard(method: &str) -> S3HttpRequestGuard { + let guard = S3HttpRequestGuard::new(method); + if telemetry_trace_subscriber_count() == 0 { + return guard; + } + guard.with_completion_observer(emit_s3_request_telemetry) +} + +fn emit_s3_request_telemetry(operation: rustfs_s3_ops::S3Operation, duration: Duration, succeeded: bool) { + let Some(operation) = telemetry_operation(operation) else { + return; + }; + let status = if succeeded { + TelemetryTraceStatus::Ok + } else { + TelemetryTraceStatus::Error + }; + telemetry_trace_emit(|| TelemetryTraceEvent::new(operation, duration, status)); +} + +fn telemetry_operation(operation: rustfs_s3_ops::S3Operation) -> Option { + use rustfs_s3_ops::S3Operation; + match operation { + S3Operation::GetObject => Some(TelemetryTraceOperation::GetObject), + S3Operation::PutObject => Some(TelemetryTraceOperation::PutObject), + S3Operation::HeadObject => Some(TelemetryTraceOperation::HeadObject), + S3Operation::ListObjects | S3Operation::ListObjectsV2 => Some(TelemetryTraceOperation::ListObjects), + _ => None, + } +} + pin_project! { pub struct ExternalRequestContextFuture { #[pin] @@ -625,7 +663,7 @@ where // Create redirect response let redirect_response = Response::builder() .status(StatusCode::FOUND) - .header(http::header::LOCATION, "/rustfs/console/") + .header(http::header::LOCATION, format!("{}/", console_prefix())) .body(HybridBody::Rest { rest_body: RestBody::default(), }) @@ -1569,6 +1607,94 @@ where .expect("failed to build virtual-host hint response") } +/// GHSA-xm99-m3gq-83g8 / GHSA-g8w9-qw9q-fghr: enforce the SigV4 unsigned +/// `x-amz-*` header rules ahead of s3s dispatch. +/// +/// s3s verifies the claimed algorithm as the first step of its own signature +/// flow and answers a swapped algorithm token with `501 NotImplemented` before +/// RustFS's access layer (`S3Access::check`) ever runs, so the `AccessDenied` +/// rulings of [`crate::auth::reject_unsigned_amz_headers_on_sigv4_request`] +/// must be applied here, in front of s3s. Rejections carry the same S3 error +/// document the access layer would have produced. +#[derive(Clone, Default)] +pub struct SigV4HeaderGuardLayer; + +impl Layer for SigV4HeaderGuardLayer { + type Service = SigV4HeaderGuardService; + + fn layer(&self, inner: S) -> Self::Service { + SigV4HeaderGuardService { inner } + } +} + +#[derive(Clone)] +pub struct SigV4HeaderGuardService { + inner: S, +} + +impl Service> for SigV4HeaderGuardService +where + S: Service, Response = Response>> + Clone + Send + 'static, + S::Future: Send + 'static, + ReqBody: Send + 'static, + RestBody: From + Send + 'static, + GrpcBody: Send + 'static, +{ + type Response = Response>; + type Error = S::Error; + type Future = Pin> + Send>>; + + fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll> { + self.inner.poll_ready(cx) + } + + fn call(&mut self, req: HttpRequest) -> Self::Future { + match crate::auth::reject_unsigned_amz_headers_on_sigv4_request(req.headers(), req.uri().query()) { + Ok(()) => {} + Err(error) => { + let version = req.version(); + return Box::pin(async move { Ok(sigv4_header_guard_rejection(version, error)) }); + } + } + let mut inner = self.inner.clone(); + Box::pin(async move { inner.call(req).await }) + } +} + +/// Serialize a header-guard rejection as the S3 error document the access +/// layer would have produced for the same rule violation. +fn sigv4_header_guard_rejection( + version: http::Version, + error: S3Error, +) -> Response> +where + RestBody: From, +{ + let status = error.status_code().unwrap_or(StatusCode::FORBIDDEN); + let message = error.message().unwrap_or_default().to_owned(); + let body = format!( + "\ + {code}{message}", + code = xml_escape(error.code().as_str()), + message = xml_escape(&message), + ); + + let mut builder = Response::builder() + .status(status) + .header(http::header::CONTENT_TYPE, "application/xml"); + // This short-circuit path does not drain the request body. For HTTP/1.x, signal + // connection close so an undrained body cannot disrupt keep-alive reuse. `Connection` + // is a forbidden header in HTTP/2+, so it is only set for HTTP/1.x. + if !matches!(version, http::Version::HTTP_2 | http::Version::HTTP_3) { + builder = builder.header(http::header::CONNECTION, "close"); + } + builder + .body(HybridBody::Rest { + rest_body: RestBody::from(Bytes::from(body)), + }) + .expect("failed to build SigV4 header guard rejection response") +} + /// Returns an actionable error for virtual-hosted-style S3 requests that cannot be /// routed because `RUSTFS_SERVER_DOMAINS` is not configured. See /// [`unroutable_virtual_host_target`]. The layer is only installed when no server @@ -1861,7 +1987,7 @@ fn is_object_attributes_request(req: &HttpRequest) -> bool { || has_path_prefix(path, RUSTFS_ADMIN_PREFIX) || has_path_prefix(path, MINIO_ADMIN_V3_PREFIX) || is_table_catalog_path(path) - || has_path_prefix(path, CONSOLE_PREFIX) + || has_path_prefix(path, console_prefix()) || has_path_prefix(path, RPC_PREFIX) { return false; @@ -2242,7 +2368,74 @@ fn rewrite_double_slash_root(uri: &Uri) -> Option { #[cfg(test)] mod tests { + #[tokio::test] + async fn console_prefix_process_case_browser_redirect() { + if std::env::var_os("RUSTFS_TEST_CONSOLE_PREFIX_PROCESS").is_none() { + return; + } + crate::server::init_console_prefix().expect("initialize console prefix"); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.expect("redirect listener"); + let addr = listener.local_addr().expect("redirect listener address"); + let server = tokio::spawn(async move { + let (stream, _) = listener.accept().await.expect("redirect client"); + let inner = tower::service_fn(|_request: Request| async { + Ok::<_, Infallible>(Response::new(HybridBody::, Empty>::Rest { rest_body: Empty::new() })) + }); + let service = RedirectLayer.layer(inner); + hyper::server::conn::http1::Builder::new() + .serve_connection( + hyper_util::rt::TokioIo::new(stream), + hyper_util::service::TowerToHyperService::new(service), + ) + .await + .expect("redirect connection"); + }); + let client = reqwest::Client::builder() + .no_proxy() + .http1_only() + .redirect(reqwest::redirect::Policy::none()) + .timeout(Duration::from_secs(5)) + .build() + .expect("redirect client"); + let response = client + .get(format!("http://{addr}/")) + .header(http::header::USER_AGENT, "Mozilla/5.0") + .header(http::header::CONNECTION, "close") + .send() + .await + .expect("browser response"); + assert_eq!(response.status(), StatusCode::FOUND); + assert_eq!(response.headers()[http::header::LOCATION], format!("{}/", console_prefix())); + response.bytes().await.expect("redirect body"); + tokio::time::timeout(Duration::from_secs(5), server) + .await + .expect("bounded redirect server shutdown") + .expect("redirect task"); + } + + #[test] + fn console_prefix_process_case_classification() { + if std::env::var_os("RUSTFS_TEST_CONSOLE_PREFIX_PROCESS").is_none() { + return; + } + crate::server::init_console_prefix().expect("initialize console prefix"); + let prefix = crate::server::console_prefix(); + let console_uri = format!("{prefix}/index.html").parse().expect("console URI"); + assert!(is_empty_body_console_path(&Method::GET, &console_uri)); + let request = HttpRequest::builder() + .uri(format!("{prefix}/index.html?attributes")) + .body(()) + .expect("console attributes request"); + assert!(!is_object_attributes_request(&request)); + let s3_request = HttpRequest::builder() + .uri("/bucket/object?attributes") + .body(()) + .expect("S3 attributes request"); + assert!(is_object_attributes_request(&s3_request)); + } + use super::*; + use crate::server::CONSOLE_PREFIX; use crate::server::compress::{HttpCompressionConfig, PathAwareHttpCompressionPredicate, PathCategoryInjectionLayer}; use crate::server::{FAVICON_PATH, LICENSE, RemoteAddr, VERSION}; use futures::future::{Ready, ready}; @@ -2260,6 +2453,20 @@ mod tests { use temp_env::{async_with_vars, with_var}; use tracing_subscriber::{Registry, fmt::MakeWriter, layer::SubscriberExt}; + #[test] + fn telemetry_adapter_accepts_only_the_frozen_s3_operations() { + use rustfs_s3_ops::S3Operation; + assert_eq!(telemetry_operation(S3Operation::GetObject), Some(TelemetryTraceOperation::GetObject)); + assert_eq!(telemetry_operation(S3Operation::PutObject), Some(TelemetryTraceOperation::PutObject)); + assert_eq!(telemetry_operation(S3Operation::HeadObject), Some(TelemetryTraceOperation::HeadObject)); + assert_eq!(telemetry_operation(S3Operation::ListObjects), Some(TelemetryTraceOperation::ListObjects)); + assert_eq!( + telemetry_operation(S3Operation::ListObjectsV2), + Some(TelemetryTraceOperation::ListObjects) + ); + assert_eq!(telemetry_operation(S3Operation::DeleteObject), None); + } + fn public_health_layer() -> PublicHealthEndpointLayer { let readiness = Arc::new(GlobalReadiness::new()); readiness.mark_stage(rustfs_common::SystemStage::FullReady); @@ -2333,7 +2540,7 @@ mod tests { for path in [ "/rustfs/admin/v3/metrics", "/minio/admin/v3/storageinfo", - "/rustfs/console/", + CONSOLE_PREFIX, "/rustfs/rpc/test", "/health/ready", "/_iceberg/v1/config", @@ -2624,7 +2831,7 @@ mod tests { for path in [ "/rustfs/admin/v3/info", "/minio/admin/v3/info", - "/rustfs/console/", + CONSOLE_PREFIX, HEALTH_PREFIX, "/iceberg/v1/config", "/rustfs/rpc/v1/read-file", @@ -3024,6 +3231,122 @@ mod tests { assert!(h2_response.headers().get(http::header::CONNECTION).is_none()); } + #[tokio::test] + async fn sigv4_header_guard_layer_rejects_swapped_algorithm_token_with_access_denied() { + let inner = CountingHybridService::default(); + let calls = inner.calls(); + let mut service = SigV4HeaderGuardLayer.layer(inner); + + let response = service + .call( + Request::builder() + .method(Method::PUT) + .uri("/xm99-private-source/target") + .header( + "authorization", + "OTHER Credential=rustfsadmin/20260914/us-east-1/s3/aws4_request, \ + SignedHeaders=host;x-amz-content-sha256;x-amz-date, \ + Signature=00e997a1db4d3b6ee6c26d3f7d3f3fb3b6ee6c26d3f7d3f3fb3b6ee6c26d3f7d", + ) + .header("x-amz-date", "20260914T000000Z") + .header("x-amz-content-sha256", "UNSIGNED-PAYLOAD") + .header("x-amz-copy-source", "/negative-sigv4-bucket/source") + .body(Full::::from(Bytes::new())) + .expect("request"), + ) + .await + .expect("guard response"); + + // The swapped token must be answered with the access-layer ruling + // instead of s3s's algorithm 501. + assert_eq!(response.status(), StatusCode::FORBIDDEN); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let body = BodyExt::collect(response.into_body()).await.expect("body").to_bytes(); + let body = String::from_utf8(body.to_vec()).expect("utf8 body"); + assert!(body.contains("AccessDenied"), "body: {body}"); + assert!(body.contains("Unsupported SigV4 authorization algorithm"), "body: {body}"); + } + + #[tokio::test] + async fn sigv4_header_guard_layer_rejects_unsigned_copy_source_header() { + let inner = CountingHybridService::default(); + let calls = inner.calls(); + let mut service = SigV4HeaderGuardLayer.layer(inner); + + let response = service + .call( + Request::builder() + .method(Method::PUT) + .uri("/xm99-private-source/target") + .header( + "authorization", + "AWS4-HMAC-SHA256 Credential=rustfsadmin/20260914/us-east-1/s3/aws4_request, \ + SignedHeaders=host;x-amz-content-sha256;x-amz-date, \ + Signature=00e997a1db4d3b6ee6c26d3f7d3f3fb3b6ee6c26d3f7d3f3fb3b6ee6c26d3f7d", + ) + .header("x-amz-date", "20260914T000000Z") + .header("x-amz-content-sha256", "UNSIGNED-PAYLOAD") + .header("x-amz-copy-source", "/negative-sigv4-bucket/source") + .body(Full::::from(Bytes::new())) + .expect("request"), + ) + .await + .expect("guard response"); + + assert_eq!(response.status(), StatusCode::FORBIDDEN); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let body = BodyExt::collect(response.into_body()).await.expect("body").to_bytes(); + let body = String::from_utf8(body.to_vec()).expect("utf8 body"); + assert!(body.contains("AccessDenied"), "body: {body}"); + assert!( + body.contains("There were headers present in the request which were not signed"), + "body: {body}" + ); + } + + #[tokio::test] + async fn sigv4_header_guard_layer_passes_unsigned_and_signed_envelope_requests_through() { + let inner = CountingHybridService::default(); + let calls = inner.calls(); + let mut service = SigV4HeaderGuardLayer.layer(inner); + + // Anonymous request: no Authorization header, no presigned query. + let response = service + .call( + Request::builder() + .method(Method::GET) + .uri("/bucket/key") + .body(Full::::from(Bytes::new())) + .expect("request"), + ) + .await + .expect("inner response"); + assert_eq!(response.status(), StatusCode::IM_A_TEAPOT); + assert_eq!(calls.load(Ordering::SeqCst), 1); + + // Header-signed request whose only x-amz-* headers are the signed envelope. + let response = service + .call( + Request::builder() + .method(Method::PUT) + .uri("/bucket/key") + .header( + "authorization", + "AWS4-HMAC-SHA256 Credential=rustfsadmin/20260914/us-east-1/s3/aws4_request, \ + SignedHeaders=host;x-amz-content-sha256;x-amz-date, \ + Signature=00e997a1db4d3b6ee6c26d3f7d3f3fb3b6ee6c26d3f7d3f3fb3b6ee6c26d3f7d", + ) + .header("x-amz-date", "20260914T000000Z") + .header("x-amz-content-sha256", "UNSIGNED-PAYLOAD") + .body(Full::::from(Bytes::new())) + .expect("request"), + ) + .await + .expect("inner response"); + assert_eq!(response.status(), StatusCode::IM_A_TEAPOT); + assert_eq!(calls.load(Ordering::SeqCst), 2); + } + #[tokio::test] async fn virtual_host_style_hint_layer_short_circuits_unroutable_put() { let inner = CountingHybridService::default(); @@ -3983,7 +4306,7 @@ mod tests { "/minio/admin/v3/pools/cancel?versionId=unused", "/rustfs/admin/v3/pools/cancel?versionId=unused", "/rustfs/rpc/read_file_stream?versionId=unused", - "/rustfs/console/index.html?versionId=unused", + &format!("{CONSOLE_PREFIX}/index.html?versionId=unused"), "/health?versionId=unused", "/health/ready?versionId=unused", "/profile/cpu?versionId=unused", diff --git a/rustfs/src/server/mod.rs b/rustfs/src/server/mod.rs index cee19e6fa..91b1ebc4d 100644 --- a/rustfs/src/server/mod.rs +++ b/rustfs/src/server/mod.rs @@ -61,6 +61,7 @@ pub(crate) use health::{ }; pub(crate) use http::HeaderMapCarrier; pub(crate) use http::active_http_requests; +pub use layer::s3_http_request_guard; pub(crate) use layer::{RequestContextLayer, is_sts_query_request}; pub(crate) use module_switch::{ MODULE_SWITCHES_SIGNAL_SUBSYSTEM, ModuleSwitchSnapshot, ModuleSwitchSource, PersistedModuleSwitches, @@ -72,7 +73,7 @@ pub(crate) use prefix::{ HEALTH_COMPAT_LIVE_PATH, HEALTH_PREFIX, HEALTH_READY_PATH, LICENSE, MINIO_ADMIN_PREFIX, MINIO_ADMIN_V3_PREFIX, MINIO_HEALTH_CLUSTER_PATH, MINIO_HEALTH_CLUSTER_READ_PATH, MINIO_HEALTH_LIVE_PATH, MINIO_HEALTH_READY_PATH, PROFILE_CPU_PATH, PROFILE_MEMORY_PATH, RPC_PREFIX, RUSTFS_ADMIN_PREFIX, TABLE_CATALOG_COMPAT_PREFIX, TABLE_CATALOG_PREFIX, TONIC_PREFIX, - VERSION, has_path_prefix, is_admin_path, is_table_catalog_path, + VERSION, console_prefix, has_path_prefix, init_console_prefix, is_admin_path, is_table_catalog_path, }; pub(crate) use readiness::ReadinessDegradedReason; pub(crate) use readiness::ReadinessGateLayer; diff --git a/rustfs/src/server/prefix.rs b/rustfs/src/server/prefix.rs index f0f4120ed..a4445dd09 100644 --- a/rustfs/src/server/prefix.rs +++ b/rustfs/src/server/prefix.rs @@ -83,10 +83,81 @@ pub(crate) const RUSTFS_ADMIN_PREFIX: &str = "/rustfs/admin/v3"; /// MinIO-compatible admin API prefix accepted by RustFS. pub(crate) const MINIO_ADMIN_V3_PREFIX: &str = "/minio/admin/v3"; -/// Predefined console prefix for RustFS server routes. -/// This prefix is used for endpoints that handle console-related tasks -/// such as user interface and management. -pub(crate) const CONSOLE_PREFIX: &str = "/rustfs/console"; +/// Console asset base path embedded at build time and used as the startup default. +/// It must match NEXT_PUBLIC_BASE_PATH when building the bundled frontend. +pub(crate) const CONSOLE_PREFIX: &str = match option_env!("RUSTFS_CONSOLE_BASE_PATH") { + Some(path) if !path.is_empty() => path, + _ => rustfs_config::DEFAULT_CONSOLE_PREFIX, +}; + +static CONFIGURED_CONSOLE_PREFIX: std::sync::OnceLock = std::sync::OnceLock::new(); + +/// The prefix is fixed before listeners start; request handling never reads the environment. +pub(crate) fn console_prefix() -> &'static str { + CONFIGURED_CONSOLE_PREFIX.get().map(String::as_str).unwrap_or(CONSOLE_PREFIX) +} + +pub(crate) fn init_console_prefix() -> std::io::Result<()> { + let raw = match std::env::var(rustfs_config::ENV_RUSTFS_CONSOLE_PREFIX) { + Ok(value) => value, + Err(std::env::VarError::NotPresent) => CONSOLE_PREFIX.to_string(), + Err(err) => return Err(std::io::Error::new(std::io::ErrorKind::InvalidInput, err)), + }; + let prefix = validate_console_prefix(&raw)?; + if CONFIGURED_CONSOLE_PREFIX.get_or_init(|| prefix.clone()) != &prefix { + return Err(std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "RUSTFS_CONSOLE_PREFIX cannot change after server initialization", + )); + } + Ok(()) +} + +fn validate_console_prefix(raw: &str) -> std::io::Result { + let prefix = raw.strip_suffix('/').unwrap_or(raw); + // Keep the value safe in HTTP headers, Axum routes, and embedded HTML/JS. + if !prefix.starts_with('/') + || prefix.len() > 256 + || prefix[1..].split('/').any(|segment| { + segment.is_empty() + || matches!(segment, "." | "..") + || !segment + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_' | b'.' | b'~')) + }) + { + return Err(std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "RUSTFS_CONSOLE_PREFIX must be a non-root absolute path of at most 256 bytes with nonempty URL-safe segments", + )); + } + let reserved = [ + ADMIN_PREFIX, + MINIO_ADMIN_PREFIX, + TABLE_CATALOG_PREFIX, + TABLE_CATALOG_COMPAT_PREFIX, + RPC_PREFIX, + TONIC_PREFIX, + "/rustfs/peer", + HEALTH_PREFIX, + "/minio/health", + "/profile", + "/index.html", + FAVICON_PATH, + APPLE_TOUCH_ICON_PATH, + APPLE_TOUCH_ICON_PRECOMPOSED_PATH, + ]; + if reserved + .iter() + .any(|path| has_path_prefix(prefix, path) || has_path_prefix(path, prefix)) + { + return Err(std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "RUSTFS_CONSOLE_PREFIX overlaps a reserved server route", + )); + } + Ok(prefix.to_string()) +} /// Predefined RPC prefix for RustFS server routes. /// This prefix is used for endpoints that handle remote procedure calls (RPC). @@ -111,3 +182,147 @@ pub const LOGO: &str = r#" ░▀░▀░▀▀▀░▀▀▀░░▀░░▀░░░▀▀▀ "#; + +#[cfg(test)] +mod console_prefix_tests { + use super::*; + + #[test] + fn console_prefix_validation() { + for (raw, expected) in [ + (CONSOLE_PREFIX, CONSOLE_PREFIX), + ("/console", "/console"), + ("/management/console/", "/management/console"), + ("/health-dashboard", "/health-dashboard"), + ] { + assert_eq!(validate_console_prefix(raw).expect("valid console prefix"), expected); + } + for raw in [ + "", + "/", + "console", + "//console", + "/console//", + "/a//b", + "/a/../b", + "/a/./b", + "/%2e%2e", + "/console?x=1", + "/console#x", + "/console\\x", + "/a\n", + "/{param}", + "/