13 Commits

Author	SHA1	Message	Date
唐小鸭	46907c05cf	fix(replication): close the GA blocker set from backlog#2366 (#7503) ... * fix(replication): close GA blockers from backlog#2366 Implements the P1 set from the pre-GA replication audit: - Replication rule tag filters now require every And.Tag to match, replacing the s3s OR semantics with a local AND matcher that fails closed on a malformed tag. - A replicated group membership change no longer writes the group status, so a membership update carrying the default Enabled status cannot silently re-enable a disabled group on the peer. - A successful IAM import schedules one collapsed full-IAM snapshot per remote peer instead of leaving the imported entities local-only. - A pending endpoint refresh is redriven by the heavyweight reconcile tick, carries its own ilm-expiry override, and no longer blocks a remove that drops every unacknowledged peer. - Site metrics expose local replication failure totals and rolling windows; node-level counters no longer report a constructed zero. - set/remove-remote-target notify peer metadata caches before returning, so a follow-up put-bucket-replication on another node sees the target. - Adds the site-replication operations runbook, a docs index, a replication support boundary section, and the Replication changelog section. * fix(site-replication): resume only a locally driven endpoint refresh The peer-side edit handler journals a pending endpoint refresh with an empty `remote_peers` map and commits it inside the same request through `apply_internal_peer_edit`. The reconcile tick could not tell that journal from the coordinator's own: with no required peers it reads as complete on sight, so the tick committed it with `edit_state` - losing the local-name sync - and cleared it under the request that owned it, whose commit then reported the refresh as changed and denied the coordinator the peer acknowledgement it was waiting for. Resume now runs only for a journal that carries the fan-out topology. A receiver's journal stays for the coordinator to redrive with the same refresh id, which is the path that already recovers it. * fix(site-replication): keep an explicit disabled group status on a snapshot Skipping the group-status write whenever an item carries members stopped a membership change from re-enabling a disabled group, but it also silenced the full-IAM snapshot, which always sends members together with the sender's real status. A peer that did not have the group yet created it through `GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM import now schedules handed every member of a frozen group live access there. The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be a default. Disabled is always explicit and is applied again. * fix(site-replication): schedule the import snapshot without recording a failure `import-iam` reused the failure-recording path to queue its full-IAM snapshot. That raises `retry_count` on every call, so three imports - the normal shape of a bulk migration done one archive at a time - escalated a healthy peer to `retryStats.failed` with the scheduling note shown as `lastError`, which is exactly the signal the runbook tells operators to repair. A full retry queue also turned a completed import into a 503. Scheduling now only ensures the collapsed entry exists, and a failure to schedule is logged instead of failing the request: the entities are already imported and the reconcile pass still closes the gap. * fix(admin): stop reporting replication failures as retries `retries` is the minio-go counter for redeliveries, and mc prints it as such. Filling it with the failure count claimed a redelivery that never happens: a failed object is not retried by an event today, it waits for the scanner heal pass. `errors` keeps the failure counters; `retries` stays zero until there is a real redelivery to count, and the runbook now says so. * perf(site-replication): aggregate failure windows without cloning bucket stats `site_metrics_snapshot` went through `get_all`, which clones every bucket's stats, and then scanned each target's sample deque twice. That deque is bounded only by the one-hour window, so an unreachable target under load - the case an operator polls this endpoint for - made every `mc admin replicate status` copy the whole backlog and hold the read lock against the failure path while doing it. It now folds under the read lock and takes both windows in one walk. The `max` against the serialized `last_minute` / `last_hour` snapshots is dropped: those are stamped onto per-bucket clones elsewhere and are always zero in this node-local cache. * fix(site-replication): reject a conflicting ilm-expiry override on a re-run The commit now reads the ilm-expiry override back out of the pending refresh journal, so a second edit that asks for a different value had it dropped while the request still reported success. Re-running without the flag keeps pinning the recorded value - that is the documented way to redrive a stuck refresh - but an explicit different value is now rejected instead of ignored. * fix(admin): do not fail a remote-target write on a peer reload error set/remove-remote-target propagated the peer metadata reload error, so a target that was already persisted and live on this node reported a 5xx to the client whenever one peer could not be reached. Every S3 bucket-config write path treats that reload as best effort and only warns; these two admin handlers now do the same, and the reason is logged with the bucket and action. * fix(site-replication): undo every bucket a cut-short refresh rewrote When a remove accepted on another node clears the refresh journal mid-pass, only the bucket holding the lock at that moment had its restored target undone. The buckets rewritten earlier in the same pass kept a target pointing at the removed peer whenever the remove's own cleanup had already walked past them. The undo now covers every bucket this pass rewrote, attempting all of them so one failure does not strand the rest. * fix(site-replication): keep replay running while an endpoint refresh is pending A pending endpoint refresh took the whole heavyweight pass with it, so a peer that never came back froze IAM and bucket replay to every healthy peer too - the stall this journal's resume path was meant to end. The refresh arm now drains the retry queue before returning; it replays per-peer deliveries against the endpoints currently committed in state, so it is unaffected by the edit in flight. Bucket wiring reconciliation still waits, because it rewrites the very targets the refresh is changing, and the runbook now says so. * test(e2e): cover the AND semantics of a two-tag replication filter The acceptance matrix only had a single-tag rule, which matches under both AND and OR semantics and therefore proved nothing about the filter this fix changed. It now also carries a two-tag `And` rule - the shape `mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object with one of the two tags is not admitted while an object with both is. No new test function, so the nightly selection digest is unchanged. * refactor(site-replication): fold the refresh state-change error into one constructor The endpoint-refresh work added three `s3_error!` invocation lines, which the s3s footprint ratchet is meant to prevent. Five copies of the same concurrent-change error now share one constructor, so the surface nets one line smaller than main; the baseline is retightened to match. * fix(site-replication): report a peer whose IAM snapshot waits for a repair An escalated snapshot entry records a deletion a snapshot cannot replay, so only a repair settles it and the marker must survive. Scheduling an import snapshot therefore leaves that peer's entry alone - and now says so, instead of returning success while nothing was scheduled for it. * docs(operations): state the group-status and escalation convergence limits Two boundaries the fixes in this branch make load-bearing: a membership change never carries an enable, so a group disabled on one site only has to be re-enabled there explicitly; and a peer holding an escalated IAM entry does not receive a scheduled snapshot, including the one a bulk import schedules, until a repair settles it.	2026-09-08 14:58:41 +00:00
Zhengchao An	0db77be5c6	fix: attest multi-pool bootstrap per creator and contain lock RPC storms (#7473) ... * fix(ecstore): attest fresh multi-pool bootstrap per pool creator A fresh deployment whose pools have their first endpoint on different nodes could never publish its initial pool.bin: each node held fresh-bootstrap proof only for the pool it formatted, combine_across_pools collapsed the deployment-wide proof to None, the elected writer never wrote a pending identity, and startup died with "no durable bootstrap identity or pool.bin replica is available" once the init retry budget ran out. Track first-hand bootstrap authority per pool. The first pool's creator mints the pending cluster identity on the pool it created, every other creator copies that nonce-bound identity onto the pool it formatted first-hand (a scoped identity write that only ever touches pools the process holds first-hand proof for), and the elected writer publishes pool.bin once it holds first-hand proof for pool 0 and every pool replica carries the same pending identity. Fresh + None is still never promoted, corrupt or disagreeing replicas still fail closed, an initialized deployment never reopens bootstrap for an expansion pool, and an elected restart without first-hand proof still cannot reuse a complete pending set. Startup classification no longer latches the pool-metadata write gate for the two transient outcomes a healthy bootstrap passes through (a non-elected node waiting for the elected writer, the elected writer waiting for the other creators); recover_pool_meta_transaction never clears write_blocked, so a non-elected node that started before pool.bin existed stayed write-blocked for the life of the process. Genuine recovery-required states still latch. Refs rustfs/backlog#2375, rustfs/backlog#2338 * fix(lock): contain remote lock RPC timeout storms A lock RPC deadline evicted the shared internode HTTP/2 channel and re-dialed it unconditionally, so one slow lock endpoint produced a cluster-wide RST_STREAM / GOAWAY too_many_resets / reconnect loop (rustfs#7363). The remote lock client now keeps a per-peer channel history: a timeout evicts only when the peer has completed no lock RPC for two deadlines, evictions and transport-failure re-dials are rate limited per peer (RUSTFS_OBJECT_LOCK_RPC_EVICTION_COOLDOWN_MS, default 5 s), and a timed-out request is detached instead of cancelled, bounded per peer by RUSTFS_OBJECT_LOCK_RPC_DETACHED_LIMIT (default 256) and by the internode RPC timeout; a lock granted after its caller gave up is released immediately. Unlocks that fail the quick retries continue on a deferred 1/2/4/8/16 s schedule before the server lease reclaims them. New rustfs_remote_lock_* metrics cover timeouts, evictions, suppressed evictions, detached streams, late completions and late releases per peer; docs/operations/lock-rpc-storm-protection.md documents the policy. Refs rustfs/backlog#2375, rustfs#7363 * ci: refresh nightly test selection digests The replication nightly membership guard expected the 68-test digest from #7422 while the current listing has 71 tests (additions only: test_bucket_replication_sse_c_compressed_passthrough from #7366, matrix_mint_own_version_ids_addresses_mutations_through_the_ledger and matrix_removed_replication_config_abandons_pending_purge from #7368), and the cluster fault lane expected 50 tests while #7374 added test_cluster_root_heal_recovers_remote_shards_after_background_target_crash. Both lanes have failed before running a single test since 2026-09-07. Bind the Linux digests to the listings from scheduled run 34187469350 and the Darwin e2e-nightly digest to the matching local listing. Refs rustfs/backlog#2375	2026-09-08 17:05:14 +08:00
Zhengchao An	d6bb61d420	fix(s3): reject unsigned x-amz headers on presigned requests (#7425) ... * fix(s3): reject unsigned x-amz headers on presigned requests A SigV4 presigned URL only binds the headers listed in X-Amz-SignedHeaders, but the handlers applied every x-amz-* request header regardless. The holder of a presigned PutObject URL signed with SignedHeaders=host could add x-amz-tagging, x-amz-storage-class, x-amz-website-redirect-location, ACL, metadata, Object Lock or SSE headers and have them applied (GHSA-g8w9-qw9q-fghr). Reject such requests at the S3 access boundary with 403 AccessDenied and the AWS message "There were headers present in the request which were not signed"; x-amz-cf-id stays tolerated for CloudFront. SigV2 and header-signed SigV4 requests are unchanged. Regression tests are named after the advisory (unit tests in rustfs/src/auth.rs, e2e in crates/e2e_test/src/presigned_negative_test.rs with a signed-tagging positive control); the security smoke floor rises to 20 and the e2e selection digests are refreshed for the two new cases. * fix(s3): apply presigned signed-header rule to custom routes and harden parsing Move the GHSA-g8w9-qw9q-fghr check to the first statement of S3Access::check, apply it in S3Router::check_access so admin, console, STS and extension routes that never reach the access hook enforce the same rule, read X-Amz-SignedHeaders with the exact key the verifier uses and treat a duplicate as signing nothing, and log the rejection as a warn event with the repository field shape. Add presigned GET, unsigned x-amz-copy-source and unsigned Content-Type e2e cases plus a router unit test; raise the security smoke floor to 26 and refresh the selection digests. * docs(testing): list the full GHSA-g8w9 regression set * fix(error): merge equivalent api message branches Combine the MaxVersionsExceeded and internal IO message branches so Clippy no longer flags identical if blocks while preserving the existing response messages. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>	2026-09-07 22:46:02 +00:00
Zhengchao An	736b6a366c	fix(storage): bound multipart admission wait below SDK write timeouts (#7408) ... A multipart UploadPart queued for a foreground write permit is not read while it waits, so the client's socket write stalls for the whole wait and the client's own write timeout decides the outcome; botocore reports that as ConnectionClosedError. Lower the default queue wait from 30 s to 10 s so the part receives SlowDown before mainstream SDK timeouts, and stop forcing a 4 MiB SO_RCVBUF on the API listener (new RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES restores a fixed size) so a queued connection no longer lets up to 8 MiB of unread body accumulate in kernel memory. Fixes #7385. Co-authored-by: houseme <housemecn@gmail.com>	2026-09-07 22:25:06 +08:00
Zhengchao An	ea9aa53fd8	docs(odm): record upgrade limits in release notes (#7233)	2026-09-06 02:09:44 +08:00
Zhengchao An	d011ec9952	feat(odm): enable on-demand migration by default (#7089) ... * feat(odm): enable on-demand migration by default The module switch RUSTFS_ON_DEMAND_MIGRATION_ENABLED now defaults to true, so the feature is reachable without an opt-in; setting it to false still keeps the module out of the read path entirely. A bucket without an on-demand-migration.json is never resolved by the runtime and makes no source call, so the flip changes nothing for unconfigured buckets. The admin plane now reads the switch through the predicate published by module_switches.rs instead of its own duplicated env constant; the behaviour (an environment read per call) is unchanged. * test(e2e): wire three on-demand migration cases into e2e-smoke The PR smoke lane gains one case per user-visible contract: a GET miss that pulls and persists, a HEAD miss that answers from the source and stores nothing, and the admin config/status pair that must redact the source secret. The HEAD case did not exist outside the nightly real-source lane, so it is added to get_basic_test. Measured on darwin: the lane goes from 168 tests in 101.98 s to 171 tests in 101.92 s, since the three cases overlap the lane's existing work. The darwin selection digests for e2e-smoke and e2e-full are regenerated; the e2e-full linux digest still needs a Linux runner. * docs(changelog): record the on-demand migration feature	2026-09-03 11:40:54 +08:00
cxymds	b0c6c4cbce	fix(storage): resolve erasure parity per pool (#4977) ... * fix(filemeta): add state-aware file info validation * fix(filemeta): validate shard arithmetic and delete paths * fix(ecstore): add fallible erasure construction * fix(ecstore): resolve storage parity per pool * fix(storage): report heterogeneous erasure layouts * fix(admin): publish prepared storage config atomically * fix(storage): harden per-pool parity boundaries * fix(storage): address pre-PR validation findings * test(ci): fix strict-topology validation fixtures * fix(heal): preserve delete markers during repair * refactor(filemeta): drop unused ValidatedFileInfo witness ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime. No behavior change. * fix(filemeta): keep compressed and MinIO-migrated tiered objects readable The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt: - Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked. - MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read. Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery. * fix(scanner): build the delete-marker test fixture without erasure geometry get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2. Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker. * docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit. * fix(heal): report parity from erasure geometry, not is_valid() heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change. * fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object `From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`. * test(filemeta): assert real historical xl.meta versions pass metadata-read validation Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt. Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red. * fix(ci): remove duplicate storage test re-exports --------- Co-authored-by: overtrue <anzhengchao@gmail.com>	2026-07-19 21:52:31 +08:00
escapecode	a80699b6dd	feat: add an opt-in NATS JetStream publish path for the notify and audit targets (#4634) ... feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets The NATS notify and audit targets publish through NATS Core, which returns before the server has durably accepted the message. A broker restart or a connection drop between the publish and the flush loses the event, even though the send queue has already cleared it, and no acknowledgement gates that clear. An opt-in JetStream publish path clears a queued event only after the server returns a durable PublishAck, so delivery is at-least-once across a broker restart or a reconnect. It applies to both the notify and audit NATS targets, is off by default, and is byte-identical to the NATS Core path when disabled. The path includes durable store-and-forward, a stable dedup id sent as the Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate window, pre-flight stream validation, and a bounded failed-events store for terminally-failed and retry-exhausted events. Three configuration keys per target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_ prefixes. The on-disk batch filename separator changes from colon to underscore so batch names are valid on Windows filesystems, with transparent read-back of files written under the previous separator. The migration affects the shared queue store for every target type and lands with this feature because the store gains its first Windows-exercised paths here. Co-authored-by: houseme <housemecn@gmail.com>	2026-07-14 15:36:14 +08:00
escapecode	b6973636b6	docs(sftp): document server operations (#3391) ... Adds an operator guide for the SFTP server: recommended configuration, the path model, host keys on Unix and Windows, the environment variable reference, session cleanup behaviour, IAM policy requirements per SFTP operation, client compatibility notes, multipart upload sizing and cleanup, and the log lines worth alerting on. Corrects documentation the platform change left stale. The module overview and the UnsupportedPlatform error text still described Windows as unsupported. A source comment referenced a document that does not exist in the repository and now points at the new guide. The changelog adds the two host-key reload variables missing from its environment list, describes the banner variable as the SSH identification string, and corrects the upload size cap and compliance case count.	2026-06-12 22:52:00 +08:00
escapecode	7a8514bdfa	feat(sftp): add macOS and Windows platform support (#3372) ... The session watchdog now selects its detection method per platform. It previously probed kernel TCP state through a Linux-only procfs path, so on macOS every healthy idle session was killed within a minute, and on Windows the watchdog never spawned at all, leaving wedged sessions with no cleanup. Linux keeps its fast-kill watchdog unchanged. Other platforms get a silence-only backstop that kills a session only at the documented 30-minute idle ceiling. The host-key loader now has a Windows arm. It loads OpenSSH format host keys from the configured directory and logs a one-time warning to restrict NTFS ACLs on the key directory, the same operator-managed approach FTPS, WebDAV, KMS, and IAM already use on Windows. Startup previously aborted with UnsupportedPlatform because the Unix mode-bit permission check has no Windows equivalent. tokio's io-uring feature is now enabled only in Linux builds. io-uring is a Linux kernel interface and enabling it unconditionally broke the Windows build. Co-authored-by: houseme <housemecn@gmail.com>	2026-06-12 18:08:23 +08:00
escapecode	96b293bf8a	feat(sftp): add SFTPv3 protocol support (#2875) ... Co-authored-by: houseme <housemecn@gmail.com>	2026-05-10 03:48:42 +00:00
Philip Schmid	9179fd5608	fix(helm): merge customAnnotations with class-specific ingress annotations (#2161) ... Signed-off-by: Philip Schmid <philip.schmid@protonmail.com> Co-authored-by: houseme <housemecn@gmail.com>	2026-03-15 09:22:12 +08:00
Senol Colak	b69183aadf	Openstack Keystone integration - v1 keeps the same mechanism as (#1961) ... Co-authored-by: loverustfs <hello@rustfs.com> Co-authored-by: 安正超 <anzhengchao@gmail.com>	2026-02-27 22:23:35 +08:00