gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-09 05:36:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
b068fe3f6cf77ea3ee54eb40c802da40ee7b910b
rustfs/CHANGELOG.md
T
唐小鸭 46907c05cf fix(replication): close the GA blocker set from backlog#2366 (#7503) 
* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
2026-09-08 14:58:41 +00:00

28 KiB
Raw Blame History

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Replication

  • Object Lock replication PUTs now carry a required integrity header, fixing target rejection introduced by the plain-payload default (#7097). This changes the default outbound request for locked objects but adds no persisted format.
  • Multipart source objects stay on the multipart transport even when their checksum record is a whole-object checksum, so objects above the single-PUT limit remain replicable (#7047).
  • Targets that mint their own version IDs now use a per-target version ledger for tag, retention, legal-hold, and permanent-delete mutations; ambiguous pre-ledger matches fail with backoff instead of guessing (#7368). This adds dual-prefixed internal metadata keys that older readers ignore.
  • Single-part source checksums are forwarded as x-amz-checksum-* headers instead of user metadata, so the replica preserves checksum responses (#7313). This changes the default outbound headers for checksummed objects.
  • Site-replication outage recovery now uses a bounded 30-second retry drain plus the 600-second full reconciliation pass, persists destructive liabilities before local deletion, and fences replay settlement and peer edits (#7148). Persisted additions are optional and ignored by older readers.
  • IAM snapshot/deletion replay, target-assigned delete-marker purges, timestamp ordering, and best-effort peer broadcast now close the control-plane gaps found by the R6 review (#7195).
  • Upgrade and rollback: upgrade every node in one site consecutively and verify reconciliation before moving to the next site; do not intentionally run a site mixed-version. Target-version ledger keys are harmless on rollback, although old code cannot use their routing. Before rolling back past #7307, drain or repair every pending version purge: older code can free a retained version's data directory before its remote purge is acknowledged. See docs/operations/site-replication-operations.md.

Security

  • Presigned URLs honour only signed headers (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an x-amz-* request header not listed in X-Amz-SignedHeaders is now rejected with 403 AccessDenied ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned PutObject URL could add unsigned x-amz-tagging, x-amz-storage-class, x-amz-website-redirect-location, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in SignedHeaders; x-amz-cf-id (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.

Fixed

  • Fresh multi-pool bootstrap with distinct format creators: a new deployment whose pools have their first endpoint on different nodes (for example two single-node pools) could never publish its initial pool.bin: each node held fresh-bootstrap proof only for the pool it formatted, the deployment-wide proof collapsed to none, and every node died with pool metadata recovery required: no durable bootstrap identity or pool.bin replica is available after the startup retry budget. The first pool's creator now mints the pending cluster identity on its own pool, every other creator copies that nonce-bound identity onto the pool it formatted first-hand, and the elected writer publishes pool.bin once every pool replica carries the same pending identity. Corrupt or disagreeing replicas, pools that merely have a format, expansion pools joining an initialized deployment, and restarts without first-hand proof still fail closed. Non-elected nodes that start before pool.bin exists, and the elected writer while it waits for the other creators, no longer latch their pool-metadata write gate for the life of the process. Refs rustfs/backlog#2338, rustfs/backlog#2375.
  • Lock RPC timeout storms (#7363): the remote lock client no longer evicts and re-dials the shared internode HTTP/2 channel on every request deadline. A timeout evicts only when the peer has not completed any lock RPC for two deadlines, evictions and transport-failure re-dials are rate limited per peer (RUSTFS_OBJECT_LOCK_RPC_EVICTION_COOLDOWN_MS, default 5 s), and a timed-out request is left running instead of being reset (bounded per peer by RUSTFS_OBJECT_LOCK_RPC_DETACHED_LIMIT, default 256), so a slow lock endpoint can no longer drive the RST_STREAM/GOAWAY too_many_resets/reconnect loop. A lock granted after its caller timed out is released immediately, and unlocks that fail the quick retries continue on a deferred 1/2/4/8/16 s schedule before the server lease reclaims them. New rustfs_remote_lock_* metrics cover timeouts, evictions, suppressed evictions, detached streams, late completions and late releases per peer. Operator guide at docs/operations/lock-rpc-storm-protection.md.
  • Multipart admission queue: an UploadPart waiting for a foreground write permit now waits at most 10 s by default (RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS, previously 30 s), so a queued part returns S3 SlowDown before the client's socket write timeout drops the connection. Separately, the API listener no longer forces a 4 MiB SO_RCVBUF on every accepted socket (kernel autotuning applies; RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES restores a fixed size), so a queued part no longer lets up to 8 MiB of unread body accumulate in kernel memory per connection, which is what throttled whole nodes under SDK-default multipart concurrency. Fixes #7385.
  • Helm Ingress: customAnnotations are now merged with class-specific annotations (nginx/traefik) instead of being ignored when ingress.className is set.
  • Per-pool erasure parity: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example 2+2 in the 4-drive pool and 1+1 in the 2-drive pool). Fixes #4801.

Added

  • On-Demand Migration: Lazy, pull-style migration of an existing S3-compatible bucket into RustFS. A local bucket is attached to an external source bucket; a GET for a key that does not exist locally fetches it from the source, streams it to the client, and stores it locally in the same pass, so every later read is served locally. The module is on by default; set RUSTFS_ON_DEMAND_MIGRATION_ENABLED=false on every node to turn it off. A bucket with no source configured behaves exactly as before — the runtime never intervenes on its reads and makes no outbound call. Operator guide at docs/operations/on-demand-migration.md.
    • Per-bucket configuration persisted as on-demand-migration.json in the bucket metadata: source provider (s3, aws, minio, rustfs, r2, gcs), endpoint, region, addressing style, credentials and TLS material, an optional key-prefix filter and source-prefix rewrite, and a policy block covering the inline size threshold, multipart part size, concurrency, queue capacity, timeouts, bandwidth limit and negative-cache TTL
    • Admin routes under /rustfs/admin/v3/on-demand-migration/{bucket}: PUT (with ?dry-run=true to validate and probe the source without saving), GET, DELETE, GET .../status, plus POST .../backfill?op=start|cancel and GET .../backfill for the background full-backfill job with its resumable checkpoint. Authorized by the new admin:GetBucketOnDemandMigration and admin:SetBucketOnDemandMigration actions; every response redacts secret_key and session_token
    • Read paths: an object at or below policy.inline_max_bytes (16 MiB by default) is teed to the client and to the local store in a single source read; a larger object or a Range read streams through and a background pull stores the whole object. A HEAD miss is proxied to the source and stores nothing (policy.head = local_only disables it). Every source-backed response carries x-rustfs-on-demand-migration: source
    • Protections: a per-source circuit breaker, a per-key negative cache, singleflight per key, a concurrency limit and a bounded pull queue shared by the inline and background paths, an optional bandwidth limit, an anti-loop request marker, and the shared outbound-endpoint (SSRF) policy
    • Metrics under rustfs_on_demand_migration_* (requests_total, pulled_bytes_total, pulled_objects_total, pull_failures_total, inflight_pulls, queue_depth, source_latency_seconds_*, breaker_state), mirrored per node by the admin status route
    • Listings: ListObjects v1 remains local with ordinary key markers. ListObjectsV2 can merge source objects when policy.list_through = true; this is off by default
    • Upgrade and rollback: finish upgrading every node before enabling ODM. An rc.5 node that writes bucket configuration drops the ODM fields from metadata; neither a later restart nor moving the service out of ECStore recovers them. Before rollback, disable ODM and securely retain the original full configuration and credentials. After every node returns to a compatible version, restore and validate that configuration. Redacted exports cannot replace the credential backup; source-only objects are unavailable through RustFS while ODM is disabled. See the upgrade and rollback section of docs/operations/on-demand-migration.md
    • Optional Google dependencies: default and full server builds retain native GCS support. cargo build -p rustfs --no-default-features --features ftps,webdav excludes Google SDKs while preserving configuration decoding and redaction; native GCS ODM and tier operations require the gcs feature. Do not use that build with existing GCS-tiered data
    • Limitations: PUT and DELETE never reach the source; a source object updated after it was pulled is not re-fetched; SSE-C source objects are unsupported and answer 424; Last-Modified on a pulled object is the local write time, with the source timestamp kept in metadata
  • NATS JetStream Publish Path: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream PublishAck, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled.
    • Three configuration keys per target: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_ prefixes
    • Durable store-and-forward with a stable dedup id sent as the Nats-Msg-Id header, so a replay after a crash is collapsed by the server duplicate window
    • Pre-flight stream validation, and a bounded failed-events store (count and TTL). Only a non-retryable rejection is recorded in the failed-events store. A retryable condition keeps the entry on the live queue until it is delivered
    • Operator guide at docs/operations/nats-jetstream.md
  • OpenStack Keystone Authentication Integration: Full support for OpenStack Keystone authentication via X-Auth-Token headers
    • Tower-based middleware (KeystoneAuthLayer) self-contained within rustfs-keystone crate
    • Task-local storage for async-safe credential passing between middleware and auth handlers
    • Automatic detection of Keystone credentials (access keys prefixed with keystone:)
    • Role-based permission mapping (admin/reseller_admin roles grant owner permissions)
    • Token caching for high-performance validation with configurable cache size and TTL
    • Dual authentication support: Keystone and standard AWS Signature v4 work simultaneously
    • Immediate 401 response for invalid tokens (no fallback to local auth)
    • XML-formatted error responses compatible with S3 API
    • Comprehensive integration documentation with manual testing guide
    • 32 unit and integration tests covering middleware, auth handlers, task-local storage, and role detection
  • SFTPv3 Protocol Support: SSH-hosted SFTPv3 subsystem that translates each file operation into S3 calls against the local object store. Authentication uses IAM credentials (SSH username = access key, SSH password = secret key).
    • Full SFTPv3 packet coverage: open, read, write, stat, lstat, fstat, mkdir, rmdir, rename, remove, opendir, readdir, realpath, close, plus the rest of the 21-packet specification
    • Streaming multipart write up to the part size times 10000 parts (156.25 GiB at the default part size)
    • Per-handle read-ahead cache with configurable window size and process-wide memory ceiling
    • Per-session liveness watchdog: Linux probes /proc/net/tcp and cancels wedged sessions on the order of 45 seconds; non-Linux falls back to an inactivity ceiling on the order of 30 minutes
    • 30-second SSH handshake deadline, per-call backend operation timeout, bounded multipart-abort fan-out, graceful-shutdown cascade
    • 34 SFTPv3 compliance test cases under crates/e2e_test/src/protocols/sftp_compliance.rs spread across three entry points: test_sftp_compliance_suite (shared session), test_sftp_compliance_readonly (read-only mode), and test_sftp_compliance_standalone (one rustfs spawn per case)
    • Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module Protocol enum assertion, end-to-end SSH banner test against the running binary

Changed

  • HTTP Server Stack: Integrated KeystoneAuthLayer middleware from rustfs-keystone crate into service stack (positioned after ReadinessGateLayer)
  • Storage-class validation on startup (upgrade note): A persisted explicit storage class (RUSTFS_STORAGE_CLASS_STANDARD / RUSTFS_STORAGE_CLASS_RRS, for example EC:2) is now validated against the actual per-pool drive counts at startup and rejected when a pool cannot satisfy it. This is fail-closed and correct, but a cluster that persisted a storage class larger than a small or heterogeneous pool can hold (for example EC:2 alongside a 2-drive pool), which earlier releases accepted and silently resolved to an invalid layout, will now refuse to start after upgrade. To recover, unset RUSTFS_STORAGE_CLASS_STANDARD so the server derives a valid per-pool default automatically, or set it to a value every pool can satisfy.
  • IAMAuth: Enhanced get_secret_key() to return empty secret for Keystone credentials (bypasses signature validation)
  • Auth Module: Modified check_key_valid() to retrieve Keystone credentials from task-local storage and determine admin status
  • StorageBackend trait: extended with multipart upload methods (create_multipart_upload, upload_part, complete_multipart_upload, abort_multipart_upload) plus upload_part_copy. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.
  • Protocol enum: new Protocol::Sftp variant with corresponding S3Action mappings. Every match arm on Protocol updated to handle the new variant exhaustively.

Technical Details

  • Middleware is self-contained in rustfs-keystone crate following the trusted-proxies pattern for integration-specific middleware
  • Uses BoxBody pattern for Hyper 1.x compatibility
  • Task-local storage provides request-scoped credential passing without modifying HTTP request/response types
  • Integration preserves existing S3 authentication flow while adding Keystone support
  • Zero breaking changes to existing functionality
  • No new top-level directories in main binary crate (middleware lives in integration crate)
  • SSH/SFTP wire handling via the russh and russh-sftp crates. SFTPv3 framing is implemented by russh-sftp; the rustfs-side SftpDriver implements russh_sftp::server::Handler and dispatches to the storage backend
  • Drop-time abort for in-flight multipart uploads honours IAM Deny on AbortMultipartUpload. start_multipart_upload caches the authorisation decision so the synchronous Drop path can honour Allow / Deny policies without re-querying IAM
  • Per-handle read cache uses an Arc<AtomicU64> shared across every SftpDriver instance to enforce a process-wide memory ceiling. On ceiling breach the populate is skipped and the read serves correctly via a single-call backend fetch
  • Per-session liveness watchdog runs as a tokio task per accepted connection. Reads /proc/net/tcp and /proc/net/tcp6 to look up the (local, peer) tuple's TCP state and cancels via tokio_util::sync::CancellationToken when wedge conditions are confirmed across two consecutive ticks
  • Path canonicalisation rejects paths containing \0, \r, or \n and resolves traversal via path::clean() before any backend dispatch
  • Cipher / KEX / MAC / host-key algorithm allowlists are hardcoded with no environment override. Strict-KEX (CVE-2023-48795 / Terrapin) marker presence asserted by unit test
  • Per-session handle cap (default 64, configurable 8 to 1024) with UUID-generated handle ids
  • Crate-level #![deny(unsafe_code)] is in force across crates/protocols. Socket fd duplication for the watchdog uses the safe AsFd::try_clone_to_owned path (Linux). Non-Linux targets use the inactivity-ceiling watchdog
  • Platform-specific imports are cfg-gated. Unix enforces owner-only host-key mode bits (no group or other permission bits). Windows loads host keys without a mode check and trusts operator-managed NTFS ACLs. Targets that are neither Unix nor Windows fail SFTP at config-load with SftpInitError::UnsupportedPlatform

Documentation

  • Updated crates/keystone/README.md with complete integration architecture and workflow
  • Added detailed manual testing guide with 10 test scenarios
  • Updated main README.md to list Keystone authentication as available feature
  • Added troubleshooting section for common integration issues
  • Module-level rustdoc on crates/protocols/src/sftp/mod.rs describing the public API surface, configuration contract, and the architecture of the read cache and the wedge watchdog

Configuration

New environment variables:

  • RUSTFS_KEYSTONE_ENABLE - Enable/disable Keystone authentication (default: false)
  • RUSTFS_KEYSTONE_AUTH_URL - Keystone API endpoint URL
  • RUSTFS_KEYSTONE_VERSION - Keystone API version (v3)
  • RUSTFS_KEYSTONE_ADMIN_USER - Admin username for privileged operations
  • RUSTFS_KEYSTONE_ADMIN_PASSWORD - Admin password
  • RUSTFS_KEYSTONE_ADMIN_PROJECT - Admin project name
  • RUSTFS_KEYSTONE_ADMIN_DOMAIN - Admin domain name (default: Default)
  • RUSTFS_KEYSTONE_CACHE_SIZE - Token cache size (default: 10000)
  • RUSTFS_KEYSTONE_CACHE_TTL - Token cache TTL in seconds (default: 300)
  • RUSTFS_KEYSTONE_VERIFY_SSL - Verify SSL certificates (default: true)
  • RUSTFS_SFTP_ENABLE - Enable/disable SFTP (default: false)
  • RUSTFS_SFTP_ADDRESS - Listen address (default: 0.0.0.0:2222)
  • RUSTFS_SFTP_HOST_KEY_DIR - Directory containing host key files (must exist). On Unix each file must grant no group or other permission bits (owner access only). On Windows the files load without a mode check and rustfs trusts the directory NTFS ACL
  • RUSTFS_SFTP_HOST_KEY_RELOAD_ENABLE - Rescan the host-key directory without a restart (default: false)
  • RUSTFS_SFTP_HOST_KEY_RELOAD_INTERVAL - Host-key rescan interval in seconds, minimum 5 (default: 30)
  • RUSTFS_SFTP_IDLE_TIMEOUT - Session idle timeout in seconds (default: 600)
  • RUSTFS_SFTP_PART_SIZE - Multipart part size in bytes (default: 16 MiB)
  • RUSTFS_SFTP_READ_ONLY - Reject write packets at the protocol layer (default: false)
  • RUSTFS_SFTP_BANNER - SSH protocol identification string, must begin with SSH-2.0- (default: SSH-2.0-RustFS)
  • RUSTFS_SFTP_HANDLES_PER_SESSION - Per-session open-handle cap, 8 to 1024 (default: 64)
  • RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS - Per-call backend deadline in seconds, 5 to 600 (default: 60)
  • RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES - Per-handle read-cache window in bytes, 256 KiB to 64 MiB or 0 to disable (default: 4 MiB)
  • RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES - Process-wide read-cache memory ceiling in bytes, 16 MiB minimum (default: 256 MiB)

Files Added

  • crates/protocols/src/sftp/mod.rs - SFTP module entry point, public API surface, crate-level rustdoc, regression-prevention test
  • crates/protocols/src/sftp/config.rs - SftpConfig and SftpInitError types, env-var resolvers, host-key directory loader with permission enforcement
  • crates/protocols/src/sftp/constants.rs - Named constants grouped by purpose: S3 error codes, HTTP error codes, POSIX mode bits, protocol identifiers, operational limits
  • crates/protocols/src/sftp/server.rs - SftpServer SSH server, russh handler, password authentication against IAM, accept loop, per-session task spawn
  • crates/protocols/src/sftp/driver.rs - SftpDriver per-session SFTPv3 handler dispatching each operation onto the StorageBackend
  • crates/protocols/src/sftp/state.rs - HandleState variants for read, write-buffering, write-streaming, write-failed handles
  • crates/protocols/src/sftp/lifecycle.rs - Per-session activity stamp, weak-ref registry, /proc/net/tcp probe for the wedge watchdog
  • crates/protocols/src/sftp/wedge_watchdog.rs - Per-session liveness watchdog cancelling sessions silent at the SFTP layer while the kernel reports CLOSE_WAIT
  • crates/protocols/src/sftp/fallback_watchdog.rs - Per-session silence-only liveness backstop for non-Linux targets, cancelling sessions only at the fallback idle ceiling
  • crates/protocols/src/sftp/read_cache.rs - Per-handle in-memory read-ahead cache with shared atomic accumulator for the process-wide memory ceiling
  • crates/protocols/src/sftp/attrs.rs - SFTPv3 FileAttributes mapping for objects and directories, longname formatting, mtime clamping
  • crates/protocols/src/sftp/dir.rs - OPENDIR / READDIR pagination, root-bucket listing, sub-directory listing under a prefix
  • crates/protocols/src/sftp/errors.rs - SftpError thiserror enum and S3-error classification into SFTPv3 status codes
  • crates/protocols/src/sftp/paths.rs - Path canonicalisation, traversal rejection, \0 / \r / \n rejection, bucket+key decomposition
  • crates/protocols/src/sftp/read.rs - READ packet handler, EOF semantics, MAX_READ_LEN bound, integration with the read cache
  • crates/protocols/src/sftp/write.rs - WRITE packet handler, in-memory buffering up to part size, transition to streaming multipart, CLOSE finalisation
  • crates/protocols/src/sftp/test_support.rs - Test fixtures and helper builders for SFTP unit tests
  • crates/protocols/src/common/dummy_storage.rs - In-memory StorageBackend test backend covering every method, used by SFTP unit tests and the FTPS / Swift / WebDAV test suites
  • crates/e2e_test/src/protocols/sftp_core.rs - End-to-end regressions for the handshake deadline, idle-timeout disconnect, and the wedge watchdog
  • crates/e2e_test/src/protocols/sftp_compliance.rs - SFTPv3 compliance suite entry points (test_sftp_compliance_suite, test_sftp_compliance_readonly, test_sftp_compliance_standalone)
  • crates/e2e_test/src/protocols/sftp_compliance_tests.rs - Per-case test bodies (CMPTST-01..34), shared fixture helpers, lifecycle counters
  • crates/e2e_test/src/protocols/sftp_helpers.rs - SFTP-specific test helpers and fixture seeders

Files Modified

  • crates/keystone/src/middleware.rs - Created Keystone authentication middleware (self-contained in keystone crate)
  • crates/keystone/src/lib.rs - Exported middleware module and KEYSTONE_CREDENTIALS
  • crates/keystone/Cargo.toml - Added Tower/HTTP dependencies for middleware functionality
  • rustfs/src/server/http.rs - Integrated KeystoneAuthLayer from rustfs-keystone crate
  • rustfs/src/auth.rs - Enhanced IAMAuth and check_key_valid for Keystone support, imported KEYSTONE_CREDENTIALS from rustfs-keystone
  • crates/keystone/README.md - Comprehensive integration documentation
  • README.md - Added Keystone as available feature
  • Cargo.toml - Added the sftp feature alongside the existing protocol features
  • Cargo.lock - Updated to include the new russh, russh-sftp, socket2, tokio-util, subtle, uuid dependencies and their transitive crates
  • crates/protocols/Cargo.toml - Declared russh, russh-sftp, socket2, tokio-util, subtle, uuid under the sftp feature flag
  • crates/protocols/src/lib.rs - Added pub mod sftp behind #[cfg(feature = "sftp")] plus the crate-level #![deny(unsafe_code)] lint
  • crates/protocols/src/common/client/s3.rs - Extended the StorageBackend trait with create_multipart_upload, upload_part, complete_multipart_upload, abort_multipart_upload, and upload_part_copy
  • crates/protocols/src/common/session.rs - Added the Protocol::Sftp variant and its S3Action mappings
  • crates/protocols/src/common/gateway.rs - Handles the new Protocol::Sftp variant exhaustively
  • crates/protocols/src/common/mod.rs - Exposed the new dummy_storage module
  • crates/protocols/src/constants.rs - Added shared POSIX mode-bit constants used by SFTP and other protocols
  • crates/config/src/constants/protocols.rs - RUSTFS_SFTP_* environment variable names and defaults
  • crates/utils/src/retry.rs - Added the generic exponential-backoff retry helper used by the SFTP write path
  • crates/e2e_test/Cargo.toml - Added the e2e test dependencies for SFTP (paramiko fixture, SSH keypair generation)
  • crates/e2e_test/src/protocols/mod.rs - Registered the new sftp_core, sftp_compliance, sftp_compliance_tests, and sftp_helpers modules
  • crates/e2e_test/src/protocols/README.md - Documented the SFTP test entry points and case index
  • crates/e2e_test/src/protocols/test_env.rs - Added SFTP host-key directory provisioning to the shared protocol test environment
  • crates/e2e_test/src/protocols/test_runner.rs - Wired the SFTP entry points into the runner
  • rustfs/Cargo.toml - Added the sftp feature flag
  • rustfs/src/lib.rs - One-line addition exporting the SFTP wiring
  • rustfs/src/init.rs - Build and start the SftpServer when RUSTFS_SFTP_ENABLE is true
  • rustfs/src/main.rs - Routed shutdown signals to the SFTP server alongside the other protocols
  • rustfs/src/protocols/client.rs - Client-builder support for the new Protocol::Sftp variant

Testing

  • 16 unit tests in rustfs-keystone crate (config, auth, middleware, identity)
  • 10 integration tests in rustfs-keystone crate (task-local storage, middleware layer, scope isolation)
  • 6 auth unit tests in rustfs crate (role detection, task-local storage, Keystone credential handling)
  • Total: 32 tests passing with zero compilation errors
  • Manual testing guide provided for end-to-end validation
  • All Keystone tests passing with cargo test --all --exclude e2e_test
  • 34 SFTPv3 compliance test cases (CMPTST-01..34) split across three entry points: test_sftp_compliance_suite (shared session, cases 01-14), test_sftp_compliance_readonly (read-only mode, cases 15-23), test_sftp_compliance_standalone (one rustfs spawn per case, cases 24-34)
  • Regression-prevention tests at four layers: compile-time module assertion in crates/protocols/src/lib.rs, module-presence unit test in crates/protocols/src/sftp/mod.rs, cross-module Protocol enum assertion, and end-to-end SSH banner test against the running binary
  • Standalone end-to-end regressions for the SSH handshake deadline, the idle-timeout disconnect path, and the wedge watchdog (Linux fast-kill and the cross-platform fallback path)
  • Inline unit tests in every SFTP source file covering pure helpers (path canonicalisation, attribute mapping, S3-error classification, env-var bound resolvers)
  • Strict-KEX (CVE-2023-48795) marker presence assertion as a unit test in crates/protocols/src/sftp/server.rs
  • All tests passing with cargo test --all --features sftp against a 64-bit Linux target

Previous Releases

See GitHub Releases for previous version history.

Reference in New Issue View Git Blame Copy Permalink