From b6973636b6b7a7807a9cec618b08975588a20930 Mon Sep 17 00:00:00 2001 From: escapecode <148354978+simon-escapecode@users.noreply.github.com> Date: Fri, 12 Jun 2026 15:52:00 +0100 Subject: [PATCH] docs(sftp): document server operations (#3391) Adds an operator guide for the SFTP server: recommended configuration, the path model, host keys on Unix and Windows, the environment variable reference, session cleanup behaviour, IAM policy requirements per SFTP operation, client compatibility notes, multipart upload sizing and cleanup, and the log lines worth alerting on. Corrects documentation the platform change left stale. The module overview and the UnsupportedPlatform error text still described Windows as unsupported. A source comment referenced a document that does not exist in the repository and now points at the new guide. The changelog adds the two host-key reload variables missing from its environment list, describes the banner variable as the SSH identification string, and corrects the upload size cap and compliance case count. --- CHANGELOG.md | 13 +- crates/protocols/src/sftp/config.rs | 12 +- crates/protocols/src/sftp/driver.rs | 6 +- crates/protocols/src/sftp/mod.rs | 6 +- docs/operations/sftp.md | 313 ++++++++++++++++++++++++++++ 5 files changed, 334 insertions(+), 16 deletions(-) create mode 100644 docs/operations/sftp.md diff --git a/CHANGELOG.md b/CHANGELOG.md index f06ffa1d8..b1cfeec37 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,11 +24,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **32 unit and integration tests** covering middleware, auth handlers, task-local storage, and role detection - **SFTPv3 Protocol Support**: SSH-hosted SFTPv3 subsystem that translates each file operation into S3 calls against the local object store. Authentication uses IAM credentials (SSH username = access key, SSH password = secret key). - Full SFTPv3 packet coverage: open, read, write, stat, lstat, fstat, mkdir, rmdir, rename, remove, opendir, readdir, realpath, close, plus the rest of the 21-packet specification - - Streaming multipart write up to S3's 5 TiB per-file ceiling + - Streaming multipart write up to the part size times 10000 parts (156.25 GiB at the default part size) - Per-handle read-ahead cache with configurable window size and process-wide memory ceiling - Per-session liveness watchdog: Linux probes `/proc/net/tcp` and cancels wedged sessions on the order of 45 seconds; non-Linux falls back to an inactivity ceiling on the order of 30 minutes - 30-second SSH handshake deadline, per-call backend operation timeout, bounded multipart-abort fan-out, graceful-shutdown cascade - - 33 SFTPv3 compliance test cases under `crates/e2e_test/src/protocols/sftp_compliance.rs` spread across three entry points: `test_sftp_compliance_suite` (shared session), `test_sftp_compliance_readonly` (read-only mode), and `test_sftp_compliance_standalone` (one rustfs spawn per case) + - 34 SFTPv3 compliance test cases under `crates/e2e_test/src/protocols/sftp_compliance.rs` spread across three entry points: `test_sftp_compliance_suite` (shared session), `test_sftp_compliance_readonly` (read-only mode), and `test_sftp_compliance_standalone` (one rustfs spawn per case) - Four-layer regression-prevention tests guard against silent feature deletion: compile-time module assertion, module-presence unit test, cross-module `Protocol` enum assertion, end-to-end SSH banner test against the running binary ### Changed @@ -77,10 +77,12 @@ New environment variables: - `RUSTFS_SFTP_ENABLE` - Enable/disable SFTP (default: false) - `RUSTFS_SFTP_ADDRESS` - Listen address (default: 0.0.0.0:2222) - `RUSTFS_SFTP_HOST_KEY_DIR` - Directory containing host key files (must exist). On Unix each file must grant no group or other permission bits (owner access only). On Windows the files load without a mode check and rustfs trusts the directory NTFS ACL +- `RUSTFS_SFTP_HOST_KEY_RELOAD_ENABLE` - Rescan the host-key directory without a restart (default: false) +- `RUSTFS_SFTP_HOST_KEY_RELOAD_INTERVAL` - Host-key rescan interval in seconds, minimum 5 (default: 30) - `RUSTFS_SFTP_IDLE_TIMEOUT` - Session idle timeout in seconds (default: 600) - `RUSTFS_SFTP_PART_SIZE` - Multipart part size in bytes (default: 16 MiB) - `RUSTFS_SFTP_READ_ONLY` - Reject write packets at the protocol layer (default: false) -- `RUSTFS_SFTP_BANNER` - Optional SSH banner text +- `RUSTFS_SFTP_BANNER` - SSH protocol identification string, must begin with `SSH-2.0-` (default: `SSH-2.0-RustFS`) - `RUSTFS_SFTP_HANDLES_PER_SESSION` - Per-session open-handle cap, 8 to 1024 (default: 64) - `RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS` - Per-call backend deadline in seconds, 5 to 600 (default: 60) - `RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES` - Per-handle read-cache window in bytes, 256 KiB to 64 MiB or 0 to disable (default: 4 MiB) @@ -95,6 +97,7 @@ New environment variables: - `crates/protocols/src/sftp/state.rs` - `HandleState` variants for read, write-buffering, write-streaming, write-failed handles - `crates/protocols/src/sftp/lifecycle.rs` - Per-session activity stamp, weak-ref registry, `/proc/net/tcp` probe for the wedge watchdog - `crates/protocols/src/sftp/wedge_watchdog.rs` - Per-session liveness watchdog cancelling sessions silent at the SFTP layer while the kernel reports CLOSE_WAIT +- `crates/protocols/src/sftp/fallback_watchdog.rs` - Per-session silence-only liveness backstop for non-Linux targets, cancelling sessions only at the fallback idle ceiling - `crates/protocols/src/sftp/read_cache.rs` - Per-handle in-memory read-ahead cache with shared atomic accumulator for the process-wide memory ceiling - `crates/protocols/src/sftp/attrs.rs` - SFTPv3 `FileAttributes` mapping for objects and directories, longname formatting, mtime clamping - `crates/protocols/src/sftp/dir.rs` - OPENDIR / READDIR pagination, root-bucket listing, sub-directory listing under a prefix @@ -106,7 +109,7 @@ New environment variables: - `crates/protocols/src/common/dummy_storage.rs` - In-memory `StorageBackend` test backend covering every method, used by SFTP unit tests and the FTPS / Swift / WebDAV test suites - `crates/e2e_test/src/protocols/sftp_core.rs` - End-to-end regressions for the handshake deadline, idle-timeout disconnect, and the wedge watchdog - `crates/e2e_test/src/protocols/sftp_compliance.rs` - SFTPv3 compliance suite entry points (`test_sftp_compliance_suite`, `test_sftp_compliance_readonly`, `test_sftp_compliance_standalone`) -- `crates/e2e_test/src/protocols/sftp_compliance_tests.rs` - Per-case test bodies (CMPTST-01..33), shared fixture helpers, lifecycle counters +- `crates/e2e_test/src/protocols/sftp_compliance_tests.rs` - Per-case test bodies (CMPTST-01..34), shared fixture helpers, lifecycle counters - `crates/e2e_test/src/protocols/sftp_helpers.rs` - SFTP-specific test helpers and fixture seeders ### Files Modified @@ -146,7 +149,7 @@ New environment variables: - **Total: 32 tests** passing with zero compilation errors - Manual testing guide provided for end-to-end validation - All Keystone tests passing with `cargo test --all --exclude e2e_test` -- 33 SFTPv3 compliance test cases (CMPTST-01..33) split across three entry points: `test_sftp_compliance_suite` (shared session, cases 01-14), `test_sftp_compliance_readonly` (read-only mode, cases 15-23), `test_sftp_compliance_standalone` (one rustfs spawn per case, cases 24-33) +- 34 SFTPv3 compliance test cases (CMPTST-01..34) split across three entry points: `test_sftp_compliance_suite` (shared session, cases 01-14), `test_sftp_compliance_readonly` (read-only mode, cases 15-23), `test_sftp_compliance_standalone` (one rustfs spawn per case, cases 24-34) - Regression-prevention tests at four layers: compile-time module assertion in `crates/protocols/src/lib.rs`, module-presence unit test in `crates/protocols/src/sftp/mod.rs`, cross-module `Protocol` enum assertion, and end-to-end SSH banner test against the running binary - Standalone end-to-end regressions for the SSH handshake deadline, the idle-timeout disconnect path, and the wedge watchdog (Linux fast-kill and the cross-platform fallback path) - Inline unit tests in every SFTP source file covering pure helpers (path canonicalisation, attribute mapping, S3-error classification, env-var bound resolvers) diff --git a/crates/protocols/src/sftp/config.rs b/crates/protocols/src/sftp/config.rs index 12b0c8894..dcb4b13d7 100644 --- a/crates/protocols/src/sftp/config.rs +++ b/crates/protocols/src/sftp/config.rs @@ -85,12 +85,12 @@ pub enum SftpInitError { #[error("SSH server error: {0}")] Server(String), - /// The host running the binary is not a Unix-family target. The - /// host-key permission enforcement (the owner-only mode-bit check) - /// requires Unix mode bits and has no equivalent on this platform, - /// so SFTP refuses to start rather than load host keys with weaker - /// guarantees. - #[error("SFTP requires a Unix-family host (current OS: {os})")] + /// The host running the binary is neither a Unix-family target nor + /// Windows. Unix enforces owner-only host-key mode bits and Windows + /// trusts operator-managed NTFS ACLs, but neither mechanism exists on + /// other targets, so SFTP refuses to start rather than load host keys + /// with weaker guarantees. + #[error("SFTP requires a Unix-family or Windows host (current OS: {os})")] UnsupportedPlatform { os: String }, } diff --git a/crates/protocols/src/sftp/driver.rs b/crates/protocols/src/sftp/driver.rs index fae030b56..adc83db85 100644 --- a/crates/protocols/src/sftp/driver.rs +++ b/crates/protocols/src/sftp/driver.rs @@ -52,7 +52,7 @@ use uuid::Uuid; /// Try-acquire returns immediately. If no permit is available the abort /// is skipped and the orphaned upload_id is reclaimed by the bucket /// AbortIncompleteMultipartUpload lifecycle rule documented in -/// OperatorDeploymentNotes.md. +/// docs/operations/sftp.md. const ABORT_PERMITS_FLOOR: usize = 8; const ABORT_PERMITS_CEILING: usize = 128; static ABORT_PERMITS: LazyLock> = LazyLock::new(|| { @@ -68,8 +68,8 @@ pub struct SftpDriver { pub(super) storage: Arc, pub(super) session_context: SessionContext, /// When true, write operations (OPEN with any write flag, WRITE, - /// REMOVE, MKDIR, RMDIR, RENAME) are rejected with PermissionDenied - /// before any backend call runs. + /// SETSTAT, FSETSTAT, REMOVE, MKDIR, RMDIR, RENAME) are rejected with + /// PermissionDenied before any backend call runs. pub(super) read_only: bool, pub(super) handles: HashMap, /// S3 multipart part size in bytes. Bytes accumulate in the per-handle diff --git a/crates/protocols/src/sftp/mod.rs b/crates/protocols/src/sftp/mod.rs index 00822801d..d4d2c8405 100644 --- a/crates/protocols/src/sftp/mod.rs +++ b/crates/protocols/src/sftp/mod.rs @@ -81,8 +81,10 @@ //! handler dispatch type. SftpError is the error type returned by SFTP //! operations. //! -//! Platform support. Host-key permission enforcement uses Unix mode bits. -//! On non-Unix targets SftpConfig::load_host_keys returns +//! Platform support. On Unix the host-key loader enforces owner-only mode +//! bits. On Windows it loads host keys without a mode check and trusts the +//! operator-managed NTFS ACLs on the key directory. On targets that are +//! neither Unix nor Windows, SftpConfig::load_host_keys returns //! SftpInitError::UnsupportedPlatform and the SFTP listener does not start. //! //! Peer-initiated signal requests on an open SFTP channel are intercepted diff --git a/docs/operations/sftp.md b/docs/operations/sftp.md new file mode 100644 index 000000000..97821f19e --- /dev/null +++ b/docs/operations/sftp.md @@ -0,0 +1,313 @@ +# SFTP Operations Guide + +The guide covers enabling and operating the SFTP server in RustFS. +It is written for operators who need to expose buckets over SFTP, manage host +keys across platforms, size the server for large transfers, and diagnose +session cleanup behaviour. + +## Enabling SFTP: Recommended Configuration + +Two settings are required: the enable flag and the host-key directory. The +listen address has a default of `0.0.0.0:2222` and setting it explicitly is +recommended practice. + +```bash +RUSTFS_SFTP_ENABLE=true +RUSTFS_SFTP_ADDRESS=0.0.0.0:2222 +RUSTFS_SFTP_HOST_KEY_DIR=/etc/rustfs/sftp-keys +``` + +- `RUSTFS_SFTP_ENABLE` starts the SFTP listener at server startup. Off by + default. +- `RUSTFS_SFTP_ADDRESS` is the listen address and port. `0.0.0.0` accepts + connections on every interface. Port `2222` avoids the privileged port 22. +- `RUSTFS_SFTP_HOST_KEY_DIR` is the directory the SSH host keys are loaded + from. No default, and startup fails when SFTP is enabled without it. + +Generate a host key before first start. On Unix, `ssh-keygen` also writes a +world-readable `.pub` file into the directory, and the server requires every +file it considers in the host-key directory (regular, non-empty, at most +1 MiB) to be owner-only, so restrict or remove it: + +```bash +ssh-keygen -t ed25519 -f /etc/rustfs/sftp-keys/ssh_host_ed25519_key -N "" +chmod 600 /etc/rustfs/sftp-keys/ssh_host_ed25519_key.pub +``` + +The private key itself is already written with owner-only permissions. The +server does not read the `.pub` file, so it can be removed instead. Keys must +be unencrypted (no passphrase). + +On success the server logs `SFTP server listening` with the bound address. A +port of `0` in `RUSTFS_SFTP_ADDRESS` is resolved to a free port at startup +and the resolved port appears in that log line. + +Every other setting has a tested default and should be left alone unless a +section below gives a concrete reason to change it. + +## Path Model + +The SFTP root directory is the account's bucket list. The first path +component names the bucket and the remainder is the object key: + +```text +/reports/2026/q1.pdf + +bucket: reports +object key: 2026/q1.pdf +``` + +Files cannot be created at the root level. Creating or removing a top-level +directory creates or removes a bucket. The root listing shows at most 10000 +buckets and logs a warning when truncated. + +## Host Keys + +`RUSTFS_SFTP_HOST_KEY_DIR` must name an existing directory containing at +least one decodable private key. Startup fails otherwise. There is no +generated fallback key. + +- Any private key that russh can decode is accepted. Ed25519, ECDSA, and RSA + are the expected formats. Passphrase-protected keys cannot be decoded and + do not count. Keys are offered to clients in the order Ed25519, ECDSA, + RSA, then anything else. Multiple keys of one algorithm all load, so during + key rotation clients can be offered the new key as soon as it is added, + not when the old one is removed. +- Empty files and files larger than 1 MiB are skipped entirely. +- On Unix, every other regular file in the directory must have no group or + other permission bits set (owner-only, for example mode `0600`, `0400`, or + `0700`). The check covers non-key files too: a world-readable README or + `.pub` file fails startup with `host key file has insecure permissions`. + Keep only owner-only files in the directory. +- On Windows there is no permission-bit check. The server logs a one-time + warning at startup whose alertable first sentence reads exactly `SFTP host + key file permission enforcement is not active on Windows`. Restrict the + NTFS ACL on the host-key directory to the running rustfs service account, + `NT AUTHORITY\SYSTEM`, and `BUILTIN\Administrators`. Default ProgramData + inheritance grants `BUILTIN\Users` read access. Remove that grant on the + host-key directory. +- Targets that are neither Unix nor Windows do not support SFTP and fail + startup. + +Host keys can be hot-reloaded without a restart by setting +`RUSTFS_SFTP_HOST_KEY_RELOAD_ENABLE=true`. The directory is rescanned every +`RUSTFS_SFTP_HOST_KEY_RELOAD_INTERVAL` seconds (default 30, silently raised +to 5 if set lower). A failed rescan, including a permission violation that +would be fatal at startup, keeps the previous keys and logs `SFTP host key +reload failed; keeping previous keys`. Reloaded keys affect new connections +only. + +## Configuration Reference + +Changing values beyond the recommended section is rarely necessary. The +defaults are the configuration the test suites and stress runs exercise. +Tuning values interact (for example part size multiplies against the handle +cap in worst-case memory), so change them deliberately and one at a time. + +Invalid values fall into four classes: + +1. Fatal at startup. The server refuses to start and names the problem. +2. Warn and use the default. The server starts and logs a warning naming + the variable, the rejected value, and the bounds. +3. Silently clamped. The reload interval is the one variable raised to its + floor without a warning. +4. Silent fallback. Non-numeric text in any numeric variable behaves as if + the variable were unset. Invalid boolean text logs a one-time warning + and uses the default. + +| Variable | Default | Valid values | On invalid | +| -------- | ------- | ------------ | ---------- | +| `RUSTFS_SFTP_ENABLE` | `false` | boolean | warn, default | +| `RUSTFS_SFTP_ADDRESS` | `0.0.0.0:2222` | see binding note below | fatal | +| `RUSTFS_SFTP_HOST_KEY_DIR` | none | existing directory, required | fatal | +| `RUSTFS_SFTP_IDLE_TIMEOUT` | `600` | seconds, greater than zero | fatal | +| `RUSTFS_SFTP_PART_SIZE` | `16777216` | bytes, 5 MiB to 5 GiB | fatal | +| `RUSTFS_SFTP_READ_ONLY` | `false` | boolean | warn, default | +| `RUSTFS_SFTP_BANNER` | `SSH-2.0-RustFS` | must start with `SSH-2.0-` | fatal | +| `RUSTFS_SFTP_HANDLES_PER_SESSION` | `64` | `8` to `1024` | warn, default | +| `RUSTFS_SFTP_BACKEND_OP_TIMEOUT_SECS` | `60` | `5` to `600` | warn, default | +| `RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES` | `4194304` | `0` disables, else 256 KiB to 64 MiB | warn, default | +| `RUSTFS_SFTP_READ_CACHE_TOTAL_MEM_BYTES` | `268435456` | at least 16 MiB | warn, default | +| `RUSTFS_SFTP_HOST_KEY_RELOAD_ENABLE` | `false` | boolean | warn, default | +| `RUSTFS_SFTP_HOST_KEY_RELOAD_INTERVAL` | `30` | seconds, minimum 5 | clamped to 5 | + +Notes on individual variables: + +- `RUSTFS_SFTP_ADDRESS`: the host part must be a wildcard (`0.0.0.0` or + `[::]`) or an address or hostname assigned to the host, otherwise startup + fails. Use `[::]:2222` to listen on IPv6, which on Linux usually accepts + IPv4 as well via dual-stack. Port `0` auto-assigns a free port. +- `RUSTFS_SFTP_BANNER` is the SSH protocol identification string sent on + connect, not a free-text login banner. Values that do not start with + `SSH-2.0-` fail startup. +- `RUSTFS_SFTP_IDLE_TIMEOUT` cannot be set to `0` to disable idle + disconnects. See the sessions section for what closes idle sessions. +- `RUSTFS_SFTP_PART_SIZE` bounds follow S3 multipart limits. See the large + files section before changing it. +- `RUSTFS_SFTP_READ_CACHE_WINDOW_BYTES` set to exactly `0` disables read + caching, which turns every client read request into one backend call. +- Worst-case buffered write memory per session is the handle cap times the + part size: 64 handles at 16 MiB is 1 GiB per session. The server imposes + no limit on concurrent sessions, so total worst case is that figure + times however many clients connect. Only the read cache has a global + cap. Enforce connection limits externally if that matters. + +## Sessions and Cleanup + +Several mechanisms close sessions: + +- The server sends an SSH keepalive every 15 seconds and closes the + connection after 3 consecutive unanswered keepalives, about 60 seconds + after a client stops responding. Not configurable. The keepalive check + cleans up clients that vanish without closing TCP. +- `RUSTFS_SFTP_IDLE_TIMEOUT` (default 600 seconds) sets the SSH inactivity + timeout, but keepalive replies count as SSH traffic and reset it, so a + client that answers keepalives is never disconnected by it. +- The session watchdog closes sessions that are silent at the SFTP request + layer for 30 minutes, on every platform. Keepalives do not count as + SFTP-layer activity. The watchdog is the mechanism that ends a healthy + but idle session. On Linux the close logs `wedge watchdog cancelling + session` with reason `fallback_silence`. On other platforms it logs + `fallback watchdog cancelling session`. +- On Linux the watchdog additionally reads kernel TCP state for fast + cleanup of wedged sessions: a socket sitting in `CLOSE_WAIT` across two + consecutive 15-second checks while the SFTP layer has been silent for 30 + seconds is cancelled, typically 45 to 60 seconds after the client + vanished. Two consecutive failed TCP-state probes are treated the same + way, so a container that blocks `/proc/net/tcp` can see sessions + cancelled on that schedule. The `reason` field of the log line names the + trigger. +- If duplicating the connection socket fails at accept time (rare, usually + file-descriptor exhaustion) the session runs with no watchdog at all and + only the keepalive and idle mechanisms apply. Log line: `wedge watchdog: + dup_socket failed`. +- Server shutdown cancels every live session immediately. Clients are + disconnected mid-transfer. The server then waits up to 30 seconds for + session cleanup, including aborting in-flight multipart uploads, before + remaining tasks are dropped. + +A session ended by any cancellation path logs `SFTP session cancelled +(watchdog or server shutdown)`. + +## Authentication and Authorization + +Authentication is by password only, verified against RustFS IAM users. +Public-key authentication is rejected. Anonymous access is not available. +Failed logins are rejected without delay and there is no lockout, so apply +rate limiting externally when the listener is exposed to untrusted +networks. Accepted logins log `SFTP auth accepted` at info level, rejections +log `SFTP auth rejected` at warn level. + +Every SFTP operation is authorized against IAM policy before it reaches +storage. Policy condition keys (for example `aws:SourceIp`) are not +evaluated on the SFTP path. Only unconditional Allow and Deny statements +take effect. + +The S3 actions a user needs: + +| SFTP activity | Required S3 actions | +| ------------- | ------------------- | +| List the root directory (the bucket list) | `s3:ListAllMyBuckets` | +| List inside a bucket | `s3:ListBucket` | +| Download a file | `s3:GetObject` | +| Upload a file | `s3:PutObject` | +| Stat a file | `s3:GetObject` | +| Stat a bucket | `s3:ListBucket` | +| Delete a file | `s3:DeleteObject` | +| Rename (implemented as copy then delete) | `s3:GetObject` on the source, `s3:PutObject` on the destination, `s3:DeleteObject` on the source | +| Create or remove a top-level directory | `s3:CreateBucket` or `s3:DeleteBucket`, removal also needs `s3:ListBucket` for the emptiness check | +| Upload cleanup on disconnect | `s3:AbortMultipartUpload` (see the large files section) | + +Upload-only users also need `s3:GetObject`, because SFTP clients stat files +as part of normal transfers. + +With `RUSTFS_SFTP_READ_ONLY=true` the server rejects all mutating packets +at the protocol layer: opening a file for write, `WRITE`, `REMOVE`, +`MKDIR`, `RMDIR`, `RENAME`, `SETSTAT`, and `FSETSTAT`. + +## Client Compatibility + +The server speaks SFTP version 3 and maps onto object storage. Differences +from a filesystem-backed SFTP server: + +- Uploads must be a single sequential stream from offset zero. Transfer + resume, append mode, in-place edits (read-write opens), and segmented or + multi-connection uploads of one file are rejected. Configure clients for + whole-file, single-connection transfers. +- In normal (read-write) mode, `SETSTAT` and `FSETSTAT` are accepted and + ignored: chmod, timestamp preservation, and ownership changes silently + have no effect, and listed permissions are fixed server-generated values. +- Symlink operations (`SYMLINK`, `READLINK`) are not supported and return + an unsupported-operation error. Object storage has no symlink equivalent. +- Renaming copies the object server-side and then deletes the source, so + large-file renames are slow and not atomic: a failure after the copy can + leave the file at both paths. Renaming a top-level directory (a bucket) + is not supported. + +## Large Files and Multipart Uploads + +An upload smaller than `RUSTFS_SFTP_PART_SIZE` bytes is buffered in memory +and written with a single `PutObject` when the file is closed. At part size +or larger the handle switches to S3 multipart, flushing a part each time a +full part accumulates. + +The server enforces the S3 limit of 10000 parts per upload, so the largest +single upload is part size times 10000: 156.25 GiB at the default 16 MiB +part size. Raise `RUSTFS_SFTP_PART_SIZE` until that product covers the +largest expected file. A larger part size raises per-session memory. A +write that would exceed the cap is rejected with log line `SFTP write would +exceed the S3 multipart parts limit`. The same cap can reject at file +close, logging `SFTP close rejected: trailing part would exceed S3 +multipart parts limit`. + +When a session ends mid-upload the server aborts the in-flight multipart +upload. Four situations leave an orphaned upload behind that the session +itself cannot clean up: + +- the user lacks `s3:AbortMultipartUpload`, +- the abort itself fails or times out, +- a burst of simultaneous disconnects exhausts the global abort task pool + (sized at twice the CPU parallelism, between 8 and 128), +- the server process is killed outright. + +Orphaned uploads are not permanent. RustFS runs a background cleanup that +aborts stale incomplete multipart uploads (by default, uploads older than +24 hours, checked every 6 hours). An `AbortIncompleteMultipartUpload` +bucket lifecycle rule reclaims them as well and gives per-bucket control of +the window. The log lines when a session-drop abort is skipped or fails: + +```text +skipped abort of orphaned multipart upload on session drop, principal lacks s3:AbortMultipartUpload, bucket lifecycle rules must reclaim parts +abort permit pool exhausted on session drop, bucket lifecycle rule must reclaim parts +failed to abort orphaned multipart upload +Drop abort of orphaned multipart upload timed out; bucket lifecycle rule must reclaim parts +``` + +Abort failures at file close log with different wording and are retried at +session drop, where the lines above appear. + +## Log Lines Worth Alerting On + +Every line below logs at warn level except `SFTP server listening` and +`SFTP auth accepted`, which log at info. The server's default log level is +error, so none of them are visible until the log level is raised to info +(for example `RUSTFS_OBS_LOGGER_LEVEL=info`, or warn to capture alerts +only). + +| Message | Meaning | +| ------- | ------- | +| `SFTP server listening` | Startup complete, address bound | +| `SFTP auth rejected` | Failed login, no built-in lockout exists | +| `SFTP host key file permission enforcement is not active on Windows` | Expected once per start on Windows, verify the ACL guidance once | +| `host key file has insecure permissions` | Fatal at startup on Unix, fix file modes. During hot reload the same text appears inside the reload-failed warning and previous keys stay active | +| `SFTP host key reload failed; keeping previous keys` | Hot reload rescan failed, service unaffected, investigate the directory | +| `wedge watchdog cancelling session` | Linux only. `reason` field: `tcp_state_close_wait_confirmed` or `probe_failed_confirmed` mean a wedged or unprobeable socket, `fallback_silence` means routine 30-minute idle cleanup | +| `fallback watchdog cancelling session` | Non-Linux platforms: session silent at the SFTP layer for 30 minutes | +| `wedge watchdog: dup_socket failed` | Rare, that session runs without any watchdog | +| `SFTP session cancelled (watchdog or server shutdown)` | Session ended by cancellation rather than client close | +| `SFTP write would exceed the S3 multipart parts limit` | Client hit the per-upload size cap, raise the part size if legitimate. Also check the close-time variant below | +| `SFTP close rejected: trailing part would exceed S3 multipart parts limit` | Same cap hit at file close | +| `root READDIR truncated` | A principal can see more than 10000 buckets, listing was cut off | +| `skipped abort of orphaned multipart upload` | Orphaned parts on storage until background cleanup or lifecycle rule reclaims them | +| `abort permit pool exhausted on session drop` | Mass-disconnect burst, orphaned parts on storage until reclaimed | +| `RUSTFS_SFTP_` prefix in a warning | A tuning variable was rejected and its default applied |