rustfs/docs/operations/reverse-proxy.md

Running RustFS behind a reverse proxy

Use this when: a request succeeds against http://<host>:9000 directly but fails, hangs, or resets through Caddy, Nginx, HAProxy, or Cloudflare.

Source of truth: crates/config/src/constants/tls.rs (DEFAULT_HTTP1_HEADER_READ_TIMEOUT, DEFAULT_HTTP_REQUEST_BODY_READ_TIMEOUT); the put_object_body_read_stalled log event.

RustFS speaks plain S3 over HTTP/1.1 and HTTP/2. Most proxy problems are not RustFS storage bugs: if the same request works directly against :9000, the fault is in proxy/CDN request forwarding. Use the checklist below to find which forwarding behavior broke.

What RustFS requires from the proxy

S3 clients sign requests with AWS SigV4. RustFS (via s3s) re-derives the signature from the forwarded request and streams the request body to storage, so the proxy must forward the signed material and the body byte-for-byte:

Requirement	Why
Do not alter the body (no compression, re-encoding, truncation).	If the client sent Content-Length: N, exactly N body bytes must arrive; with fewer, RustFS waits for the rest and the request appears to hang until the client aborts.
Do not rewrite signed headers (Host, x-amz-*).	Rewriting Host is fine only if the client signed with that same host; otherwise SignatureDoesNotMatch.
Preserve Content-Length; do not re-chunk or buffer large bodies.	Switching to Transfer-Encoding: chunked or buffering the whole body changes the framing and timing RustFS sees.
Keep the proxy's upstream idle keep-alive shorter than RustFS's timeout (next section).	Otherwise the proxy reuses a connection RustFS has already closed.
Do not strip ETag from responses.	Breaks multipart completion.

Idle keep-alive: the main cause of socket hang up on writes

RustFS closes idle upstream HTTP/1.1 keep-alive connections after RUSTFS_HTTP1_HEADER_READ_TIMEOUT seconds (DEFAULT_HTTP1_HEADER_READ_TIMEOUT, 75). Reverse proxies pool and reuse upstream connections. If the proxy's upstream idle-keepalive window is longer than RustFS's timeout, the proxy can pick a connection RustFS has already FIN'd, write a request onto the dead socket, and the client sees:

TimeoutError: socket hang up          # ECONNRESET
AbortError: Request aborted

This is most visible on large PutObject uploads: PUT is non-idempotent, so proxies will not transparently retry it, and a larger body keeps the connection in use longer, widening the race window, so small uploads on the same path often succeed.

Fix by making the two windows agree (doing both is safest):

1. RustFS side: keep RUSTFS_HTTP1_HEADER_READ_TIMEOUT above the proxy's upstream idle-keepalive. To harden slowloris protection on a directly exposed node instead, lower it, and then also lower the proxy keepalive below it.
2. Proxy side: lower the proxy's upstream idle-keepalive below RustFS's timeout, or disable upstream keep-alive entirely.

Known-good Caddy configuration

your-domain.example.com {
	reverse_proxy http://127.0.0.1:9000 {
		transport http {
			# Talk HTTP/1.1 to RustFS.
			versions 1.1

			# Keep the proxy's upstream idle-keepalive BELOW RustFS's
			# RUSTFS_HTTP1_HEADER_READ_TIMEOUT (default 75s) so Caddy never
			# reuses a connection RustFS already closed. Set to 0 to disable
			# upstream keep-alive entirely (simplest, slightly less efficient).
			keepalive 30s
			keepalive_idle_conns_per_host 0

			# Never let the proxy compress/transform the request body.
			compression off

			# Generous timeouts for multi-MB single-request PUTs.
			dial_timeout 30s
			read_timeout 300s
			write_timeout 300s
		}

		# Forward the body untouched; do not negotiate compression upstream.
		header_up Accept-Encoding identity

		# Preserve the host the client signed with.
		header_up Host {upstream_hostport}

		# Stream immediately instead of buffering.
		flush_interval -1
	}
}

Nginx equivalent (essentials)

location / {
    proxy_pass http://127.0.0.1:9000;
    proxy_http_version 1.1;

    # Nginx default upstream keepalive is 60s; keep it under RustFS's 75s.
    # (set `keepalive` in the matching `upstream {}` block)
    proxy_set_header Connection "";

    proxy_set_header Host $host;
    proxy_set_header Accept-Encoding "identity";

    # Do not buffer/limit large uploads.
    proxy_request_buffering off;
    client_max_body_size 0;
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
}

Cloudflare (orange-cloud) caveats

Cloudflare's proxy may buffer the entire request body before forwarding and can rewrite requests to Transfer-Encoding: chunked, dropping the client's Content-Length. The symptom is exactly the pattern above: tiny uploads succeed, larger uploads fail with socket hang up. For large object writes prefer DNS-only (grey cloud) for the S3 endpoint, or a plan/tunnel configuration that does not buffer or re-chunk the body. The Accept-Encoding and Content-Length rows in the issue table below are the Cloudflare-specific failures seen so far.

Diagnosis checklist

1. Bypass the proxy. Send the failing request to http://<host>:9000 directly. Success confirms the fault is in the proxy/CDN path.
2. Bypass the CDN, keep the proxy. Point the proxy straight at the origin (Cloudflare grey cloud / direct DNS). If it now works, the CDN was buffering or re-chunking the body.
3. Check idle reuse. Intermittent failures that correlate with upload size are almost always the keep-alive mismatch. Lower the proxy keepalive (or disable it) and retry.
4. Check for a stalled body. A client or intermediary can stop forwarding data without closing the connection. RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT defaults to 300 seconds. PutObject logs put_object_body_read_stalled when its body-read guard expires. UploadPart logs upload_part_body_read_stalled and returns RequestTimeout (HTTP 400); its log records raw_bytes_received, expected_decoded_bytes, timeout_secs, bucket, key, and request ID. The event identifies missing input progress, without attributing the cause to a particular proxy.
5. Compare bytes. Confirm the proxy forwards exactly Content-Length body bytes with no compression or transformation.
6. Confirm signed headers survive. Host and x-amz-* must reach RustFS unchanged; a SignatureDoesNotMatch (rather than a hang) points here.

For HTTP UploadPart, the inactivity budget counts time waiting for raw request-body bytes while storage is requesting input. Positive raw bytes reset the budget, including fragments of a signed AWS chunk that has not yet finished decoding. Foreground admission, capped-session staging, and storage backpressure do not consume the budget. Finishing the declared payload does not bypass the signed terminator, required trailers, or final body validation. This is an inactivity limit, so an upload making progress can take longer than 300 seconds overall.

Setting the timeout to 0 disables it for ordinary uploads. Multipart sessions with an explicit total-object-size cap retain a minimum 300-second timeout, including when the configured value is 0. After a body-stall timeout, HTTP/1 uses the existing raw-body drain and closes the connection; HTTP/2 releases the affected stream and keeps the connection usable.

UploadPart requires a known logical byte length. RustFS uses the length normalized by S3S after authentication and decoding, with an exact logical stream length as a fallback. A bare x-amz-decoded-content-length or Content-Encoding: aws-chunked declaration cannot supply this length by itself. Requests reaching an ordinary upload session without a known length return MissingContentLength (HTTP 411) before body ingestion; capped sessions retain their UnexpectedContent rejection. Preserve the client's framing and signed headers through the proxy. The 5 GiB limit applies to each part request, not the combined size of an ordinary multipart upload.

Known failure signatures

Symptom	Forwarding fault	Issue
Large single-request PutObject fails behind Caddy	Upstream idle keep-alive longer than RustFS's timeout	#3076
Bucket inaccessible via Cloudflare proxied DNS	Accept-Encoding negotiation / body transformation	#609
SigV4 SignatureDoesNotMatch on Cloudflare tunnel	Accept-Encoding header rewritten	#1492
Console fails behind Cloudflare tunnels	Chunked re-encoding drops Content-Length	#934
Large multipart upload fails through Nginx	ETag stripped from responses	#1766