rustfs/docs/operations/reverse-proxy.md

# Running RustFS behind a reverse proxy

RustFS speaks plain S3 over HTTP/1.1 and HTTP/2 and works behind reverse
proxies (Caddy, Nginx, HAProxy) and CDNs (Cloudflare). Most proxy problems are
**not** RustFS storage bugs — the same request sent directly to `:9000`
succeeds, while the proxied request fails. This page documents the request
semantics RustFS expects from the proxy layer and gives known-good
configurations.

> Rule of thumb: if a request works against `http://<host>:9000` directly but
> fails through the proxy, the fault is in the proxy/CDN request forwarding, not
> in RustFS object handling. Use the checklist below to find which forwarding
> behavior broke.

## What RustFS requires from the proxy

S3 clients sign requests with AWS SigV4. RustFS (via `s3s`) re-derives the
signature from the forwarded request, and streams the request body to storage.
For this to succeed the proxy must forward the request **byte-for-byte** with
respect to the signed material and the body:

1. **Do not alter the body.** No transparent compression, no re-encoding, no
   truncation. If the client sent `Content-Length: N`, exactly `N` body bytes
   must reach RustFS. If fewer bytes arrive, RustFS waits for the rest per the
   HTTP spec and the request appears to hang until the client aborts.
2. **Do not rewrite signed headers.** `Host` and any `x-amz-*` / signed headers
   must reach RustFS unchanged. Rewriting `Host` is fine only if the client
   signed with that same host.
3. **Preserve `Content-Length`; avoid re-chunking large bodies.** Some CDNs
   drop `Content-Length` and switch to `Transfer-Encoding: chunked`, or buffer
   the whole request body before forwarding — both change the timing and
   framing RustFS sees.
4. **Keep upstream idle keep-alive shorter than RustFS's, or vice-versa** (see
   next section) so the proxy never reuses a connection RustFS has already
   closed.
5. **Do not strip `ETag`** from responses (breaks multipart completion).

## Idle keep-alive: the #1 cause of `socket hang up` on writes

RustFS closes **idle** upstream HTTP/1.1 keep-alive connections after
`RUSTFS_HTTP1_HEADER_READ_TIMEOUT` seconds (default **75s**; see
`crates/config/src/constants/tls.rs`). Reverse proxies keep a pool of upstream
connections and reuse them. If the proxy's upstream idle-keepalive window is
**longer** than RustFS's timeout, the proxy can pick a connection that RustFS
has already FIN'd, write a request onto the dead socket, and the client sees:

```
TimeoutError: socket hang up          # ECONNRESET
AbortError: Request aborted
```

This is most visible on large `PutObject` uploads because:

- `PUT` is non-idempotent, so proxies will **not** transparently retry it; and
- a larger body keeps the connection in use longer, widening the race window,
  so small uploads on the same path often succeed.

### Fix — make the two windows agree

Pick **either** side; doing both is safest:

- **RustFS side:** keep `RUSTFS_HTTP1_HEADER_READ_TIMEOUT` (default 75s) *above*
  the proxy's upstream idle-keepalive. To harden slowloris protection on a
  directly-exposed node instead, lower it — but then also lower the proxy
  keepalive below it.
- **Proxy side:** lower the proxy's upstream idle-keepalive below RustFS's
  timeout, or disable upstream keep-alive entirely.

## Known-good Caddy configuration

```caddy
your-domain.example.com {
	reverse_proxy http://127.0.0.1:9000 {
		transport http {
			# Talk HTTP/1.1 to RustFS.
			versions 1.1

			# Keep the proxy's upstream idle-keepalive BELOW RustFS's
			# RUSTFS_HTTP1_HEADER_READ_TIMEOUT (default 75s) so Caddy never
			# reuses a connection RustFS already closed. Set to 0 to disable
			# upstream keep-alive entirely (simplest, slightly less efficient).
			keepalive 30s
			keepalive_idle_conns_per_host 0

			# Never let the proxy compress/transform the request body.
			compression off

			# Generous timeouts for multi-MB single-request PUTs.
			dial_timeout 30s
			read_timeout 300s
			write_timeout 300s
		}

		# Forward the body untouched; do not negotiate compression upstream.
		header_up Accept-Encoding identity

		# Preserve the host the client signed with.
		header_up Host {upstream_hostport}

		# Stream immediately instead of buffering.
		flush_interval -1
	}
}
```

## Nginx equivalent (essentials)

```nginx
location / {
    proxy_pass http://127.0.0.1:9000;
    proxy_http_version 1.1;

    # Nginx default upstream keepalive is 60s; keep it under RustFS's 75s.
    # (set `keepalive` in the matching `upstream {}` block)
    proxy_set_header Connection "";

    proxy_set_header Host $host;
    proxy_set_header Accept-Encoding "identity";

    # Do not buffer/limit large uploads.
    proxy_request_buffering off;
    client_max_body_size 0;
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
}
```

## Cloudflare (orange-cloud) caveats

Cloudflare's proxy (orange cloud) may **buffer the entire request body** before
forwarding, and can rewrite requests to `Transfer-Encoding: chunked`, dropping
the client's `Content-Length`. Symptoms match this pattern exactly: tiny uploads
succeed, larger uploads fail with `socket hang up`.

- For large object writes, prefer **DNS-only (grey cloud)** for the S3 endpoint,
  or a Cloudflare plan/tunnel configuration that does not buffer/re-chunk the
  request body.
- Force `Accept-Encoding: identity` so nothing in the path negotiates
  compression (see issues #609, #1492).
- Ensure `Content-Length` reaches RustFS; disable chunked re-encoding in tunnel
  settings (see issue #934).

## Diagnosis checklist

Run each step and note where behavior diverges:

1. **Bypass the proxy.** Send the failing request to `http://<host>:9000`
   directly. Success here confirms the fault is in the proxy/CDN path.
2. **Bypass the CDN, keep the proxy.** Point the proxy straight at the origin
   (Cloudflare grey cloud / direct DNS). If it now works, the CDN was
   buffering/re-chunking the body.
3. **Check idle reuse.** If failures are intermittent and correlate with upload
   size, it is almost always the keep-alive mismatch above. Lower the proxy
   keepalive (or disable it) and retry.
   - If instead the upload **hangs indefinitely** (rather than resetting), the
     proxy is likely forwarding a *partial* body and then going silent without
     closing the connection. RustFS bounds this wait with
     `RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT` (default 300s; `0` disables) and, on
     timeout, logs a `put_object_body_read_stalled` event with the
     received/expected byte counts — grep the server log for it to confirm a
     truncated-body forwarding problem.
4. **Compare bytes.** Confirm the proxy forwards exactly `Content-Length` body
   bytes with no compression/transformation.
5. **Confirm signed headers survive.** `Host` and `x-amz-*` headers must reach
   RustFS unchanged; a `SignatureDoesNotMatch` (rather than a hang) points here.

## Related issues

- #3076 — Large single-request PutObject fails behind Caddy (this document)
- #609 — Bucket inaccessible via Cloudflare proxied DNS (`Accept-Encoding`)
- #1492 — SigV4 `SignatureDoesNotMatch` on Cloudflare tunnel (`Accept-Encoding`)
- #934 — Console fails behind Cloudflare tunnels (chunked / `Content-Length`)
- #1766 — Large multipart upload fails through Nginx (`ETag` stripping)