gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-08 04:58:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(knowledge-base): prune stale content and add agent-facing index (#7035)

Browse Source
This commit is contained in:
Zhengchao An
2026-09-02 08:26:59 +08:00
committed by GitHub
parent ceeff52229
commit 0a975f2fe2
99 changed files with 3312 additions and 10590 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations/reverse-proxy.md
+34 -90
View File
@@ -1,70 +1,38 @@
# Running RustFS behind a reverse proxy
RustFS speaks plain S3 over HTTP/1.1 and HTTP/2 and works behind reverse
proxies (Caddy, Nginx, HAProxy) and CDNs (Cloudflare). Most proxy problems are
**not** RustFS storage bugs — the same request sent directly to `:9000`
succeeds, while the proxied request fails. This page documents the request
semantics RustFS expects from the proxy layer and gives known-good
configurations.
**Use this when:** a request succeeds against `http://<host>:9000` directly but fails, hangs, or resets through Caddy, Nginx, HAProxy, or Cloudflare.
> Rule of thumb: if a request works against `http://<host>:9000` directly but
> fails through the proxy, the fault is in the proxy/CDN request forwarding, not
> in RustFS object handling. Use the checklist below to find which forwarding
> behavior broke.
**Source of truth:** `crates/config/src/constants/tls.rs` (`DEFAULT_HTTP1_HEADER_READ_TIMEOUT`, `DEFAULT_HTTP_REQUEST_BODY_READ_TIMEOUT`); the `put_object_body_read_stalled` log event.
RustFS speaks plain S3 over HTTP/1.1 and HTTP/2. Most proxy problems are not RustFS storage bugs: if the same request works directly against `:9000`, the fault is in proxy/CDN request forwarding. Use the checklist below to find which forwarding behavior broke.
## What RustFS requires from the proxy
S3 clients sign requests with AWS SigV4. RustFS (via `s3s`) re-derives the
signature from the forwarded request, and streams the request body to storage.
For this to succeed the proxy must forward the request **byte-for-byte** with
respect to the signed material and the body:
S3 clients sign requests with AWS SigV4. RustFS (via `s3s`) re-derives the signature from the forwarded request and streams the request body to storage, so the proxy must forward the signed material and the body byte-for-byte:
1. **Do not alter the body.** No transparent compression, no re-encoding, no
truncation. If the client sent `Content-Length: N`, exactly `N` body bytes
must reach RustFS. If fewer bytes arrive, RustFS waits for the rest per the
HTTP spec and the request appears to hang until the client aborts.
2. **Do not rewrite signed headers.** `Host` and any `x-amz-*` / signed headers
must reach RustFS unchanged. Rewriting `Host` is fine only if the client
signed with that same host.
3. **Preserve `Content-Length`; avoid re-chunking large bodies.** Some CDNs
drop `Content-Length` and switch to `Transfer-Encoding: chunked`, or buffer
the whole request body before forwarding — both change the timing and
framing RustFS sees.
4. **Keep upstream idle keep-alive shorter than RustFS's, or vice-versa** (see
next section) so the proxy never reuses a connection RustFS has already
closed.
5. **Do not strip `ETag`** from responses (breaks multipart completion).
| Requirement | Why |
| --- | --- |
| Do not alter the body (no compression, re-encoding, truncation). | If the client sent `Content-Length: N`, exactly `N` body bytes must arrive; with fewer, RustFS waits for the rest and the request appears to hang until the client aborts. |
| Do not rewrite signed headers (`Host`, `x-amz-*`). | Rewriting `Host` is fine only if the client signed with that same host; otherwise `SignatureDoesNotMatch`. |
| Preserve `Content-Length`; do not re-chunk or buffer large bodies. | Switching to `Transfer-Encoding: chunked` or buffering the whole body changes the framing and timing RustFS sees. |
| Keep the proxy's upstream idle keep-alive shorter than RustFS's timeout (next section). | Otherwise the proxy reuses a connection RustFS has already closed. |
| Do not strip `ETag` from responses. | Breaks multipart completion. |
## Idle keep-alive: the #1 cause of `socket hang up` on writes
## Idle keep-alive: the main cause of `socket hang up` on writes
RustFS closes **idle** upstream HTTP/1.1 keep-alive connections after
`RUSTFS_HTTP1_HEADER_READ_TIMEOUT` seconds (default **75s**; see
`crates/config/src/constants/tls.rs`). Reverse proxies keep a pool of upstream
connections and reuse them. If the proxy's upstream idle-keepalive window is
**longer** than RustFS's timeout, the proxy can pick a connection that RustFS
has already FIN'd, write a request onto the dead socket, and the client sees:
RustFS closes idle upstream HTTP/1.1 keep-alive connections after `RUSTFS_HTTP1_HEADER_READ_TIMEOUT` seconds (`DEFAULT_HTTP1_HEADER_READ_TIMEOUT`, 75). Reverse proxies pool and reuse upstream connections. If the proxy's upstream idle-keepalive window is longer than RustFS's timeout, the proxy can pick a connection RustFS has already FIN'd, write a request onto the dead socket, and the client sees:
```
```text
TimeoutError: socket hang up # ECONNRESET
AbortError: Request aborted
```
This is most visible on large `PutObject` uploads because:
This is most visible on large `PutObject` uploads: `PUT` is non-idempotent, so proxies will not transparently retry it, and a larger body keeps the connection in use longer, widening the race window, so small uploads on the same path often succeed.
- `PUT` is non-idempotent, so proxies will **not** transparently retry it; and
- a larger body keeps the connection in use longer, widening the race window,
so small uploads on the same path often succeed.
Fix by making the two windows agree (doing both is safest):
### Fix — make the two windows agree
Pick **either** side; doing both is safest:
- **RustFS side:** keep `RUSTFS_HTTP1_HEADER_READ_TIMEOUT` (default 75s) *above*
the proxy's upstream idle-keepalive. To harden slowloris protection on a
directly-exposed node instead, lower it — but then also lower the proxy
keepalive below it.
- **Proxy side:** lower the proxy's upstream idle-keepalive below RustFS's
timeout, or disable upstream keep-alive entirely.
1. RustFS side: keep `RUSTFS_HTTP1_HEADER_READ_TIMEOUT` above the proxy's upstream idle-keepalive. To harden slowloris protection on a directly exposed node instead, lower it, and then also lower the proxy keepalive below it.
2. Proxy side: lower the proxy's upstream idle-keepalive below RustFS's timeout, or disable upstream keep-alive entirely.
## Known-good Caddy configuration
@@ -127,47 +95,23 @@ location / {
## Cloudflare (orange-cloud) caveats
Cloudflare's proxy (orange cloud) may **buffer the entire request body** before
forwarding, and can rewrite requests to `Transfer-Encoding: chunked`, dropping
the client's `Content-Length`. Symptoms match this pattern exactly: tiny uploads
succeed, larger uploads fail with `socket hang up`.
- For large object writes, prefer **DNS-only (grey cloud)** for the S3 endpoint,
or a Cloudflare plan/tunnel configuration that does not buffer/re-chunk the
request body.
- Force `Accept-Encoding: identity` so nothing in the path negotiates
compression (see issues #609, #1492).
- Ensure `Content-Length` reaches RustFS; disable chunked re-encoding in tunnel
settings (see issue #934).
Cloudflare's proxy may buffer the entire request body before forwarding and can rewrite requests to `Transfer-Encoding: chunked`, dropping the client's `Content-Length`. The symptom is exactly the pattern above: tiny uploads succeed, larger uploads fail with `socket hang up`. For large object writes prefer DNS-only (grey cloud) for the S3 endpoint, or a plan/tunnel configuration that does not buffer or re-chunk the body. The `Accept-Encoding` and `Content-Length` rows in the issue table below are the Cloudflare-specific failures seen so far.
## Diagnosis checklist
Run each step and note where behavior diverges:
1. Bypass the proxy. Send the failing request to `http://<host>:9000` directly. Success confirms the fault is in the proxy/CDN path.
2. Bypass the CDN, keep the proxy. Point the proxy straight at the origin (Cloudflare grey cloud / direct DNS). If it now works, the CDN was buffering or re-chunking the body.
3. Check idle reuse. Intermittent failures that correlate with upload size are almost always the keep-alive mismatch. Lower the proxy keepalive (or disable it) and retry.
4. Check for a truncated body. If the upload hangs indefinitely rather than resetting, the proxy is forwarding a partial body and then going silent without closing the connection. RustFS bounds this wait with `RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT` (`DEFAULT_HTTP_REQUEST_BODY_READ_TIMEOUT`, 300; `0` disables) and on timeout logs `put_object_body_read_stalled` with the received/expected byte counts.
5. Compare bytes. Confirm the proxy forwards exactly `Content-Length` body bytes with no compression or transformation.
6. Confirm signed headers survive. `Host` and `x-amz-*` must reach RustFS unchanged; a `SignatureDoesNotMatch` (rather than a hang) points here.
1. **Bypass the proxy.** Send the failing request to `http://<host>:9000`
directly. Success here confirms the fault is in the proxy/CDN path.
2. **Bypass the CDN, keep the proxy.** Point the proxy straight at the origin
(Cloudflare grey cloud / direct DNS). If it now works, the CDN was
buffering/re-chunking the body.
3. **Check idle reuse.** If failures are intermittent and correlate with upload
size, it is almost always the keep-alive mismatch above. Lower the proxy
keepalive (or disable it) and retry.
- If instead the upload **hangs indefinitely** (rather than resetting), the
proxy is likely forwarding a *partial* body and then going silent without
closing the connection. RustFS bounds this wait with
`RUSTFS_HTTP_REQUEST_BODY_READ_TIMEOUT` (default 300s; `0` disables) and, on
timeout, logs a `put_object_body_read_stalled` event with the
received/expected byte counts — grep the server log for it to confirm a
truncated-body forwarding problem.
4. **Compare bytes.** Confirm the proxy forwards exactly `Content-Length` body
bytes with no compression/transformation.
5. **Confirm signed headers survive.** `Host` and `x-amz-*` headers must reach
RustFS unchanged; a `SignatureDoesNotMatch` (rather than a hang) points here.
## Known failure signatures
## Related issues
- #3076 — Large single-request PutObject fails behind Caddy (this document)
- #609 — Bucket inaccessible via Cloudflare proxied DNS (`Accept-Encoding`)
- #1492 — SigV4 `SignatureDoesNotMatch` on Cloudflare tunnel (`Accept-Encoding`)
- #934 — Console fails behind Cloudflare tunnels (chunked / `Content-Length`)
- #1766 — Large multipart upload fails through Nginx (`ETag` stripping)
| Symptom | Forwarding fault | Issue |
| --- | --- | --- |
| Large single-request PutObject fails behind Caddy | Upstream idle keep-alive longer than RustFS's timeout | #3076 |
| Bucket inaccessible via Cloudflare proxied DNS | `Accept-Encoding` negotiation / body transformation | #609 |
| SigV4 `SignatureDoesNotMatch` on Cloudflare tunnel | `Accept-Encoding` header rewritten | #1492 |
| Console fails behind Cloudflare tunnels | Chunked re-encoding drops `Content-Length` | #934 |
| Large multipart upload fails through Nginx | `ETag` stripped from responses | #1766 |