Three hardening fixes with the same shape: a number that was true under the
defaults and untrue at the edges.
1. QUEUE MEMORY WAS AN OPERATOR SETTING, NOT A LIMIT.
The queue was bounded by ROW COUNT only, and how much a row weighs is
`requestlog.max_body_bytes` - editable from Settings, documented ceiling 256 KB,
and a row can hold that twice (request + response). Measured on the real
dataclass with distinct buffers per row:
defaults, 2 000 rows x 8 KB 33.9 MiB 3.3% of the 1 GiB pod limit
max_body_bytes at its 256 KB ceiling 1003 MiB at the pod limit
REQUEST_LOG_QUEUE_MAX at its ceiling 1695 MiB over the pod limit
Both are reachable from in-range, documented values, and the drop warning
advised "raise REQUEST_LOG_QUEUE_MAX" - so following the tool's own advice on a
busy install could OOM the worker. REQUEST_LOG_QUEUE_MAX_BYTES (default 64 MiB)
now caps the queue in bytes as well as in rows, whichever binds first, released
as rows drain. Verified: with max_body_bytes at 256 KB the queue holds 7.5 MiB
against an 8 MiB budget where it would otherwise have held 1003 MiB, and it
accepts rows again as soon as the writer drains it. The warning text now names
the setting that actually helps.
2. SINK COUNTERS ARE PER WORKER AND DID NOT SAY SO.
The sink is a module global, so with UVICORN_WORKERS > 1 each process has its
own queue and its own counters, and `GET /api/request-logs/stats` reports
whichever worker happened to serve the request. The feature is sold on "a
saturated logger drops rows visibly"; at 4 workers the visible number was a
quarter of the truth. Labelled `"scope": "this worker only"` rather than
aggregated - there is no cross-process channel here, and a number that looks
fleet-wide but is not is worse than one that admits its scope.
3. AN EMPTY EXCLUDE LIST IS NOT APPLIED AS "LOG EVERYTHING".
normalize_exclude_paths() falls back to the shipped defaults when the list comes
out empty, which is the right call - it keeps the log viewer and the raw-body
heartbeat endpoint excluded - but the UI kept displaying the empty list the
operator typed, so the form showed a policy that was not in effect. The save
handler now re-applies whatever the server actually stored (which also surfaces
server-side clamping of every numeric field) and says plainly that the defaults
were restored.
Until now the only record of what happened was `user_activity_logs`, which
stores non-GET 2xx operations with no bodies. When something failed you could
see that a counter went up, never what was sent or what came back.
This adds one queryable timeline covering both directions:
- inbound: every API call, including GETs and including 4xx/5xx, with the
user, client IP, status, duration and — redacted, size-capped — the request
and response bodies.
- outbound: every HTTP call the backend makes, tagged with who it went to
(ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME
diagnostics probe).
Outbound rows inherit the inbound request's id, so one operator action and the
CA/DNS calls it triggered read as a single trace: opening a failed "Request
Certificate" shows the exact POST /acme/new-order and the CA's 429 underneath.
Implementation notes:
- Capture is a pure-ASGI middleware that TEES the request and response streams
rather than draining them. `await request.body()` inside a BaseHTTPMiddleware
would consume the receive channel and break the raw-body agent heartbeat
handler. Registered last so it is outermost: it then sees the final
client-visible response and seeds correlation_id_context before the error
handler reads it.
- Rows are written by a batching background writer with a bounded queue, so the
request path never awaits the database and a saturated logger drops rows
visibly (surfaced on the page) instead of blocking. Redaction runs on the
writer, off the request coroutine.
- Secrets never land: headers are an allowlist with Authorization/Cookie kept
only as a presence marker; body keys and value shapes are redacted
(passwords, tokens, api_token, API keys, private-key PEMs, JWTs); the ACME
JWS request body is never stored, because a stored protected+signature pair
is a replayable credential — a summary is logged instead; DNS-provider errors
record only the exception type; the ACME HTTP-01 challenge endpoint is
excluded so key_authorization is never captured.
- Retention is operator-configurable in Settings -> Request Log: separate day
counts for successful and failed rows (7 / 30) plus a hard row cap (500k),
whichever is reached first. Pruned in batches under a Postgres advisory lock,
with the day counts bound as parameters, never interpolated.
- New permissions requestlog.read / requestlog.manage. super_admin and
security_admin get both, operator gets read, viewer gets neither.
Schema: one new table (request_logs) plus its settings seed, SCHEMA_VERSION
10 -> 11, auto-migrated. No existing table altered, no agent or rendered-config
change. Kill switches: REQUEST_LOG_ENABLED=false (middleware never registered)
or the `enabled` toggle in Settings.
Tests: 245 new (7 backend files + 1 frontend), full suite 1655 backend +
17 frontend passing.
- Fix frontend port conflict validation to consider bind_address+port
combination instead of port-only. HAProxy allows same port on different
bind addresses (e.g., bind 10.0.0.1:443 vs bind 10.0.0.2:443). This
was blocking frontend edit/save in multi-VIP environments.
- Add form field dependency so port re-validates when bind_address changes.
- Fix API URL construction using window.location.host instead of hostname
to preserve non-standard ports (e.g., :8080), preventing CORS errors
in BulkConfigImport.
- Make CORS_ORIGINS configurable via environment variable.
Changed ENTITY_SNAPSHOT_ENABLED default from false to true.
Reasoning:
- Code is tested and deployed to production
- Backward compatibility verified
- No need for gradual rollout with feature flag
- Entity rollback should work by default
- Users expect reject to rollback entities (not just status change)
Feature flag still exists for emergency disable if needed:
- Set ENTITY_SNAPSHOT_ENABLED=false to disable
- Useful for troubleshooting or rollback scenarios
Default behavior (ENTITY_SNAPSHOT_ENABLED=true):
- Entity update creates snapshot in metadata
- Reject operation rolls back entities to old values
- Bulk import reject deletes new entities, restores updated ones
- Restore reject returns to pre-restore state
taylanbakircioglu