The release notes inherited from the feature branch described the version it was
written against, not the one going out.
- `SCHEMA_VERSION` is 11 -> 12, not 10 -> 11, and the upgrade notes now say why:
11 was taken by v1.10.4 while this was in review, and the version gate would
have skipped the migration entirely on every existing install. Includes the
no-op recovery path for anyone running a pre-release build that recorded 11.
- Successful agent polls are not logged by default, with the measured table
behind it: 2 424 bytes/row on PostgreSQL 15 against the real schema and all
nine indexes, ~9 792 logged calls/day/agent, and what that means at 20, 200
and 500 nodes both ways. The point is not the disk, it is that the row cap
holds by DELETING, so without this the configured 7-day/30-day retention
quietly becomes a few hours for everything in the table.
- Runtime cost stated as measured numbers rather than adjectives: 27.7 us per
request, 1.4 us on an excluded path, 18.8 us per row on the writer, 0.096 %
of one core at 500 nodes.
- REQUEST_LOG_QUEUE_MAX_BYTES documented in .env.template and CONFIG.md, with
the reason it exists: the row count alone does not bound memory when
max_body_bytes is operator-editable to 256 KB.
- The old "raise REQUEST_LOG_QUEUE_MAX if you see drops" advice is corrected -
following it could OOM the worker. Lower max_body_bytes or sample_rate first;
if you do raise the queue, raise its byte ceiling with it.
- Two behaviours that used to be silent are now written down: sink counters are
per worker, and clearing the exclude-path list falls back to the shipped
defaults rather than logging everything.
- The `operator` role's visibility of agent rows is documented, including what
it deliberately does NOT extend to (anonymous traffic and the usernames in
failed logins).
The v1.10.4 through v1.10.14 notes are unchanged and still above this in both
files.
Until now the only record of what happened was `user_activity_logs`, which
stores non-GET 2xx operations with no bodies. When something failed you could
see that a counter went up, never what was sent or what came back.
This adds one queryable timeline covering both directions:
- inbound: every API call, including GETs and including 4xx/5xx, with the
user, client IP, status, duration and — redacted, size-capped — the request
and response bodies.
- outbound: every HTTP call the backend makes, tagged with who it went to
(ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME
diagnostics probe).
Outbound rows inherit the inbound request's id, so one operator action and the
CA/DNS calls it triggered read as a single trace: opening a failed "Request
Certificate" shows the exact POST /acme/new-order and the CA's 429 underneath.
Implementation notes:
- Capture is a pure-ASGI middleware that TEES the request and response streams
rather than draining them. `await request.body()` inside a BaseHTTPMiddleware
would consume the receive channel and break the raw-body agent heartbeat
handler. Registered last so it is outermost: it then sees the final
client-visible response and seeds correlation_id_context before the error
handler reads it.
- Rows are written by a batching background writer with a bounded queue, so the
request path never awaits the database and a saturated logger drops rows
visibly (surfaced on the page) instead of blocking. Redaction runs on the
writer, off the request coroutine.
- Secrets never land: headers are an allowlist with Authorization/Cookie kept
only as a presence marker; body keys and value shapes are redacted
(passwords, tokens, api_token, API keys, private-key PEMs, JWTs); the ACME
JWS request body is never stored, because a stored protected+signature pair
is a replayable credential — a summary is logged instead; DNS-provider errors
record only the exception type; the ACME HTTP-01 challenge endpoint is
excluded so key_authorization is never captured.
- Retention is operator-configurable in Settings -> Request Log: separate day
counts for successful and failed rows (7 / 30) plus a hard row cap (500k),
whichever is reached first. Pruned in batches under a Postgres advisory lock,
with the day counts bound as parameters, never interpolated.
- New permissions requestlog.read / requestlog.manage. super_admin and
security_admin get both, operator gets read, viewer gets neither.
Schema: one new table (request_logs) plus its settings seed, SCHEMA_VERSION
10 -> 11, auto-migrated. No existing table altered, no agent or rendered-config
change. Kill switches: REQUEST_LOG_ENABLED=false (middleware never registered)
or the `enabled` toggle in Settings.
Tests: 245 new (7 backend files + 1 frontend), full suite 1655 backend +
17 frontend passing.
- Add configuration guide for environment variables
- Add agent upgrade guide
- Use example.com instead of company-specific URLs
- No sensitive information included
taylanbakircioglu