3 Commits

Author	SHA1	Message	Date
taylanbakircioglu	bd4a50943f	fix(requestlog): bound queue memory, and stop the UI reporting things it cannot know ... Three hardening fixes with the same shape: a number that was true under the defaults and untrue at the edges. 1. QUEUE MEMORY WAS AN OPERATOR SETTING, NOT A LIMIT. The queue was bounded by ROW COUNT only, and how much a row weighs is `requestlog.max_body_bytes` - editable from Settings, documented ceiling 256 KB, and a row can hold that twice (request + response). Measured on the real dataclass with distinct buffers per row: defaults, 2 000 rows x 8 KB 33.9 MiB 3.3% of the 1 GiB pod limit max_body_bytes at its 256 KB ceiling 1003 MiB at the pod limit REQUEST_LOG_QUEUE_MAX at its ceiling 1695 MiB over the pod limit Both are reachable from in-range, documented values, and the drop warning advised "raise REQUEST_LOG_QUEUE_MAX" - so following the tool's own advice on a busy install could OOM the worker. REQUEST_LOG_QUEUE_MAX_BYTES (default 64 MiB) now caps the queue in bytes as well as in rows, whichever binds first, released as rows drain. Verified: with max_body_bytes at 256 KB the queue holds 7.5 MiB against an 8 MiB budget where it would otherwise have held 1003 MiB, and it accepts rows again as soon as the writer drains it. The warning text now names the setting that actually helps. 2. SINK COUNTERS ARE PER WORKER AND DID NOT SAY SO. The sink is a module global, so with UVICORN_WORKERS > 1 each process has its own queue and its own counters, and `GET /api/request-logs/stats` reports whichever worker happened to serve the request. The feature is sold on "a saturated logger drops rows visibly"; at 4 workers the visible number was a quarter of the truth. Labelled `"scope": "this worker only"` rather than aggregated - there is no cross-process channel here, and a number that looks fleet-wide but is not is worse than one that admits its scope. 3. AN EMPTY EXCLUDE LIST IS NOT APPLIED AS "LOG EVERYTHING". normalize_exclude_paths() falls back to the shipped defaults when the list comes out empty, which is the right call - it keeps the log viewer and the raw-body heartbeat endpoint excluded - but the UI kept displaying the empty list the operator typed, so the form showed a policy that was not in effect. The save handler now re-applies whatever the server actually stored (which also surfaces server-side clamping of every numeric field) and says plainly that the defaults were restored.	2026-08-15 11:04:31 +03:00
taylanbakircioglu	4e2d936c27	fix(requestlog): stop the table size from scaling with fleet size ... The row rate of `request_logs` was a function of how many nodes are installed, not of what anyone did. Counted from the agent loop in linux_install.sh, each agent's 30s cycle issues three logged calls - config, pending-requests, upgrade-status (the heartbeat is already on the default exclude list) - plus keepalived-config and keepalived-status every fifth cycle. That is ~9 800 rows/day per agent, essentially all of them 200s meaning "nothing changed". Measured on PostgreSQL 15 against the real DDL and all nine indexes, at 2 424 bytes/row: 20 agents ~196k rows/day 453 MB/day row cap reached in 2.5 days 200 agents ~2.0M rows/day 4.4 GB/day row cap reached in 6 hours 500 agents ~4.9M rows/day 11 GB/day row cap reached in 2 hours The cap holds, so nothing runs away - but it holds by deleting, and what it deletes is everything else. The shipped policy says 7 days of successes and 30 days of failures; on a 200-node fleet it delivers about six HOURS of both. The forensic record the feature exists for is evicted by polling noise, and the larger the installation the less history it keeps. `requestlog.capture_agent_success`, default FALSE: a SUCCESSFUL inbound call from an agent is not recorded. Failures always are, whatever the flag says - they are what an operator needs and they are rare, so they cost nothing. With this the table's size follows operator activity, and adding nodes does not shorten anyone's retention. Agent traffic is identified by header only, no database round-trip on the hot path: the installed agent sends `X-API-Key` and never `Authorization`, the UI sends a JWT and never an agent key. `generate-install-script`, the one endpoint that accepts either, classifies correctly under the same rule - an operator generating a script sends Authorization, a self-upgrading agent sends only the key. The result is stored in the existing `target` column, which already means "who was on the other end" for outbound rows and now means the same for inbound ones, so no schema change and the existing target index applies. Second half, and the reason this is one commit: `operator` holds `requestlog.read` because, per the migration that grants it, "operators debug failing applies and ACME orders". They could not. An apply fails on the NODE, and the node reports that over its own API key, so the row carrying the diagnosis has `user_id IS NULL` - and own-rows-only scoping hid it from exactly the role the grant was written for. Scoping now admits agent rows alongside the caller's own. Deliberately keyed on `target = 'agent'` rather than `user_id IS NULL`: anonymous traffic is not agent traffic, so failed logins and their usernames, and unauthenticated probes, stay admin-only. Verified end to end through the real middleware: a successful agent poll is dropped, a 422 from config-validation-failed is kept, operator and anonymous calls are unaffected, and flipping the setting on restores the old behaviour.	2026-08-15 11:04:30 +03:00
mustafa.ulukaya	ef26860df9	feat(logging): unified request/response log with configurable retention (v1.11.0) ... Until now the only record of what happened was `user_activity_logs`, which stores non-GET 2xx operations with no bodies. When something failed you could see that a counter went up, never what was sent or what came back. This adds one queryable timeline covering both directions: - inbound: every API call, including GETs and including 4xx/5xx, with the user, client IP, status, duration and — redacted, size-capped — the request and response bodies. - outbound: every HTTP call the backend makes, tagged with who it went to (ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME diagnostics probe). Outbound rows inherit the inbound request's id, so one operator action and the CA/DNS calls it triggered read as a single trace: opening a failed "Request Certificate" shows the exact POST /acme/new-order and the CA's 429 underneath. Implementation notes: - Capture is a pure-ASGI middleware that TEES the request and response streams rather than draining them. `await request.body()` inside a BaseHTTPMiddleware would consume the receive channel and break the raw-body agent heartbeat handler. Registered last so it is outermost: it then sees the final client-visible response and seeds correlation_id_context before the error handler reads it. - Rows are written by a batching background writer with a bounded queue, so the request path never awaits the database and a saturated logger drops rows visibly (surfaced on the page) instead of blocking. Redaction runs on the writer, off the request coroutine. - Secrets never land: headers are an allowlist with Authorization/Cookie kept only as a presence marker; body keys and value shapes are redacted (passwords, tokens, api_token, API keys, private-key PEMs, JWTs); the ACME JWS request body is never stored, because a stored protected+signature pair is a replayable credential — a summary is logged instead; DNS-provider errors record only the exception type; the ACME HTTP-01 challenge endpoint is excluded so key_authorization is never captured. - Retention is operator-configurable in Settings -> Request Log: separate day counts for successful and failed rows (7 / 30) plus a hard row cap (500k), whichever is reached first. Pruned in batches under a Postgres advisory lock, with the day counts bound as parameters, never interpolated. - New permissions requestlog.read / requestlog.manage. super_admin and security_admin get both, operator gets read, viewer gets neither. Schema: one new table (request_logs) plus its settings seed, SCHEMA_VERSION 10 -> 11, auto-migrated. No existing table altered, no agent or rendered-config change. Kill switches: REQUEST_LOG_ENABLED=false (middleware never registered) or the `enabled` toggle in Settings. Tests: 245 new (7 backend files + 1 frontend), full suite 1655 backend + 17 frontend passing.	2026-08-11 02:36:03 +03:00