Until now the only record of what happened was `user_activity_logs`, which
stores non-GET 2xx operations with no bodies. When something failed you could
see that a counter went up, never what was sent or what came back.
This adds one queryable timeline covering both directions:
- inbound: every API call, including GETs and including 4xx/5xx, with the
user, client IP, status, duration and — redacted, size-capped — the request
and response bodies.
- outbound: every HTTP call the backend makes, tagged with who it went to
(ACME/Let's Encrypt, Cloudflare, GoDaddy, HAProxy stats, agents, the ACME
diagnostics probe).
Outbound rows inherit the inbound request's id, so one operator action and the
CA/DNS calls it triggered read as a single trace: opening a failed "Request
Certificate" shows the exact POST /acme/new-order and the CA's 429 underneath.
Implementation notes:
- Capture is a pure-ASGI middleware that TEES the request and response streams
rather than draining them. `await request.body()` inside a BaseHTTPMiddleware
would consume the receive channel and break the raw-body agent heartbeat
handler. Registered last so it is outermost: it then sees the final
client-visible response and seeds correlation_id_context before the error
handler reads it.
- Rows are written by a batching background writer with a bounded queue, so the
request path never awaits the database and a saturated logger drops rows
visibly (surfaced on the page) instead of blocking. Redaction runs on the
writer, off the request coroutine.
- Secrets never land: headers are an allowlist with Authorization/Cookie kept
only as a presence marker; body keys and value shapes are redacted
(passwords, tokens, api_token, API keys, private-key PEMs, JWTs); the ACME
JWS request body is never stored, because a stored protected+signature pair
is a replayable credential — a summary is logged instead; DNS-provider errors
record only the exception type; the ACME HTTP-01 challenge endpoint is
excluded so key_authorization is never captured.
- Retention is operator-configurable in Settings -> Request Log: separate day
counts for successful and failed rows (7 / 30) plus a hard row cap (500k),
whichever is reached first. Pruned in batches under a Postgres advisory lock,
with the day counts bound as parameters, never interpolated.
- New permissions requestlog.read / requestlog.manage. super_admin and
security_admin get both, operator gets read, viewer gets neither.
Schema: one new table (request_logs) plus its settings seed, SCHEMA_VERSION
10 -> 11, auto-migrated. No existing table altered, no agent or rendered-config
change. Kill switches: REQUEST_LOG_ENABLED=false (middleware never registered)
or the `enabled` toggle in Settings.
Tests: 245 new (7 backend files + 1 frontend), full suite 1655 backend +
17 frontend passing.
Registers a third pluggable DNS provider for ACME DNS-01 alongside Manual and
Cloudflare. Credentials are an API Key + Secret pair; leaving the Secret blank
sends the Key as a Personal Access Token (Bearer), which is the migration path
as GoDaddy retires the sso-key scheme.
GoDaddy's Domains API v1 has no per-value TXT write: PUT on a record set
replaces every value at that name. A certificate covering example.com and
*.example.com publishes two different TXT values at the same
_acme-challenge.example.com, so add/remove are read-modify-write - read the
current set, merge, put the whole list back - with empty-data tombstone rows
filtered out (they are rejected on echo) and DELETE used for the last value,
since PUT with an empty array is rejected.
The zone-wide sibling endpoints (.../records/TXT and .../records) would wipe
SPF/DKIM/DMARC and the whole zone respectively, so the record path is built in
one place that refuses an empty or dot segment. An unreadable record-set read
fails closed rather than being treated as an empty set, because the PUT that
follows would otherwise destroy the coexisting values.
Zone lookup walks name suffixes probing the records API rather than the domain
listing, so zones delegated to GoDaddy nameservers resolve and accounts that
are rejected from the domain-details endpoint still work. Credential and
eligibility failures during the walk surface instead of being reported as
"no managed domain".
Provider errors are sanitized at the single point where GoDaddy-supplied text
enters a message, since those strings are persisted to order events and shown
in the UI. No new dependency, no schema change, no frontend change - the
credential form is rendered from the provider schema.
Follow-up fixes for the DNS-01 feature reported on #35:
- Cloudflare: sanitize the API token (strip surrounding quotes + any non
token68 chars) so a pasted token with quotes/spaces no longer fails with
"Invalid request headers"; verify-on-save shows a precise hint when it
cleaned the input. Covers the automated orchestrator path too.
- ZeroSSL/Google EAB: enter the EAB Key ID and HMAC Key per-account in the
Register Account dialog (falls back to the global Settings value when blank);
base64-validate the HMAC key; humanize the externalAccountRequired failure;
and preserve the deliberate 409/422 instead of downgrading them to 400.
- Apply Management: cluster ACME enable/disable changes now show under a
dedicated "ACME Challenge Routing" section, are counted in the Apply/Reject
dialogs, and Apply/Reject All process them (previously "Rejected 0 HA/VIP
change(s)") - consistent with every other entity. Reject rolls acme_enabled
back to the original via ORDER BY created_at ASC over the snapshot chain.
- getErrorMsg surfaces field-level validation messages.
Backward compatible (additive / strict superset; HTTP-01 unchanged).
Addresses #35.
Add ACME DNS-01 (TXT-record) validation alongside the existing HTTP-01,
for internal/isolated clusters with no public port 80 and for wildcard
certificates. Opt-in via a global kill-switch (default off); HTTP-01 is
byte-for-byte unchanged, with zero agent or rendered-config changes.
- Pluggable DNS provider interface (Manual + Cloudflare). Per-account
credentials are Fernet-encrypted at rest, verified on save, and never
returned by the API or written to logs/events/error_detail.
- Non-blocking per-cycle orchestrator: publish (CAS) -> propagation grace
(across cycles, no in-loop sleep) -> respond -> finalize/download, with a
bounded fresh-order retry chain (1 original + 3 retries) on propagation lag.
- Manual flow: user publishes the TXT record and confirms; manual DNS-01
cannot auto-renew unattended (auto-renew forced off and surfaced in the UI).
- Migration v8: additive, idempotent columns on letsencrypt_accounts/orders
and acme_challenges, plus a new letsencrypt_account_dns_credentials table.
- Challenge-type-aware diagnostics (port80/routing/DNS checks skipped for
DNS-01) and a DNS-01 event timeline in the order detail.
- Frontend: DNS-01 account + credentials management, cert wizard adaptation,
order-detail TXT records + verify, orders/renewal Method columns, and a
Settings kill-switch. README, release notes, and API docs updated.
Implements #35.
mustafa.ulukaya