xarmian
c73584088f
* fix(watchevents): detect a half-open Redis connection with a bus heartbeat (BUG-2769) internal/watchevents had the same defect as internal/events did, by the same mechanism: ChannelWithSubscriptions on a connection whose go-redis health check only writes. PubSub.Ping calls writeCmd and returns without reading a reply (v9.22.0), so a route that stops carrying traffic without closing is invisible — the instance blocks on a read forever while its replay buffer goes on looking complete. Named as a class sweep in BUG-2738's filing and deferred there. It became load-bearing when that unit shipped: docs/deployment.md told operators the gap was "closed on the activity stream and still open on the watch stream". This diff falsifies that, which is why the prose sweep is part of it. THE PORT IS SMALLER THAN THE ORIGINAL BY DESIGN. This bus holds ONE process-wide subscription created in its constructor, off any request path, so none of BUG-2747's establishment machinery exists to interact with: no per-workspace map, no establishment record, no single-establisher wall, no concurrency cap, no bounded-parallel recovery, and no per-workspace cycle scoping. Cost is flat too — one frame per instance per interval regardless of workspace count. NO COMPANION COUNTER, and that was CHECKED rather than inherited. internal/events needs pad_event_subscription_cycled_total because its dropWorkspaceCoverage returns early when a workspace has no buffer, so the reset reason under-reports the early-wedge case. dropCoverage here has no such branch: it replaces the buffer and reports unconditionally, so idle_timeout is a complete count on its own and a second metric would be a number needing to be explained against its neighbour for no signal. THE RECEIVE LOOP NOW OWNS ITS SUBSCRIPTION AND CONTEXT. A cycle replaces the subscription under a running bus, and the loop reading the old one must tell "I was replaced" from "the client died" — the second logs an ERROR and moves a counter documented to mean the instance has gone deaf. The cycle cancels that loop's own context before closing its PubSub, so it leaves by the quiet door. Its own test. I PORTED A FLAW ALONG WITH THE STRUCTURE, and the wiring test caught it: both maintenance halves shared one kick channel, so whichever goroutine was waiting consumed it and the other stayed on the stale cadence. internal/events' mutation matrix found exactly that (M11c) and fixed it; the fix did not come across. That is the contamination hazard this port's grounding warned about, in its literal form, caught by the CONVE-19 test rather than by review. Two more found by mutation, both missing tests rather than missing code: nothing asserted that ordinary traffic keeps the instance alive (removing the per-frame stamp survived, because every other test drives idleness through the clock), and nothing asked for a SECOND detection (a replacement inheriting stale stamps gives a detector that works exactly once, which is worse than one that never runs because it looks like it works). The second needed a direct assertion on the install stamps, because the behavioural route re-stamps the field it was meant to be testing. Trio in one commit as required: reason enumeration, the pad_watchevents_sequence_resets_total Help string, and docs/deployment.md — plus the two BUG-2738 sentences this falsifies and a new section explaining how the watch bus differs from the activity one. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(watchevents): fence stragglers and re-validate before the drop (codex r1) Three findings, and two of them are BUG-2738 fixes I again failed to bring across with the structure. That is now three times in one port: the shared kick channel, the stale idle decision, and the missing generation. The mechanism is the same each time — I ported what the code DOES and not what its review history taught it, and each was caught by a test or a reviewer rather than by me reading the source I was copying. STALE IDLE DECISION. cycleIfIdle decided under one lock and tore down under another; a heartbeat or notification arriving between them left a demonstrably alive subscription being dropped and every client on the instance resynced for nothing. BUG-2738 fixed exactly this at its round 11. Re-validated immediately before the drop, with a positional seam so a test can land the recovery inside the window rather than racing it. NO GENERATION FENCE. Cancelling a receive loop and closing its PubSub does not JOIN the goroutine, and go-redis's channel is buffered, so a frame from a replaced subscription could still stamp the replacement's liveness, append to its buffer, or drop its coverage. On a wedged route that is the worst direction: the dead connection's buffered tail suppressing the detector for its successor. One check at the top of the frame handler covers all three, because the three must agree about whether a frame belongs to the live subscription. The probe stamp is fenced separately, since a slow publish can outlive the subscription it was sent for. A COPIED COST PARAGRAPH THAT CONTRADICTED ITS OWN SECTION. The activity bus's "each workspace has its own subscription, N frames per interval" text sat below the new watch-specific section saying the opposite. Retitled and moved above it. FOUR INSTRUMENT DEFECTS ON THE WAY, all found by mutation: - Nothing asserted ordinary traffic keeps the instance alive — every other test drives idleness through the clock, so removing the per-frame stamp survived. - Nothing asked for a SECOND detection, so a replacement inheriting stale stamps gave a detector that works exactly once — worse than one that never runs, because it looks like it works. Needed a direct assertion on the install stamps, since the behavioural route re-stamps the field under test. - The generation tests asserted the PREDICATE, not that the loop calls it. - And that wiring test could not discriminate on a frozen clock, where a stamp writes the value already there. It advances the clock first now. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(watchevents): make the generation fence atomic with what it guards (r2) Two P1s, both mine, both the same shape: a check in one lock acquisition and the mutation it guards in another. THE FENCE WAS NOT ATOMIC WITH ITS MUTATIONS. One check at the top of the frame handler read well and guarded nothing reliably — a replacement between that check and stampLastSeen / fanOutFromRedis / dropCoverage let a straggler through to any of them. The generation now travels TO each mutation and is re-checked under the same lock that mutates. A stale notification entering the replacement's buffer is the worst of the three: it makes the instance vouch for a span it never received, which is the false coverage claim this whole family exists to remove. THE OLD GENERATION STAYED CURRENT ACROSS THE REPLACEMENT. subGen was incremented only after the new subscription was confirmed, leaving the cancel, the close, the dial and a round trip during which the OLD generation still passed every fence. Retired at teardown now, so during resubscribe NO generation is current and a late frame is ignored everywhere. That also makes the failure path honest: the "no notifications until restarted" log was false — no generation is current, so the next idle tick tries again. Revalidation and the drop are now ONE critical section rather than two, for the same reason at one level down: a frame arriving between them was silently discarded by a drop already decided on. Also: phase 1 no longer starts the maintenance goroutines, and the watch bus's phase is logged at startup — an operator cannot read an absence of idle_timeout without knowing whether the detector was running, and the two flags are independent. DOCS still described the workspace model in the section that claims to cover both buses: one heartbeat "per subscribed workspace", a phase table naming only PAD_EVENTS_HEARTBEAT, and coverage described as a workspace's. Generalised. Two more instrument gaps, both found by mutation: nothing asserted a straggler cannot enter the replacement's BUFFER (only the stamp was covered), and the phase-1 goroutine gate is untested by design — removing it changes no behaviour, only goroutine count, and the only assertion is a census that would be flaky here. Said out loud rather than left to look like coverage. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * test(watchevents): prove each generation fence on its own Round 3's fix put a generation check in each of the four places a frame from a replaced subscription can mutate shared state, rather than one check at the top of the receive path — a check in one lock acquisition and a write in another is a TOCTOU, which is what codex blocked. Four checks means four mutations, and the matrix found the first pass of tests could not tell them apart: removing the append's check, or the coverage drop's, left every test green. Not because the guards were redundant — because no test drove those paths with a stale generation. The straggler tests all enter through fanOutFromRedis, whose own guard returns early and hides the one below it, and nothing at all drove dropCoverageForGen with a straggler. So the fences are asserted one at a time, each through the entry point that actually reaches it: epoch bookkeeping fanOutFromRedis with a foreign epoch — the loudest of the four, since an accepted straggler would rewrite the id space and resync every client on the instance buffer append fanOutLocally directly, under the guard above it coverage drop dropCoverageForGen, previously undriven liveness stamp stampLastSeen, which would otherwise let a dead socket's traffic hold detection open Each fails against removal of the single check it names (M16/M17/M19 and the existing stamp mutation), and the four together still pass the end-to-end straggler tests unchanged. Refs BUG-2769 * test(metrics): prove the two new watch signals reach the registry Both were wired and neither was asserted at the metrics layer, which is where docs/deployment.md's claims about them actually live. A reason or a callback that never reaches the registry is a runbook pointing at a series that does not exist, and nothing in internal/watchevents can catch that — its observer is an interface, satisfied by a test double. pad_watchevents_heartbeat_publish_failures_total incremented six times, a count no other assertion in that test uses, so a callback wired to the wrong counter cannot land on the right number by coincidence. Fails when the increment is pointed at a neighbour. sequence_resets_total{reason="idle_timeout"} asserted with the literal label, alongside the four spellings already pinned there and for the same reason BUG-2739's rename left that test behind. Fails when the constant drifts. Also corrects the shared "what happens if you run them out of order" paragraph, which moved under a heading covering both buses while still describing only one: it said the frame travels on "the workspace's event channel" and that an un-upgraded instance resyncs "for every workspace", neither of which is the watch bus, where there is one channel and one buffer per instance. The blast radius differs in scale between the two and the paragraph now says so. Refs BUG-2769 * docs(watchevents): correct three counted claims that stopped being true All three said "three" where the code now has four, and each was accurate when written — the fourth fence (the epoch bookkeeping in fanOutFromRedis) was identified after them, in the pass that found the matrix could not tell the guards apart. That is the whole failure mode: a count is a claim, and a claim written before the last change is wrong afterwards with nothing to notice it. Two of the three sat inside a comment ABOUT how carefully the guards were enumerated, and one names them now instead of counting them, so the next site added has to appear in the list or contradict it visibly. Found by sweeping the branch diff for counted prose rather than by rereading, which is what had already missed them twice. Refs BUG-2769 * test(config): close the other half of the two-flag independence claim The flag tests asserted PAD_WATCH_HEARTBEAT does not move EventsHeartbeat and stopped there, while the comment above them and the deployment doc both claim the two buses roll INDEPENDENTLY. That is a biconditional and one leg does not establish it: a Load() that pointed PAD_EVENTS_HEARTBEAT at both fields passed everything. Now both directions are asserted, and the events leg checks its own premise first, so a fixture that stopped setting the flag fails as a fixture rather than as a pass. Also pins env-over-file precedence for the watch flag, in the direction that actually matters: PAD_WATCH_HEARTBEAT=false over watch_heartbeat=true in config.toml. That is the rollback for a bad phase-2 flip, and an operator reaching for it mid-incident cannot be editing a file on every host. Mutation matrix, each detected: the env var wired to the neighbouring field, the env var never read at all, and the toml tag dropped. Refs BUG-2769 * test(watchevents): fix five tests that passed for the wrong reason Codex round 4 went at test honesty rather than correctness and found no BLOCK, but it found five assertions that hold whether or not the thing they name works. Each is now driven through the path it claims, and each was mutation-checked against the specific defect it exists to catch. the malformed-frame contract only ever called isWatchHeartbeat. The predicate can be perfect while the receive loop routes every "hb|…" payload to the ignore arm without asking it, which is the defect, and the test's name promises coverage ends — a claim about the loop. Now published on the real channel, with a well-formed frame as the control so the assertion cannot be satisfied by a loop that finds everything undecodable. the receive-loop wiring test published, slept 300ms, and asserted nothing had changed. A loop that stalled or never started satisfies that perfectly. There is no natural signal to wait on instead, because a frame the fence refuses is by design invisible — hence a seam that fires after the loop handles a frame whichever arm it took. Bounded, so a stalled loop fails with a message rather than a package timeout, and followed by a control that the same loop still accepts a frame whose generation matches. the quiet-exit test asserted only that no loud exit was reported, which a replaced goroutine that never exits at all also satisfies — a leak, and the worse outcome. Now joins the loop first via a process-wide live-loop count, then checks the counter, so it is a statement about a goroutine that has finished. the maintenance-loop wiring test claimed both halves and observed a heartbeat, which a loop that started only the publisher passes. The idle half cannot be proved there at all: against a live miniredis this bus's own heartbeats come back and refresh liveness every cadence, so wedging it with the loop running is a race against the publisher — which is what my first fix for this turned out to be, flaky at 2 in 3. Renamed to what it proves, pointing at the blackhole end-to-end test, which drives the scanner for real and detects both mutations. the straggler test never delivered a straggler. It incremented subGen by hand, called isCurrentGen, and compared an unchanged timestamp without touching a mutation path — green with every fence removed. Deleted rather than repaired: the four-way per-fence test added earlier covers it properly, and isCurrentGen went with it. Plus two ordering changes in Close/resubscribe that ARE NOT fixes for an observed race, and say so in the test. Making b.pubsub reassignable made Close's unlocked read of it look wrong, and resubscribe's wg.Add outside the lock look like it could land after Close reached Wait. Both windows turn out to be shut already by resubscribe's b.closed check, which sits under the same acquisition as the count — reverting either fix leaves the new Close-during-cycle test green. Kept as defence because the invariant they lean on is three functions away, and documented so nobody later reads them as evidence of a bug that existed. Also corrects the metric help and two comments that said an idle cycle "replaced the connection" when it attempts a replacement that can fail; the deployment doc already said attempted. And the deployment doc's rollback, frame-validation, what-to-watch and startup-log paragraphs, all of which moved under a heading covering both buses while still describing only the activity one. Refs BUG-2769 * refactor(watchevents): drop an always-empty return and the branch reading it dropCoverageIfStillIdle returned (string, bool) where the string was never anything but empty — the reset it reports goes out through the pending/flush path inside the lock, so the caller's `if report != ""` was unreachable. A second reporting path that exists in the signature and never fires is a thing a later change wires up by accident. Refs BUG-2769 * fix(watchevents): a failed re-dial retries without re-dropping coverage Codex round 5, on behaviour across a full Redis outage. No BLOCK; this was its one P2 and it is real. The probe-failure suspension does not cover this case, and the reason is worth stating because the suspension looks like it should. Suspension asks "did our last probe get through", and that can be YES with the route already gone: the last successful publish stamps lastProbeOK, Redis dies before that frame comes back, and lastSeen stays behind it. From there both timestamps are frozen — the probe fails so nothing stamps lastProbeOK, nothing arrives so nothing stamps lastSeen — and the cycle's precondition stays true for the whole outage. Every pass then dropped coverage, announced to every subscriber, and re-dialled. Only the re-dial is owed. The second drop empties an already-empty buffer and re-announces a hole every subscriber has been told about, and it moves pad_watchevents_sequence_resets_total{reason="idle_timeout"} once per cadence — so a five-minute outage read as ten incidents on the series operators are told to alert on. cycleIfIdle now has a retry-only arm ahead of the decision, entered when there is no subscription at all, and the teardown clears b.pubsub / b.subCancel so that state is representable. Clearing them also stops Close closing an already-closed PubSub a second time. Two tests, discriminating in OPPOSITE directions, because the obvious fix for the noise is to suspend the pass and that would trade a noisy outage for one the instance never returns from — retrying the dial IS the recovery path: three passes with Redis away one reset, not three Redis returns after a failed pass the subscription is re-established and the counter does not move again Matrix: removing the retry arm, making it return without retrying, and leaving the torn-down subscription in place are each detected, the middle one only by the recovery test. internal/events has no equivalent defect. Its teardown deletes the workspace's subscription entry, so its next scan finds nothing live and abandons; recovery there runs off the request path. Refs BUG-2769 * fix(watchevents): only one caller may install a replacement subscription Codex round 6, verifying round 5's fix. No BLOCK; this was its P2. Both the cycle and its new retry arm dial with the lock RELEASED, which is deliberate — a Redis round trip under the bus's hot mutex would stall every fan-out on the instance — so two passes can each find no subscription and each dial one. Installing both is wrong twice over: two receive loops would run on the SAME generation, so both accept every frame and each notification is processed twice, and the loser's PubSub would be untracked, closed by nothing including Close. The install is what needs serialising, not the dial, so the loser discards its own connection under the lock rather than the two racing to overwrite b.pubsub. Only the idle scanner calls this today, so this guards an invariant rather than fixing an observed fault. Written down because the invariant lives in a different file from the code relying on it, and because the failure is silent duplication rather than a crash. The test races two resubscribes through the install seam. Two details it needed, both found by running it rather than reading it: the loop count is incremented INSIDE the goroutine, so sampling it right after the constructor returns reads zero — the first version did, and measured every later count against that wrong baseline. It waits for the loop now. the seam release is deferred, because without it the guard's mutation parks both callers in the callback, Close waits on receive loops that cannot start, and the detection arrives as a package-wide hang with no message. That is how the mutation first appeared to pass. Also completes the idle_timeout reason in three comment/help sites that still enumerated four reasons and said "the last two" — the same stale count corrected in the observer contract earlier on this branch, missed in its neighbours because I fixed the one the reviewer named instead of grepping for the claim. Refs BUG-2769 * test(watchevents): count installs instead of waiting for one that never comes Codex round 7 returned no BLOCK and no P2 on the production code, and two NITs on what round 6 added. Both are real. The concurrency test synchronised on a WaitGroup expecting BOTH callers to reach the install seam. Only the winner does — that is the property under test — so in the passing case the goroutine waiting on it blocks forever. A leak inside a test written to prove a leak does not happen is not a shape to leave standing. An atomic the abandoning caller never touches carries the same information and blocks nobody, and it removes the release channel and its deferred close along with it. The final assertion also moved off liveReceiveLoops and onto that count. A loop starts AFTER its install, so reading the loop count can catch a second caller's goroutine before it has begun and see the passing value on a failing run. Both callers have returned by the time the install count is read, so it is final. Detection over ten runs with the guard removed: 10/10, where the loop-count version was a race against a goroutine's first instruction. Also softens the retry arm's log line. It said the instance receives no notifications until an attempt succeeds, which is true for today's single scanner and stale the moment there are two: one caller's dial can fail while another has already installed. It now claims only what the failing call knows. Refs BUG-2769 * test(watchevents): hold both callers at the window, and say what that misses Codex round 8's P2, on the test the previous commit rewrote. Starting two goroutines from a start gate makes overlap likely and guarantees nothing: one can finish resubscribe before the other begins, so the window the install guard closes need never have been open. A seam at the dial/install boundary — connection dialled, lock not yet taken — lets both callers announce their arrival and wait for each other. Now the window is open by construction rather than by luck, and the test fails as a fixture if only one caller ever reaches it, instead of passing on evidence it never gathered. AND IT STILL DOES NOT DETECT EVERYTHING, which the test now says in place of leaving it implied. Measured: guard removed entirely 10 runs, 10 detected guard checked in its own acquisition, then 10 runs, 0 detected the lock retaken to install The second is the regression round 8 asked about, and catching it would mean landing the second caller inside a check-to-install gap that exists only in the mutant — there is nothing to yield on there, and no seam can be placed in code that is not written. So this test covers "a guard exists", not "the guard is in the right critical section". The latter is held by the comment at the guard and by review, and a test comment claiming otherwise would be worth less than the honest note. Refs BUG-2769 * fix(watchevents): make the frame seam and the cycle log tell the truth Codex round 9 was asked whether this should merge and said hold for a cleanup pass. Five findings, no correctness blocker, and every one of them a claim that had stopped matching the code. the frame seam did not fire for every arm, though its comment said so. The arms that decline to act — a heartbeat, an undecodable payload, an unsubscribe confirmation — were `continue` statements, which skipped everything after the switch. A test waiting on the seam for one of those frames would have HUNG rather than failed, which is the worst way to find this out. The switch is now its own method so every arm ends the frame by returning, and a test drives one frame per publisher-reachable arm and counts three. Detected against restoring the skip. the idle-cycle warning was emitted before the revalidation that can abandon the cycle, so it could announce coverage ending and resumes answering sync_required for a subscription that was then left alone — a log line with no counter behind it, and an on-call hunting a bug that is not there. internal/events learned this at its own round 6; the reason did not come across with the port. Moved after the decision is final, still saying "attempting" to replace because the resubscribe can fail. the quiet-exit test sampled liveReceiveLoops instead of waiting for it, so its "the replaced loop left" assertion could be satisfied by a loop that never ran. Same defect fixed in the sibling concurrency test a commit earlier and missed here, because I looked at the test the reviewer named rather than at the pattern. Latent rather than observed: sampling survives 10 runs, so this removes a possibility. the probe-failure log and metric help called an errored Publish a failure to publish. A returned error can also mean the reply was lost after Redis accepted the frame, so the honest claim is that the probe is UNCONFIRMED. It changes no behaviour — an unconfirmed probe is not evidence about the receive path either, so detection suspends the same way — but an operator reading the counter should not be told more than the instance knows. the deployment doc said the watch stream differs in "three things" and listed four, the fourth being the bullet I added last round. Third instance of that species on this branch; the count is gone rather than corrected. Refs BUG-2769 * docs(watchevents): stop one unconfirmed probe standing in for a broken path Codex round 10 confirmed four of round 9's five fixes and held the fifth as partial. It was right on all three residual sites. Renaming the condition to "could not confirm" did not fix the sentences downstream of it. The log still said silence cannot be read as a finding "when we could not ask" — but we may well have asked, and lost only the answer. And both the metric help and the observer contract said an instance in this state "is also failing to deliver its own notifications to every other instance", which is a conclusion about the outbound path drawn from a single call that did not come back. The inference is sound at a SUSTAINED rate and worthless at one increment, so both now say which is which. That distinction is the whole value of the counter to an on-call: a blip is a lost reply, a rate is a broken path, and the same wording for both makes the first look like the second. No behaviour change. An unconfirmed probe suspends detection exactly as a definite failure does, because it is not evidence about the receive path either way. Refs BUG-2769 * docs: sweep the BUG-2738 prose this change makes false BUG-2738 shipped documentation that describes the watch stream as still carrying the half-open defect. Merging this makes those sentences wrong, and I flagged the sweep as owed twice during the groundwork and then did not do it — the lead caught that the package said nothing about it. Five sites, each re-read after editing rather than grepped for, because grepping for a phrasing I chose is how I have twice verified a sweep that had not landed: the residual enumeration opened "One gap remains everywhere, and a second remains on the watch stream only", then described one gap and said it was open on both. The second WAS the half-open case. Now states one gap, on both streams, and says where the second went. the half-open paragraph already said "closed on both streams" — the one site I had fixed — but omitted that each half is behind its own phase-2 flag, so a reader takes it as closed on their deployment when it is closed only once they turn it on. "A third residual" counted the item it followed. With the second gone the ordinal was wrong; it does not need one. "these two gaps" in the closing sentence, same arithmetic. the pad_event_subscription_cycled_total row told an operator to read heartbeat_phase off the startup log. There are now two such fields on two lines under two flags, and only one bears on that counter. It names the line. No code change; suite 28/28 and lint 0 re-run because the branch is under review and a docs commit that skips them is a commit nobody checked. Refs BUG-2769
79 KiB
Deployment Guide
Pad is a single Go binary with an embedded web UI. It supports SQLite (default) for single-node deployments and PostgreSQL + Redis for production multi-node setups.
Architecture
┌─────────────────┐
│ Reverse Proxy │
│ (Caddy/nginx) │
└────────┬────────┘
│ :443
┌────────▼────────┐
│ Pad │
│ Go binary │
│ (web UI + API) │
└──┬──────────┬───┘
│ │
┌────────▼──┐ ┌───▼────────┐
│ PostgreSQL │ │ Redis │
│ (storage) │ │ (pub/sub) │
└────────────┘ └────────────┘
- Pad serves the REST API and embedded SvelteKit web UI on a single port (default: 7777)
- PostgreSQL stores all data (workspaces, items, users, activity). SQLite works for single-node.
- Redis carries real-time events, watch/push notifications, and the shared session-presence registry across multiple Pad instances. Optional for single-node.
Quick Start with Docker Compose
# Clone the repo
git clone https://github.com/PerpetualSoftware/pad.git
cd pad
# Start everything (Pad + PostgreSQL + Redis)
docker compose up -d
# Check status
docker compose ps
# View logs
docker compose logs -f pad
Access the web UI at http://localhost:7777. On first visit, you'll be prompted to create an admin account.
Production Docker Compose
# Use the production overlay for resource limits and secure settings
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
Edit docker-compose.prod.yml to set your domain, email credentials, and database password.
Environment Variables
All configuration is via environment variables or a config file (~/.pad/config.toml / /data/config.toml).
Core
| Variable | Default | Description |
|---|---|---|
PAD_HOST |
127.0.0.1 |
Listen address (0.0.0.0 for Docker/production) |
PAD_PORT |
7777 |
Listen port |
PAD_URL |
— | Public-facing base URL (e.g., https://pad.example.com). Used for invitation, password-reset, and share-link emails. Required when PAD_HOST=0.0.0.0 — otherwise emailed links point at http://0.0.0.0:port and are unreachable to recipients. |
PUBLIC_URL |
— | Alternative to PAD_URL using the generic env-var convention. Server-side only — does not affect CLI mode, does not influence the CLI's API endpoint, and is not persisted to config.toml. Precedence: PAD_URL > PUBLIC_URL > constructed http://host:port. |
PAD_DATA_DIR |
~/.pad |
Data directory for SQLite DB, logs, and config |
PAD_LOG_LEVEL |
info |
Log level: debug, info, warn, error |
PAD_MODE |
local |
Mode: local, remote, cloud |
Database
| Variable | Default | Description |
|---|---|---|
PAD_DB_DRIVER |
sqlite |
Database driver: sqlite or postgres |
PAD_DB_PATH |
~/.pad/pad.db |
SQLite database path (ignored when using PostgreSQL) |
PAD_DATABASE_URL |
— | PostgreSQL connection string (required when PAD_DB_DRIVER=postgres) |
Real-time Events
| Variable | Default | Description |
|---|---|---|
PAD_REDIS_URL |
— | Redis URL for cross-instance pub/sub and the session-presence registry. Without Redis, SSE events, watch notifications, and session presence are all in-process only. |
PAD_REDIS_NAMESPACE |
— | Scopes every Redis key and channel to one installation. Set it when two Pad installations share a Redis endpoint. Unset means the historical names; a whitespace-only value is rejected at startup rather than treated as unset. |
PAD_SSE_MAX_CONNECTIONS |
1000 |
Maximum streaming connections per instance, across both /api/v1/events and /api/v1/events/stream |
PAD_SSE_MAX_PER_WORKSPACE |
100 |
Per-workspace maximum connections on /api/v1/events, per instance |
PAD_SSE_MAX_PER_USER |
50 |
Per-user maximum streaming connections across both endpoints, per instance |
PAD_EVENTS_PUBLISH_EPOCH |
false |
Phase 2 of the event ID-space migration: publish the <epoch>|<id>|<json> wire form. Only set this once every instance runs a binary that accepts it — see Event ID-space migration below. Ignored without Redis. |
PAD_WATCH_HEARTBEAT |
false |
Phase 2 of half-open detection on the watch stream (/api/v1/events/stream). Independent of PAD_EVENTS_HEARTBEAT — the two buses hold different connections with different fates — and rolled the same way, phase 1 everywhere first. Ignored without Redis. |
PAD_EVENTS_HEARTBEAT |
false |
Phase 2 of the half-open-connection detection rollout: publish a bus-internal liveness frame on each subscribed workspace channel every 30s. Only set this once every instance runs a binary that recognises it — see Half-open connection detection below. Setting it early makes every un-upgraded instance resync all its clients every 30 seconds. Ignored without Redis. |
Streaming connection limits
Pad has two SSE endpoints and they share one budget. /api/v1/events is
workspace-scoped (the web UI's activity stream); /api/v1/events/stream is
user-scoped (agent watch notifications, pad watch --stream). A held
connection costs a goroutine and a bus subscription whichever one opened it —
and, on the watch stream only, a session-presence registration in shared Redis —
so PAD_SSE_MAX_CONNECTIONS and PAD_SSE_MAX_PER_USER bound them together. Only PAD_SSE_MAX_PER_WORKSPACE
is endpoint-specific, because the watch stream has no workspace to count
against.
Upgrading:
PAD_SSE_MAX_CONNECTIONSpreviously bounded/api/v1/eventsalone. It now covers both, so a tuned value may be reached sooner than before. The server logs the effective limits at startup (Stream connection limits)./api/v1/events/streamhad no limit at all before this change; if you run many agent sessions per user, checkPAD_SSE_MAX_PER_USERagainst your fleet size.
A refused connection is 429 with code sse_limit_exceeded and a Retry-After
header. The CLI monitor (pad watch --stream) treats it like any other non-200
and backs off (5s, growing linearly, capped at 5 minutes), so refusal does not
produce a reconnect storm. pad project watch is interactive and exits with an
actionable message instead.
Browsers do not back off. The web UI's activity stream uses
EventSource, which retries on its own fixed schedule and cannot see the status code or theRetry-Afterheader — so a refused browser tab reconnects roughly every few seconds until capacity frees up. SizePAD_SSE_MAX_CONNECTIONSwith that in mind: reaching it does not shed load from browser clients the way it does from the CLI. Tracked as BUG-2733.
The per-user limit applies to every caller. On /api/v1/events, callers with no
resolved user — a legacy workspace-scoped token, or the fresh-install window
before the first admin exists — are bounded per workspace instead, at the same
number, so two legacy tokens for one workspace share a bucket.
/api/v1/events/stream has no equivalent case: it requires a resolved user and
answers 401 without one.
All three limits are PER INSTANCE, not deployment-wide. They are enforced
in-process; there is no shared counter. A three-replica deployment with
PAD_SSE_MAX_CONNECTIONS=1000 admits up to 3000 connections in total, and a
single user can hold PAD_SSE_MAX_PER_USER connections on each replica. Size
them per pod and multiply by replica count for the deployment ceiling. Watch
pad_stream_connections_active (per instance) rather than inferring the total
from the configured number.
Redis configuration notes
One namespace per installation, or one endpoint per installation. Every
Redis key and channel Pad uses carries PAD_REDIS_NAMESPACE when it is set:
pad:<namespace>:events:…, pad:<namespace>:watchevents…,
pad:<namespace>:session:…. When it is unset the names are the historical flat
ones, so upgrading changes nothing.
Two installations sharing one Redis endpoint without distinct namespaces cross-feed notifications and merge their session-presence registries. Selecting different logical DB numbers only half helps: ordinary keys are DB-scoped, so the presence registries stay separate — but Redis pub/sub is not namespaced by DB at all, so both buses cross-feed regardless. The practical exposure is a cloned database (a staging environment restored from a production dump), because delivery is filtered on user id and user ids are per-installation UUIDs; for that case it is a genuine cross-tenant leak.
Changing the namespace on a running deployment is a cutover, not a tweak. Set it before going multi-installation rather than after, and take a brief maintenance window if you can. Three things to know:
It partitions a rolling upgrade. Replicas with the namespace set and replicas without it do not share pub/sub channels, counters, or the presence registry — they behave as two separate installations for as long as the rollout takes.
GET /api/v1/sessionsanswers differently depending on which replica handles it, and a session-targeted push aimed across the partition is skipped (reported honestly asdelivered_sessions: 0, but not delivered). Roll all replicas together, or accept a split for the duration.Rolling BACK re-creates the split unless the namespace is unset at the same time. The env var and the binary version have to move together in both directions.
Client resync is honest on both streams, with one documented edge. Each answers a resume whose cursor belongs to the old keyspace with
sync_required(see Whatsync_requiredmeans to a client), by way of its cold replay-buffer coverage check rather than an epoch comparison — a freshly namespaced bus has no old epoch to compare against. Expect a burst of client reconciliation as they reconnect — for ACTIVITY-stream clients (the web UI) an incremental/changesdelta each, not a full page load. WATCH-stream clients cost less:pad watch --streamanswerssync_requiredby clearing its cursor and keeping the connection open, so it refetches nothing. Either way that is the cutover being paid for, and it is bounded by the number of reconnecting clients — each RESUME is counted, so a client that reconnects several times counts several times.The edge: a cursor that lands exactly one below the first ID a replica sees in the new keyspace is served rather than refused, because nothing in an integer cursor distinguishes the two keyspaces. It is narrow — that one value, on a client that reconnects before the replica has seen anything else — and closing it needs the ID space's identity to reach the client. The SSE spec would allow that (an event ID is arbitrary UTF-8); what excludes it is Pad's own
id:contract, an int64 that every deployed client already parses. Tracked as BUG-2736. A maintenance window narrows it — clients reconnect against an already-cut-over instance rather than racing the cutover — but does not remove it, since their storedLast-Event-IDvalues still belong to the old keyspace and the wire format still cannot say which one they came from.Before BUG-2731 the activity stream (
/api/v1/events) was the silent one: a client reconnecting with aLast-Event-IDfrom the old keyspace against a fresh replay buffer was treated as caught up and silently missed everything that happened during the cutover, until its next full page load. If you are running a build older than that fix, the old behaviour still applies and a namespace change wants a maintenance window rather than a live cutover.Session-presence entries are transient — 90s TTL — and cost nothing either way.
Pad's Redis integration assumes a single Redis node — redis://…, not a
cluster. Key names carry no hash tags and Pad dials a non-cluster client, so a
user's presence index and their session entries would hash to different slots
and the Lua scripts would fail CROSSSLOT. Pointing Pad at a Redis Cluster is
not supported.
What sync_required means to a client
Both SSE endpoints — /api/v1/events (activity, workspace-scoped) and
/api/v1/events/stream (watch, user-scoped) — emit a sync_required event
when the server cannot honestly claim the client has seen everything. The
client's answer is to reconcile: the web client runs an incremental /changes
delta (not a full page load), and the pad CLI clears its cursor so its next
reconnect starts fresh.
It is emitted in two situations, not one. The distinction matters for reading the metrics below, and for anyone writing a third-party consumer:
-
On a resume. The client reconnected with a
Last-Event-IDthis instance cannot vouch for — an evicted or cold replay buffer, coverage that starts above the cursor, an ID-space change, or a cursor it cannot parse. -
Mid-stream, on a connection that is still open. The instance discovered it under-delivered to a client that never disconnected. Two causes: that one connection was too slow to drain its buffer, so an event was dropped for it; or this instance itself missed messages from Redis, which every subscriber on it shares.
Both streams now detect a pub/sub resubscription and a message they could not decode (BUG-2739), and end the affected coverage when they do. Before this the watch stream detected neither, and for a client HOLDING A STREAM OPEN its only signal was a hole in the received ID sequence — which needs a LATER notification to expose it, so a flap that lost the newest notification on a stream that then went quiet left a connected CLI silently stale indefinitely. Detecting the two conditions directly is what covers the case ID arithmetic never reaches.
A RECONNECTING client was never in that position and still is not: a resume asks the shared counter what the newest ID is rather than trusting the instance's local view, so a cursor the instance cannot vouch for is refused whether or not the instance ever noticed the flap. The gap this closes is specifically the open-stream one.
ID-sequence detection itself is watch-stream only, and that asymmetry is by construction rather than an omission. The watch stream has one channel and one counter, so its IDs are consecutive and a hole is visible as a jump. The activity stream's IDs come from a counter shared across workspaces, so per-workspace holes are the NORMAL state and no arithmetic on them means anything. That is why
pad_watchevents_sequence_gaps_totalhas nopad_event_*counterpart.What a failover now COSTS, since detecting it is not free. A watch-bus resubscription ends coverage for that instance's whole watch stream — there is one replay buffer, not one per workspace — so every
/api/v1/events/streamsubscriber on that instance is told mid-stream at once. Activity-stream coverage is per-workspace, so a resubscription there ends only the affected workspace's.What that costs depends entirely on the client, and for the one client that uses the watch stream today it is nearly nothing:
pad watch --streamreacts tosync_requiredby clearing its cursor and KEEPING THE CONNECTION OPEN (cmd/pad/cmd_watch.go), so a failover produces no reconnect and no refetch — the next notification simply starts a fresh coverage span. The cost to watch out for is a future consumer that answerssync_requiredwith a refetch instead: for that client the announcement is one request per connection, arriving together, since per-connection coalescing smooths repeats WITHIN a wave and not the wave itself. That is the deliberate trade this family makes — chatty-but-correct beats quiet-but-lossy — and if it ever becomes a capacity problem the answer is fewer connections per instance, not a quieter bus.What
undecodable_messageactually indicates. Genuinely unreadable input on the watch channel: a non-Pad publisher on the key, a wire format from a mixed-version fleet mid-upgrade, or corruption. It does NOT usually mean two current Pad installations sharing a Redis — those publish the same wire format, so their messages DECODE, and the damage is cross-feeding real notifications between installations while this counter stays flat. That is the failurePAD_REDIS_NAMESPACEexists to prevent, and it is both worse and quieter than the one this counter reports.Who can force a resync with it, and what a flood costs. Anyone who can
PUBLISHonto the watch channel — which sounds worse than it is, since the same access allows publishing FORGED notifications, so a channel writer is outside the threat model already. Under a flood, what IS bounded: the announcement, a non-blocking send onto a capacity-1 flag that is already raised, so it collapses to nothing after the first; and heap GROWTH, since each discarded replay buffer is garbage immediately and the receive loop is serial. What is NOT bounded: per-message CPU and allocation — a fresh replay buffer plus a pass over every subscriber, per malformed message, on the single goroutine that also delivers real notifications, so a sustained flood is receive-loop starvation as much as it is garbage collection. And log volume, one ERROR line per message. Bounding either needs a rate threshold, which is a deployment decision this code declines to make on your behalf. Payload size is deliberately not capped in Pad, because go-redis has read the whole message into memory before Pad sees it; bound it with Redis'sproto-max-bulk-lenand with who holdsPUBLISH.One gap in that detection remains, and it remains on both streams. A message lost in transit with the connection intact — no flap, no decode failure, just a message that never arrived (BUG-2735): on the watch stream a LATER notification exposes it as an ID gap, while on the activity stream, whose per-workspace IDs are non-consecutive by construction, nothing local ever does. (Until BUG-2769 there was a SECOND gap, on the watch stream only — the half-open connection immediately below. It is closed on both streams now.)
A HALF-OPEN connection — a route that stopped carrying traffic without closing, so nothing ever resubscribes and no message ever arrives to be non-consecutive with — is closed on both streams: on the activity stream by BUG-2738 and on the watch stream by BUG-2769, each behind its own phase-2 flag, so it is closed on a given deployment only once that flag is on. Do not assume go-redis's pub/sub health check covers it on either:
PubSub.Pingwrites the command and never reads a reply, so it reports healthy for as long as the socket accepts writes. What closes it is application-level idle tracking with a heartbeat that makes the threshold answerable — see Half-open connection detection, which covers both buses.A further residual affects RESUMES rather than open streams (BUG-2743): if the watch counter restarts without the epoch rotating — evicted under
maxmemory, lost to aFLUSHDB, restored from a stale snapshot — the old and new ID spaces overlap, and aLast-Event-IDinside that overlap cannot be attributed to either. The instance refuses the cursors it can identify as stale and serves the rest, so a client holding an old-space cursor in the overlap can be handed new-space notifications as though they followed it. Arithmetic on the IDs cannot close this — telling two sequences apart is what the epoch token is for, and this is precisely the case the epoch does not see. Rotating the epoch (see Event ID-space migration) is what makes a deliberate counter reset safe.A RECONNECTING client is largely covered on the watch stream anyway, because a resume consults the shared counter rather than local state alone. Not entirely: that check reads the counter at one instant, so a notification published AFTER the read and missed is invisible to it — an at-most-once pub/sub residual with no per-connection ack, documented on
resumeOutrunsLocalViewand again in the CLI. What these gaps reliably leave stale is the client holding a stream OPEN.
The second case is newer — before it, a held-open stream that missed events was
never told, and a later delivered event advanced its cursor past the missing
IDs so no replica would ever replay them. A mid-stream sync_required carries
an empty id: field, exactly as the resume case does, so a client stops
resending a position the server has just disclaimed.
There is no separate event name for the mid-stream case, deliberately: every client acts on the two identically.
What a client should DO with it, since the two endpoints recover differently and a third-party consumer cannot infer this from the frame:
- Keep the connection open. The frame is not a close and does not ask for a
reconnect. The server keeps streaming; a client that tears down and redials
on every
sync_requiredturns one delta into a reconnect storm. - Expect events after it, possibly with IDs below the hole. A mid-stream
sync_requiredis not ordered against events the server had already queued for that connection, so a client can receive the frame and then events that predate the gap. Their IDs re-establish a cursor at a position the server has just disclaimed. This is deliberate and bounded: reconciling is what the frame asked for, and a later reconnect from such a cursor is refused by the coverage check and answered withsync_requiredagain. Holding the announcement back until those events drained was tried and removed — every version of it could defer the announcement indefinitely while a busy workspace kept the queue full, and an unbounded silence is worse than a redundant resync. - Stop trusting your cursor. The empty
id:retires it, so a compliant SSE client stops sendingLast-Event-IDon its next reconnect. Do not re-send the old value: the server has just said it cannot vouch for that position. - On
/api/v1/events, reconcile the workspace. Its events describe item state, so a delta refetch recovers everything missed. The web client uses/changes; any client can re-read the items it cares about. - On
/api/v1/events/stream, reconcile what you can and accept the rest is gone. Watch-matched notifications describe item state and can be re-derived by re-reading those items. One-shot PUSHES cannot: they are not stored as recoverable state, there is no backfill endpoint for them, and a push missed during a hole is missed permanently. This endpoint is best-effort for pushes by design, andsync_requiredon it means "your position is untrustworthy", not "re-fetch and you will be whole again". ThepadCLI monitor does exactly this: it clears its cursor and keeps listening. There IS a separate metric — seepad_event_midstream_resyncs_totalbelow — so the two populations stay distinguishable to an operator without changing what any existing alert means.
A connection gets at most one MID-STREAM announcement every 5 seconds (the resume-time signal is not rate-limited and never needed to be — it happens once per connection, at the start), and nothing is lost to that bound: a gap arriving inside the window is remembered and announced when the window closes. The bound exists because the subscriber most likely to be signalled is a slow one, and answering "you could not keep up" with "now fetch a delta" can feed back into more drops.
Redis health and metrics
/api/v1/health/ready reports Redis in its payload but does not gate readiness on
it — the REST API, the web UI and every item-writing path work with Redis down, so
failing readiness over a Redis blip would pull healthy replicas out of the load
balancer and turn a degraded feature into an outage. When Redis is unreachable
the payload carries redis.reachable: false, the probe error, and a degrades
list naming what is lost. Note what that list says about activity events: they
stop for all clients, not only across instances — the activity bus does not
fall back to a local fan-out when its publish fails. The block is absent entirely when no Redis is
configured.
Alert on these instead:
| Metric | Meaning |
|---|---|
pad_redis_up |
0 when the last probe (every 15s) failed. Exported only when Redis is configured — absence means "no Redis", not "down" |
pad_stream_connections_active |
Held streaming connections on this instance, across both SSE endpoints — the population the limits bound |
pad_watchevents_sequence_gaps_total |
This instance missed notifications — a delivery fault |
pad_watchevents_resume_gaps_total |
Resumes this instance could not serve — from a hole, a cold start, an epoch change, or a shared-counter disagreement. Each sends a client sync_required. RESUME-TIME ONLY; a subscriber told mid-stream is counted separately, so an alert on this keeps the meaning it had |
pad_watchevents_midstream_resyncs_total |
Watch-stream subscribers told MID-STREAM that they missed notifications, on a connection that stayed open. New in BUG-2730 |
pad_watchevents_notifications_missed_total |
How many notifications those gaps spanned |
pad_watchevents_notifications_dropped_total |
Received but not delivered to a local subscriber — that connection's buffer was full. Since BUG-2730 that subscriber is told (sync_required, mid-stream) rather than silently under-served, so a rise here produces a rise in pad_watchevents_midstream_resyncs_total, one client at a time |
pad_watchevents_sequence_resets_total |
Watch replay coverage dropped, by reason. epoch_change — the watch epoch token changed, so the IDs now come from a different sequence; the token is an opaque UUID here, not a numeric generation. counter_backward — an ID arrived at or below the high-water mark with the epoch unchanged. (This label was spelled counter_backwards while BUG-2739 was in development. If you are reading a dashboard that uses the plural, it was built against an unreleased build — see the note below.) subscription_resumed — a pub/sub connection dropped and re-subscribed, so whatever was published during the outage never arrived; expect these during a Redis failover and expect them to stop afterwards. undecodable_message — a message on the watch channel could not be parsed. The instance cannot tell whether that was a notification it should have had or something foreign, and it stops vouching because it cannot tell; expect zero, and suspect a namespace collision. idle_timeout — the subscription received nothing at all (no notification, no heartbeat, no subscription confirmation) for longer than the idle timeout, so this instance stopped vouching for its buffer and attempted to replace the connection — attempted, because the resubscribe can fail, in which case a later pass tries again and this counter has already moved; it means the socket stopped PROVING it works, not that notifications were observed going missing, and it is structurally never emitted on watch-heartbeat phase 1. Unlike the activity stream's twin it needs no companion counter, because dropCoverage here replaces the buffer and reports unconditionally rather than only when one existed. The first two mean the ID space changed under this instance. subscription_resumed means it did not and something demonstrably went missing. undecodable_message means neither is established — only that coverage can no longer be proved. Each also announces to the watch subscribers connected at that moment, so each moves pad_watchevents_midstream_resyncs_total by AT MOST one per such subscriber — at most, because the signal is capacity-1 and coalescing, so a second cause firing before a client has acted on the first adds no announcement. For the same reason the announcement counter is not a ratio against this one in aggregate: it also counts gaps and slow-subscriber drops, and only a reset observed in isolation, against idle clients, lets you read the fan-out off these two counters |
pad_watchevents_receive_loop_exits_total |
Non-zero outside shutdown means an instance publishes but receives nothing |
pad_event_resume_gaps_total |
The ACTIVITY stream's (/api/v1/events) twin of the watch resume counter above. Expect a step around a deploy, with the RATE settling back to baseline (the counter itself only ever increases) — each instance starts with no replay coverage, so an early resume against a workspace it has not seen yet is a warranted resync. It counts RESUMES, not clients: a deploy with no reconnects does not move it at all, and a client that reconnects several times is counted several times. A rate that does not settle is the thing to alert on |
pad_event_midstream_resyncs_total |
Activity-stream subscribers told MID-STREAM that they missed events, on a connection that stayed open. New in BUG-2730, and the counter to watch when judging whether that fix is costing more resyncs than it is worth. It counts ANNOUNCEMENTS, not causes and not distinct clients: a reset that drops buffers moves it once per live subscriber (and that ratio against pad_event_sequence_resets_total is the fan-out); a burst of drops on ONE connection moves it once, because signals coalesce and are rate-limited per connection; and a coverage loss on a workspace with no buffer yet moves it while every cause counter stays flat, because there was no coverage to end but the subscribers still have a hole |
pad_watchevents_midstream_resyncs_total (see also, listed above) |
Same meaning for the watch stream. Its causes are a slow-subscriber drop and a received sequence gap or reset; a gap announces to EVERY subscriber on the instance, so it can exceed all of its cause counters |
pad_event_sequence_resets_total |
Activity replay coverage dropped, by reason. subscription_resumed — a pub/sub connection dropped and resubscribed, dropping that workspace's buffer; expect it during a Redis failover and expect it to stop afterwards. epoch_change — the shared counter's ID space changed generation, dropping every buffer; expect a handful per cutover. counter_backward — an ID arrived at or below a buffer's high-water mark with no generation change; see Event ID-space migration for what to expect per phase. epoch_regressed — a LOWER generation was seen, so this instance stopped vouching for its buffers. One alongside an epoch_change is a message that was in flight when the generation rotated; a RUN of them means the counter itself went backwards — usually Redis lost writes, and since BUG-2740 possibly a repaired generation key (see A repaired generation counter). undecodable_message — a message on these channels could not be parsed, so that workspace's coverage ended; expect zero, and suspect a namespace collision. subscription_unconfirmed — a subscription was admitted before Redis acknowledged the SUBSCRIBE and the acknowledgement then arrived, so the span in between is one that stream cannot account for; it reaches THIS counter only when a buffer existed to drop, so read pad_event_subscription_unconfirmed_total for the dependable count. idle_timeout — a subscription received nothing at all (no event, no heartbeat, no acknowledgement) for longer than the idle timeout, so this instance stopped vouching for its buffer. It means coverage ended, not that the connection was replaced: the replacement is attempted afterwards and installs nothing if the instance is shutting down or the workspace loses its last subscriber, so only pad_event_subscription_cycled_total proves a replacement. Unlike subscription_resumed it does NOT establish that events went missing, only that the socket stopped proving it works, and like subscription_unconfirmed it reaches this counter only when a buffer existed to drop |
pad_event_events_dropped_total |
Activity events not delivered to a live subscriber, by reason — today only slow_subscriber (that connection's 64-deep channel was full). Per-SUBSCRIBER: every subscriber that was keeping up received the event. Pairs with pad_event_midstream_resyncs_total, though not one-for-one in either direction — see that row. New in BUG-2730, along with the fix that stops the drop being silent, so a deploy that starts reporting these is not necessarily a regression — it may be the first time they were countable |
pad_event_subscription_cycled_total |
Activity-stream workspace subscriptions torn down and replaced because nothing arrived on them — no event, no heartbeat, no acknowledgement — within the idle timeout. It counts replacements, not teardowns: a cycle that installed nothing because the instance was shutting down or the workspace lost its last subscriber does not increment it, so a restart cannot manufacture this signal. Detects a half-open connection: no FIN, no RST, just a route that stopped working, which go-redis cannot see because its pub/sub health check writes a PING and never reads the reply. Expect zero. Read this rather than pad_event_sequence_resets_total{reason="idle_timeout"}, which moves only when a buffer existed to drop and so under-reports exactly the early-wedge case this detector exists for. A non-zero rate means connections to Redis are being silently blackholed — a NAT idle timeout, a stateful firewall, an overlay network dropping long-lived flows; check TCP keepalive on the path before changing the interval. On heartbeat phase 1 this counter is structurally zero — detection is part of phase 2, so a zero there says nothing at all about whether any route has wedged. Read heartbeat_phase off the startup log before drawing any conclusion from it, and take it from the "Event bus using Redis pub/sub" line: since BUG-2769 the watch bus logs a heartbeat_phase of its own, on its own line, under a separate flag, and it has no bearing on this counter |
pad_event_subscription_unconfirmed_total |
Activity-stream subscriptions admitted before Redis acknowledged the SUBSCRIBE, because the wait for it timed out (BUG-2747). Expect zero. Counts ESTABLISHMENTS, not clients — one workspace subscription that timed out increments it once however many subscribers were waiting on it. Nothing is known to have been lost; what it says is that a stream was admitted whose coverage this instance cannot describe, and that every subscriber waiting on it will be told to reconcile when the acknowledgement lands. A non-zero rate means the SUBSCRIBE round trip is slow or stalling — read it alongside SSE connect latency rather than alongside pad_event_sequence_resets_total |
pad_event_receive_loop_exits_total |
A workspace's activity subscription loop stopped. Unlike the watch stream's twin this does not stay at zero — it is expected at shutdown and whenever a workspace's last local subscriber leaves. Read it as a rate against a stable subscriber count |
pad_session_presence_failures_total |
Presence operations failing — read the op label, the risks differ and run in opposite directions: register/renew may under-report (a live session unlisted and untargetable), deregister may over-report (a dead session left listed, and a push aimed at it reaches nobody), list returns a 503, prune is benign. A failure means the operation reported an error — Redis can fail a pipeline after applying it, so the write may have landed anyway |
A repaired generation counter
event_epoch_gen is a shared Redis key, and the same things that corrupt any
shared key can corrupt it: a namespace collision with another installation, a
hand-edit during an incident, a restore that mixed keyspaces. Since BUG-2740 a
corrupted one is REPAIRED rather than fatal — before that, every phase-2
publish consumed a sequence ID and then failed, permanently, because the branch
that would have rotated the generation was the branch that could not run.
Two operator-visible consequences, neither of which had documentation:
- A repair reseeds the generation from wall-clock SECONDS. That is above
any counted history, so it normally reads as an ordinary
epoch_change. It is not guaranteed to be above a counter that a collision or a hand-edit had pushed higher, so it can instead surface asepoch_regressed— which otherwise means a failover to a replica that lost writes. The tell is the value: read the key, and a repaired generation looks like a unix timestamp (ten digits, around 1.7e9) rather than a small count of ID-space resets. There is no repair-specific counter or log line, because the repair happens inside a Lua script. - Clients reconcile, and normally once. A repair is an ID-space change
like any other, so receivers stop vouching for their buffers. It does not
loop, because the repaired key is valid and the next rotation increments it
normally. The exception is a repaired generation that lands BELOW the one a
receiver already holds: that instance discards the lower epoch as a
straggler for its 30-second window rather than adopting it, so the same
space can be disclaimed again when it is finally adopted. Bounded by that
window, and visible as
epoch_regressedrather thanepoch_change.
Two repairs CAN collide, and what catches it is not the epoch. The seed is
above any COUNTED history; it is not a monotonicity guarantee. Corrupt the key
twice inside one second and both repairs seed the same value, so two genuinely
different ID spaces carry the identical epoch — and an equal epoch means "same
space" by design, so neither epoch_change nor epoch_regressed fires.
The detection chain that does hold, stated so nobody has to rediscover it:
A merge requires IDs to be REUSED at a receiver. Reuse requires the sequence counter to go BACKWARDS. A backwards counter is detected regardless of what the epoch says — it is the
counter_backwardreason, which drops the affected buffers and refuses cursors below the discarded high-water mark.
So the guarantee is carried by a different detector than the epoch mechanism
suggests. That is deliberate and it is tested end to end
(TestACollidingRepairIsCaughtBySequenceRatherThanEpoch), because a future
change that weakened counter_backward would remove a protection nothing else
here advertises.
Two cases that look similar and are not. A sequence counter set FORWARD — say to 50, so the next ID is 51 — is a jump inside ONE space: IDs stay unique and increasing, nothing is reused, and per-workspace IDs are non-consecutive by construction anyway. And a receiver that never held the colliding range has nothing to merge; what it experiences is a gap, which is the pre-existing undetectable-loss case tracked as BUG-2735.
pad_watchevents_sequence_resets_total has no released contract yet, and
that is why BUG-2739 could change it freely. The whole metric was introduced
after v0.14.0 and no tagged release emits it, so nothing outside a
development deployment can be alerting on it. Two things about it changed on
that branch: the counter_backward label lost a trailing s, and the metric
widened from "the ID space changed" to "replay coverage was dropped", which
added the subscription_resumed and undecodable_message reasons. A
reason-specific alert on epoch_change is unaffected; one on
counter_backward must have its expression updated for the spelling, which is
the whole reason this paragraph exists. An alert on the unlabelled total now
counts more things, which is the metric doing what its name says rather than a
regression. During a rolling deploy an instance on
the older build reports neither new reason and keeps the old spelling — so a
mixed fleet reports two shapes under one name for the rollout's length, which
is acceptable precisely because no released version is in that fleet.
Re-derive that rather than trusting this paragraph, because it is a claim about release state and release state changes without anyone editing this file:
git describe --tags --abbrev=0 origin/main # the latest tag
git log --reverse --format=%H -S pad_watchevents_sequence_resets_total \
-- internal/metrics/metrics.go | head -1 # the commit that introduced it
git merge-base --is-ancestor <commit> <tag> # non-zero exit => still unreleased
Once a release does ship this metric, the next change to it is a real contract break and needs versioned treatment instead of a note here.
Avoid an evicting maxmemory-policy for Pad's Redis.
docker-compose.prod.yml sets noeviction for this reason; the plain
docker-compose.yml keeps allkeys-lru on its 64 MB dev instance, where the
consequence below is a momentary annoyance rather than a lost instruction —
change it too if you run that file in anger.
Under an evicting policy Redis may drop live session-presence entries under
memory pressure. Nothing can distinguish that from a TTL lapsing, so a
connected agent session briefly disappears from the picker and a push targeted
at it reports
delivered_sessions: 0. It self-repairs on the session's next 30-second
renewal, and Pad's keyspace is small — a few hundred bytes per connected
session plus two counters — so there is nothing to gain by evicting it.
If push stops finding a session (on-call)
The most likely Redis-related symptom is a transient write failure while registering a session. The agent's event stream stays up — the connection is never refused over a registry problem — but the session is absent from the shared registry, so:
- it does not appear in
GET /api/v1/sessionsor the web picker, and - a push targeted at it returns
200 pushed:truewithdelivered_sessions: 0and skips publication, so the instruction is not delivered.
What you'll see: session presence: failed to register session or failed to renew session entry warnings (rate-limited to one per minute, carrying
failures_since_last_log — a large count means the replica, a small one means
a single session), and the session missing from the listing.
What to do: restore Redis connectivity, capacity, or ACLs. Registration self-heals — each session's renewal re-writes its full entry, so an affected session reappears within ~30 seconds without reconnecting. Confirm it is listed again before re-sending anything.
What NOT to do: do not blindly re-send. A targeted push reporting
delivered_sessions: 0 is safe to resend, because the server skipped the
publish. A broadcast is always published, and a 502 push_unconfirmed means
the outcome is unknown — re-sending either can deliver a second instruction the
agent acts on twice. Only re-send what the server told you it skipped.
Upgrading a multi-instance deployment
PAD_REDIS_URL now also backs the session-presence registry — the list of
which agent sessions are connected, which pad push and the web UI's "Push to
agent" picker read to decide where a push goes. Previously that registry was
per-process even when Redis was configured, so a push aimed at a session held
by another replica was silently dropped.
During a rolling upgrade, old and new replicas disagree about presence. An
old replica has only its own connections in view, so a push it answers cannot
see a session held on a new replica, and GET /api/v1/sessions returns a
different list depending on which replica answers. A TARGETED push reports this
honestly — delivered_sessions: 0, and the publish is skipped, so nothing was
sent — but the instruction is not delivered.
This is the same behaviour every replica had before this build, so the rollout is not a regression; it is a window in which the fix is only partly in effect. Two ways to avoid the window:
- Blue/green — bring up the new replicas, cut traffic over, retire the old ones. No mixed period.
- Drain first — scale old replicas out of the load balancer and let agent
monitors reconnect (
pad watch --streamreconnects on its own) before serving pushes from the new set.
If neither is practical, a rolling upgrade is still safe: nothing is corrupted
and no migration is needed. Targeted pushes may report delivered_sessions: 0
and go undelivered until every replica runs the new build; those are safe to
re-send once the rollout completes, because a targeted miss skips the publish
entirely.
That safety does not extend to broadcasts. A broadcast push is always
published, on old and new replicas alike, and the shared notification bus
carries it across instances regardless of which registry the answering replica
used — so a broadcast reporting 0 during the rollout may well have been
delivered. Re-sending one is a second instruction the receiving agent will act
on twice. Only re-send a push the server told you it skipped. There is no Redis or database migration; the
registry's keys are transient and expire on their own TTL.
Event ID-space migration (PAD_EVENTS_PUBLISH_EPOCH)
Events on the workspace activity stream (GET /api/v1/events) carry a
Last-Event-ID so a reconnecting client can be replayed what it missed. With
Redis, every instance shares one counter, so those IDs are meaningful across
replicas.
The problem this migration fixes. If that shared counter is ever reset —
the key evicted under maxmemory, deleted by hand, a fresh Redis after a
restore — IDs start again from 1. A replica that was buffering the old
sequence cannot tell the new 101 from the old 101, so it can merge two ID
spaces into one replay buffer and answer a resume across the boundary as
though nothing was missed. Numeric detection alone cannot see it: by the time
the new sequence passes the replica's high-water mark, it looks like ordinary
progress.
What the fix does and does not close, stated before the procedure. It stops a REPLICA from mixing two ID spaces in one replay buffer, which is what turns a counter reset into a silently wrong replay. It does NOT make a CLIENT'S CURSOR say which space it came from — that would change the wire format every deployed browser speaks. So this is a substantial mitigation and not a closure; the residual case and why it is deferred are at the end of this section.
The fix gives each ID space an epoch — a generation number (monotonic in
normal operation; see A repaired generation counter below for the one case
that is not),
minted by Redis when the space is created and carried as a
<epoch>|<id>|<json> prefix on every message published by a phase-2
instance. Phase-1 instances publish the historical bare JSON and carry no
epoch at all, which is what the two phases are about. A replica that sees a HIGHER
generation drops its replay buffers and answers resumes across the change with
sync_required, which is honest rather than silent. A message carrying a
LOWER generation is a straggler from a space that has been abandoned, and is
discarded rather than delivered.
The generation is a number rather than an opaque token so the two spaces can be ORDERED. Workspaces have independent subscriptions and Redis does not order messages across channels, so a pre-rotation message on one channel can arrive after a post-rotation message on another; with an unordered token that is indistinguishable from a second rotation.
It rolls out in two phases, and the order is not optional.
| Phase | What you do | What instances publish | What they accept |
|---|---|---|---|
| 1 | Roll the new binary everywhere. Leave PAD_EVENTS_PUBLISH_EPOCH unset. |
The historical bare JSON | Both forms |
| 2 | Set PAD_EVENTS_PUBLISH_EPOCH=true and roll again. |
<epoch>|<id>|<json> |
Both forms |
The asymmetry that makes two phases necessary: an instance running a pre-phase-1 binary cannot parse a prefixed payload at all. It fails to unmarshal the message and drops the event for its own clients. So flipping before every instance is upgraded loses events on the ones that are not — not a resync, a silent loss.
Both rolls are zero-loss in the other direction, because accept-both is on from phase 1: during the phase-2 roll, flipped and un-flipped instances are publishing different forms at the same time and every instance reads both.
Rolling back to phase 1 is safe: make the effective value false and roll. Peers accept the bare form throughout, so there is no window where this direction loses events.
Two things about rolling back that are easy to get wrong:
- Setting the value to false is not the same as unsetting the environment
variable.
events_publish_epochcan also be set in~/.pad/config.toml, and the config file's value stands when the environment variable is absent. Clear both, or set the environment variable explicitly tofalse. - Downgrading past phase 1 is a SECOND step, and the order is the reverse of the upgrade. A pre-phase-1 binary cannot parse the prefixed form. So: first roll every instance to phase 1 (new binary, flip off) and let the roll finish, then downgrade the binary. Introducing an old binary while any flipped instance is still publishing drops events on the old one — the same asymmetry that makes the upgrade two phases, in reverse.
There is no Redis or database migration in either direction. The epoch key and its generation counter are created by the first flipped publisher; a phase-1 instance deletes it if it ever sees the sequence counter restart, so a counter that is reset while the deployment sits on phase 1 does not leave a stale epoch for a later phase 2 to adopt.
What you should see when phase 2 lands. A replica learns the epoch from
the first prefixed message it RECEIVES — which means only replicas currently
subscribed to a workspace see it, and only when that workspace next has
traffic. If such a replica had already buffered un-prefixed events, it drops
its buffers once, records
pad_event_sequence_resets_total{reason="epoch_change"}, and clients resuming
across that moment get sync_required and re-fetch. A replica whose buffers
are EMPTY adopts the epoch without dropping anything and without a reset
count, deliberately: otherwise every replica would report a reset at startup
and the counter would grow a per-deploy baseline instead of meaning something.
Do not delete the generation counter (<namespace>event_epoch_gen) by hand,
and keep it out of any eviction policy: it is what makes one ID space orderable
against the next. Losing it lets a later reset reuse a generation that has
already been seen, which makes two different ID spaces look identical — the
one shape the epoch exists to prevent. It is a single small integer key; the
events keyspace should not be under allkeys-lru (see Redis configuration
notes). One drop per replica
per roll — if the counter keeps climbing, something is deleting the epoch or
sequence key repeatedly; check maxmemory-policy against the events keyspace
(see Redis configuration notes).
pad_event_sequence_resets_total{reason="counter_backward"} is the other
counter to watch. It fires when an ID arrives at or below what a buffer had
already seen.
On phase 1 it can be non-zero at any time, not only during a roll. Phase 1
keeps the historical two-call publish — INCR, then PUBLISH — so two
instances can interleave (INCR 5, INCR 6, PUBLISH 6, PUBLISH 5) and a receiver
sees 5 arrive after 6. That window is older than this migration; phase 2 is
what closes it, by moving ID assignment into a single atomic script so publish
order equals ID order globally.
So the expectation depends on where you are:
- Phase 1, before this replica has ever seen a prefixed message — expect ZERO. The check is deliberately not armed until an epoch has been adopted, because a phase-1 deployment's two-call publish interleaves as ordinary traffic and reacting to that would drop every replay buffer on a busy multi-instance deployment. The cost of that gate is that a counter reset on a never-flipped deployment goes undetected — which is exactly the behaviour before this migration existed, and precisely what phase 2 fixes.
- During the phase-2 roll, once a replica has adopted the epoch — expect it to rise for the length of the roll: un-flipped publishers are still assigning and publishing in two calls, and this replica is now armed.
- Phase 2, every publisher flipped — expect it at or near zero. A persistent rate here is an anomaly worth investigating rather than tuning away.
Which phase an instance is publishing in is in its startup log, as
id_space_phase=1 or id_space_phase=2 on the "Event bus using Redis pub/sub"
line — the counter above cannot be read without it. An unparseable
PAD_EVENTS_PUBLISH_EPOCH is ignored (a typo must not flip a migration whose
wrong direction loses events) and logs a warning naming the value.
One narrow window during the phase-2 roll. Once a replica has adopted the
epoch, a message from an un-flipped instance carries no epoch and is treated as
belonging to the current space — which it does, unless the sequence counter
reset between that publisher assigning its ID and publishing it. An ID from the
dead space can then land in a buffer describing the new one. There is no way to
tell the two apart from the message alone, and the alternatives are worse: a
replica that refused un-flipped messages would resync its clients on every one
of them for the length of the roll. It usually ends loudly and quickly: the next
event that workspace receives is lower than the straggler's ID, which trips
counter_backward, drops the buffers and is reported. It is not guaranteed to
— the sequence counter is shared across workspaces while that check is per
workspace, so if other workspaces carry the counter past the straggler's value
first, nothing fires and the dead-space ID stays in that workspace's buffer.
Closing that needs the same thing the residual below needs.
What this migration does not fix. A client's Last-Event-ID is still a
bare integer with no epoch in it, and that is deliberate — every deployed
browser speaks that format, and EventSource echoes the header with no
application code in the path to translate it. So an old ID and a new ID of the
same numeric value remain indistinguishable to a resume, even though the
replica's buffers can no longer mix them. The exposure is a client that
reconnects with a cursor whose number the new sequence has already reached.
Tracked on BUG-2736.
Single-process deployments (no PAD_REDIS_URL) need none of this and ignore
the variable: that bus owns its counter, so it identifies its own ID space
from its start time. Two runs' IDs can only collide if the earlier process
published more than 2^20 events per millisecond of its own lifetime, or if a
restart completed inside a single millisecond — both deterministic bounds
rather than probabilities, and neither reachable by a process that has to bind
a listener and open a database before it can publish anything. A clock stepped backwards across a restart degrades the other
way, into extra sync_required responses rather than wrong replays.
Half-open connection detection (PAD_EVENTS_HEARTBEAT, PAD_WATCH_HEARTBEAT)
Two buses, two flags, rolled independently. The activity stream
(/api/v1/events) and the watch stream (/api/v1/events/stream) hold different
Redis subscriptions with different fates, so each has its own phase-2 flag and
you can roll one before the other. Everything below applies to both; the
differences are collected at the end.
The problem this fixes. A TCP connection can stop carrying traffic without closing — no FIN, no RST, just a route that stopped working. A NAT table expiring, a stateful firewall dropping an idle flow, an overlay network silently rerouting. The instance behind it blocks on a read that will never return, receives nothing, and its replay buffer goes on looking complete. Every resume for that workspace is then answered "caught up" from a coverage window that ended when the route did — silent loss, with nothing in any metric.
Why go-redis does not cover it. Its pub/sub health check writes a PING
and never reads a reply, so its error stays nil for as long as the socket
accepts writes — which a half-open socket does until its send buffer fills. The
channel path sets no read deadline either. Measured, not assumed: against a TCP
proxy that silently stopped forwarding, with the health check running, there
was no reconnect in 24 seconds.
What the fix does. Every subscription records when it last received
anything — an event or notification, a subscription acknowledgement, or a
heartbeat. When that goes stale past the idle timeout, the instance ends that
subscription's replay coverage (a workspace's on the activity stream, the
instance's on the watch stream — so the next resume answers sync_required
rather than "caught up")
and replaces the connection. Dropping coverage alone would not recover: the
resync it demands is served from the same dead socket, and the detector fires
again on the next pass — a loop metering the failure rather than fixing it.
Why a heartbeat, rather than just a threshold on real traffic. "Is this stream quiet, or is the route dead?" cannot be answered from traffic — it depends on your publish rate, and no constant is right for every deployment. Publishing our own frame replaces it with "did our heartbeat arrive?", which is answerable everywhere. The instance publishes one frame every 30 seconds (T) — per subscribed workspace on the activity stream, once per instance on the watch stream, and cycles a subscription that has received nothing for 90 seconds (3T). Three intervals rather than two so a single lost or late frame is not a cycle. Detection latency measured from the last frame that got through is 90–120s — the scan runs on its own 30s cadence, which adds up to one interval on top of the threshold. Measured from the moment the route actually died it is wider, roughly 60–120s: the publisher runs on an independent schedule, so the last frame through may have been sent anywhere in the interval before the fault.
Detection is part of phase 2, not phase 1. Publishing and detecting are one capability with one switch, because an instance detects off its own frames — it publishes to the workspace channels it subscribes to and receives them back, so it never depends on peers having flipped. A phase-1 instance therefore detects nothing; it only recognises the frame so that a phase-2 peer costs it nothing. Splitting them was tried and is wrong: with no heartbeat and no events, a perfectly healthy quiet workspace crosses the threshold every 90–120s and gets cycled, which is a resync storm on the default configuration every deployment lands in first.
It rolls out in two phases, and the order is not optional.
| Phase | What you do | What instances publish | What they do with a frame |
|---|---|---|---|
| 1 | Roll the new binary everywhere. Leave both flags unset. | No heartbeats | Recognise and ignore it. No idle detection. |
| 2 | Set the flag (PAD_EVENTS_HEARTBEAT and/or PAD_WATCH_HEARTBEAT) and roll again. |
One frame per 30s — per subscribed workspace on the activity bus, once per instance on the watch bus | Recognise and ignore it. Idle detection active. |
What happens if you run them out of order. The frame has to travel on the same channel the stream's own traffic does — the workspace's event channel on the activity bus, the single watch channel on the watch bus — because that channel's connection is the thing whose liveness is in question; a probe anywhere else proves the wrong thing. An instance running a pre-phase-1 binary cannot classify it: the frame falls through to that bus's decoder, fails to parse, and is treated as a hole in coverage. That instance drops the replay buffer and tells every one of its live subscribers to resync — every 30 seconds, for as long as the deployment is mixed. On the activity bus that is per workspace, so the noise scales with how many an instance is subscribed to; on the watch bus it is one buffer and one announcement round per instance. Either way the blast radius is the instances you have not upgraded, which no amount of care in the new code can reach. This is noisier than the ID-space migration's equivalent mistake and it is the reason both defaults are off.
Both rolls are zero-loss in the other direction: phase-1 instances recognise the frame from the release that introduces it, so during the phase-2 roll a mix of publishing and non-publishing instances is exactly the case ignore-the-frame exists for.
Rolling back to phase 1 is safe and takes effect immediately: make the effective value false and roll. Peers ignore the frame throughout, and idle detection stops with it — you are back to the pre-BUG-2738 behaviour, which is a wedged route going unnoticed, not a worse one. The same two wrinkles as the ID-space migration apply, for the same reasons:
- Setting the value to false is not the same as unsetting the environment
variable.
events_heartbeatandwatch_heartbeatcan each also be set in~/.pad/config.toml, and the file's value stands when the environment variable is absent — so unsettingPAD_WATCH_HEARTBEATon a host whose config file sayswatch_heartbeat = trueleaves that instance on phase 2. Clear both, or set the environment variable explicitly tofalse, which wins over the file. - Downgrading past phase 1 is a SECOND step, in the reverse order. A pre-phase-1 binary still cannot classify the frame. Roll every instance to phase 1 (new binary, flip off), let it finish, then downgrade the binary.
The frame is validated, not just prefix-matched. A liveness frame is
hb|<version> plus optional short tokens, under a length cap. Anything else
that happens to begin with hb| is treated exactly as any other unreadable
payload: coverage ends — that workspace's on the activity bus, the instance's
on the watch bus — and the bus's own reset counter moves with
reason="undecodable_message" (pad_event_sequence_resets_total or
pad_watchevents_sequence_resets_total), which is the signal that says suspect
a namespace collision. A forged frame cannot
fake liveness in any case — liveness means "this socket carried traffic", and a
frame that arrives demonstrates that whoever sent it.
There is no Redis or database migration in either direction, and the frames are
never persisted: a heartbeat consumes no event ID, carries no epoch, is never
buffered or replayed, never reaches a subscriber, and is never counted as an
event. That last part is load-bearing rather than tidy — three of this bus's
reset reasons (counter_backward, epoch_change, epoch_regressed) are
derived from the shared ID counter, so a probe that consumed IDs would
manufacture the resets it exists to avoid.
Which phase an instance publishes in is in its startup log, as
heartbeat_phase=1 or heartbeat_phase=2. Each bus logs its own: the activity
bus on the "Event bus using Redis pub/sub" line, alongside id_space_phase; the
watch bus on a separate "Watch bus using Redis pub/sub" line, which has no
id_space_phase because that migration does not apply to it. All of these are
independent — any combination is valid. An unparseable PAD_EVENTS_HEARTBEAT or
PAD_WATCH_HEARTBEAT is ignored and logs a warning naming the value.
What this covers, and what it does not. It is a receive-side detector, not a round-trip health check. It measures whether frames arrive on a workspace's subscription, so:
- A subscription whose outbound direction is broken but which still receives looks healthy — correctly, since nothing is being lost.
- The PUBLISH path is not covered and cannot be.
PUBLISHtravels on the client's ordinary connection pool while a subscription holds a connection from a separate pub/sub pool; those are different sockets with different fates, and a reconnect of one repairs nothing about the other. An instance whose publish path is wedged loses its own events for every other instance, and this feature will not tell you. - The replacement is attempted, not guaranteed. If the path is still
blackholed when the cycle re-dials, the new connection cannot receive either
and the detector fires again on the next pass. Coverage stays ended
throughout, so nothing is ever falsely claimed — but delivery resuming is a
statement about your network, not about Pad. One case where the replacement
can fail on a healthy path is tracked as BUG-2764: go-redis discards the
error from the initial
SUBSCRIBE, so a failed subscribe yields a connection that looks live and is subscribed to nothing. The detector cycles it again on the next pass, which is why this self-heals on phase 2 and does not on phase 1.
What to watch. On the activity bus,
pad_event_subscription_cycled_total — expect zero. Read it rather than that
bus's idle_timeout reset label, which only moves when there was a buffer to
drop and therefore misses the early-wedge case. The watch bus has no such
metric and needs none: its dropCoverage has no early return, so
pad_watchevents_sequence_resets_total{reason="idle_timeout"} is already the
complete count — that is the series to watch there. A non-zero rate is a
network fact about the path between your instances and Redis, not a Pad
condition: compare it against TCP keepalive settings on that path before
changing the interval, because a shorter interval treats the symptom and a
longer one widens the window the detector exists to bound.
A residual an operator should know about, not fixed here. When many
workspaces are cycled at once — a NAT table flush, a firewall rule change, an
overlay network dropping every long-lived flow — every connected subscriber of
every affected workspace is told to resync in the same instant. The SSE
connections stay open, so this is not a reconnect storm and the admission
limits are not involved; what it produces is a burst of /changes requests
against the database, coalesced per browser tab but with no jitter and no
global budget. This is not new with the heartbeat: a Redis failover already
signals every workspace at once through subscription_resumed. What is new is
a second trigger of the same class. Tracked separately; if you run a large
fleet, watch database load alongside
pad_event_sequence_resets_total after any network event that could wedge many
routes simultaneously. Tracked as BUG-2761.
Cost on the activity stream. Each workspace has its own Redis subscription — and therefore its own connection — so liveness is genuinely per-workspace and there is no cheaper shared probe. An instance subscribed to N workspaces publishes N frames every 30s; at N=1000 that is roughly 33 publishes/sec, which is noise for Redis. If fleet workspace counts ever make it matter, the fix is connection consolidation, not a longer interval.
How the watch stream differs. It holds ONE process-wide subscription on one channel rather than one per workspace, which changes the following and nothing else:
- Cost is flat. One frame per instance per interval regardless of how many workspaces or clients exist, against the activity bus's one per subscribed workspace.
- A cycle affects every watch subscriber on the instance, not one
workspace's — but that is the path
dropCoveragealready takes for a resubscription or an undecodable message, so it is a third trigger on an existing announcement rather than a wider one. It is bounded: at most one announcement per connected subscriber per cycle (the signal is capacity-1 and coalescing), and at most one cycle per idle timeout per instance. - No separate cycle counter. On the activity stream
pad_event_subscription_cycled_totalexists because its reset reason only fires when a buffer existed to drop. The watch bus drops its buffer unconditionally, sopad_watchevents_sequence_resets_total{reason="idle_timeout"}is already a complete count and a second metric would be noise. - A failed re-dial retries without re-deciding. Read
pad_watchevents_sequence_resets_total{reason="idle_timeout"}as one per outage, not one per cadence. When the cycle's replacement cannot be established — Redis away, the path still blackholed — the watch bus keeps retrying the dial every interval but does not drop coverage or announce again: coverage is already ended and the buffer is already empty, so a second drop would re-announce a hole every subscriber has been told about and turn one outage into an incident per cadence on the series you alert on. The activity stream reaches the same place by a different road: its teardown removes the workspace's subscription entry outright, so the next scan finds nothing to cycle and its recovery runs off the request path instead.
pad_watchevents_heartbeat_publish_failures_total is the watch twin of the
activity stream's probe-failure counter and reads the same way: detection
degraded, not a peer broken.
Security
| Variable | Default | Description |
|---|---|---|
PAD_SECURE_COOKIES |
false |
Set Secure flag on session cookies (requires TLS) |
PAD_CORS_ORIGINS |
— | Comma-separated allowed CORS origins |
Email (Optional)
Email enables sending workspace invitation links. Without it, users can still join via CLI invite codes.
| Variable | Default | Description |
|---|---|---|
PAD_MAILEROO_API_KEY |
— | Maileroo sending API key |
PAD_EMAIL_FROM |
noreply@getpad.dev |
Sender email address |
PAD_EMAIL_FROM_NAME |
Pad |
Sender display name |
Password recovery (when email is not configured)
Without an email provider, the web "Forgot password" flow can't send a reset
link — the page says so and points users at the host-side recovery below.
Recover a locked-out account from the server host (the same trust model
as pad auth setup — shell access to the box):
# Print a single-use reset link (open it in a browser to choose a new password)
pad auth reset-password admin@example.com
# Or set a random temporary password, printed to the terminal (headless boxes).
# Log in with it, then change it immediately — all existing sessions are signed out.
pad auth reset-password admin@example.com --temp-password
This calls a loopback-only endpoint (POST /api/v1/auth/local-reset): it
needs no login (you're locked out, after all), but it only works for a
direct request from the server itself — proxied or remote requests are
refused, and it's disabled entirely in cloud mode.
Alternatively, if a user submits the web reset form, the server logs the reset path on a non-cloud instance with no email configured:
password reset generated (email not configured) ... reset_path=/reset-password/<token>
Open <base-url>/reset-password/<token> to finish the reset by hand.
Deployment Options
Single Binary (SQLite)
The simplest deployment — one binary, one file for the database.
# Download or build
make build
# Run directly
PAD_HOST=0.0.0.0 ./pad server start
# Or install as a systemd service (see below)
Best for: single-user, small teams, evaluations.
Docker Compose (PostgreSQL + Redis)
See Quick Start above. This is the recommended setup for teams.
Kubernetes
Manifests are in deploy/k8s/. Apply them in order:
# Create namespace
kubectl apply -f deploy/k8s/namespace.yaml
# Configure secrets (edit first!)
kubectl apply -f deploy/k8s/secret.yaml
# Deploy
kubectl apply -f deploy/k8s/configmap.yaml
kubectl apply -f deploy/k8s/deployment.yaml
kubectl apply -f deploy/k8s/service.yaml
kubectl apply -f deploy/k8s/ingress.yaml
kubectl apply -f deploy/k8s/hpa.yaml
Prerequisites:
- External PostgreSQL (e.g., AWS RDS, Cloud SQL, managed PG)
- External Redis (e.g., ElastiCache, Memorystore)
- Ingress controller (nginx-ingress or similar)
- TLS certificates (cert-manager recommended)
Systemd Service
# /etc/systemd/system/pad.service
[Unit]
Description=Pad
After=network.target postgresql.service redis.service
[Service]
Type=simple
User=pad
Group=pad
ExecStart=/usr/local/bin/pad server start
Environment=PAD_HOST=0.0.0.0
Environment=PAD_DATA_DIR=/var/lib/pad
Environment=PAD_DB_DRIVER=postgres
Environment=PAD_DATABASE_URL=postgres://pad:secret@localhost:5432/pad
Environment=PAD_REDIS_URL=redis://localhost:6379
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now pad
Reverse Proxy
Pad needs a reverse proxy for TLS termination. SSE connections require specific proxy settings to avoid buffering.
Caddy (Recommended)
Caddy handles TLS automatically. See deploy/Caddyfile:
pad.example.com {
reverse_proxy pad:7777 {
flush_interval -1
}
}
nginx
See deploy/nginx.conf. Critical settings for SSE:
location /api/v1/events {
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 86400s;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
Monitoring
Pad exposes Prometheus metrics at /metrics (unauthenticated). Key metrics:
| Metric | Type | Description |
|---|---|---|
pad_http_requests_total |
counter | Total HTTP requests by method, path, status |
pad_http_request_duration_seconds |
histogram | Request latency |
pad_http_response_size_bytes |
histogram | Response body sizes |
pad_sse_connections_active |
gauge | Connections on the workspace activity stream (/api/v1/events) only |
pad_stream_connections_active |
gauge | Held connections across both SSE endpoints — the population the limits bound |
pad_eventbus_publish_total |
counter | Events HANDED to the bus — attempts, not confirmed publishes. A failed Redis publish is logged and still counted (BUG-2732) |
pad_eventbus_subscribers |
gauge | Active event subscribers |
pad_db_open_connections |
gauge | Database connection pool stats |
Redis-specific metrics are listed under Redis health and metrics.
Health Check
Three endpoints, and they answer different questions:
# Liveness — is the process up? Kubernetes restarts the pod when this fails.
curl http://localhost:7777/api/v1/health/live
# {"status":"ok"}
# Readiness — can it serve traffic? Gated on the DATABASE only.
# Kubernetes should point its readinessProbe here.
curl -s http://localhost:7777/api/v1/health/ready
# {
# "status": "ready",
# "db": {"open_connections": 2, "in_use": 0, "idle": 2, "driver": "sqlite"},
# "redis": {"reachable": true, "probed": true, "last_check": "2026-08-22T01:00:00Z"}
# }
# Build info.
curl http://localhost:7777/api/v1/health
# {"status":"ok","version":"...","commit":"..."}
With Redis configured but unreachable, readiness stays 200 and the redis
block carries the failure. Readiness deliberately does not fail: Pad still
serves the API, the web UI and every item write. What it cannot do is
cross-instance delivery, and the paths whose job that is say so —
POST .../push answers 503 for a session-targeted push it cannot resolve
and 502 push_unconfirmed when the publish fails:
{
"status": "ready",
"redis": {
"reachable": false,
"probed": true,
"error": "dial tcp ...: connect: connection refused",
"degrades": [
"all activity events, including to clients on this instance",
"watch notifications",
"session presence and session-targeted push"
]
}
}
The redis block is absent entirely when no Redis is configured — "not
applicable" rather than "down".
Upgrading
Pad releases a new binary roughly weekly. Migrations run automatically at startup — only the ones your database is missing are applied, and each one commits atomically, so a failed migration rolls back cleanly and is retried on the next boot.
Only ever move forward. A newer binary can migrate an older database; an older binary cannot understand a newer schema. Pad enforces this with a schema-ahead guard: if the binary finds a database that carries migrations it doesn't ship (the signature of a downgrade — a rolled-back brew formula, an older Docker tag, a redeployed prior binary), it refuses to start instead of silently running old code against a newer schema and corrupting data.
database schema is newer than this pad binary: the database has N migration(s)
this binary doesn't ship (...) ... This almost always means the binary was
DOWNGRADED (e.g. brew/docker rollback) ... Upgrade pad back to a build that
includes those migrations, or ... re-run with `pad start --force`.
- Recover by reinstalling the newer binary (
brew upgrade pad, pull the newer Docker tag, redeploy the newer image). - Override — only if you have intentionally downgraded and accept the
data-corruption risk — with
pad start --forceorPAD_ALLOW_SCHEMA_AHEAD=1.
Pre-migration snapshot (SQLite)
When a SQLite-backed instance has pending migrations, Pad copies the database
file to pad.db.pre-<version> (next to the DB) before applying them. If an
upgrade goes wrong, stop the server and copy that file back over pad.db. It
is a convenience net, not a substitute for backups — take a real backup first
(see backup.md). The copy is best-effort: if it can't be written
(read-only volume, full disk) the server logs a warning and proceeds, so keep
your own backups regardless.
PostgreSQL is not snapshotted this way — take a pg_dump or provider snapshot
before upgrading (see backup.md).
Recommended flow
# 1. Back up (SQLite shown; pg_dump for Postgres — see backup.md)
pad db backup -o pad-backup-$(date +%Y%m%d).db
# 2. Stop, install the new binary, restart. Migrations + the pre-migration
# snapshot run automatically on start.
brew upgrade pad # or: docker pull, binary download, systemctl restart pad
# 3. Verify
pad --version
curl -s http://localhost:7777/api/v1/health # {"status":"ok"}
Production Checklist
- Database: PostgreSQL configured with
PAD_DB_DRIVER=postgres - Redis: Connected for multi-instance events, notifications, and session presence (
PAD_REDIS_URL), on a non-evictingmaxmemory-policy, single node (not a cluster) - Redis namespace:
PAD_REDIS_NAMESPACEset if this endpoint is shared with another Pad installation - Streaming limits:
PAD_SSE_MAX_CONNECTIONS/PAD_SSE_MAX_PER_USERsized for your fleet (both cover both SSE endpoints) - Redis alerting:
pad_redis_upandpad_watchevents_sequence_gaps_totalwired to alerts - Stream-honesty alerting:
pad_event_resume_gaps_totalalerting on a rate that does NOT settle after a deploy (a step around one is expected — cold replay buffers), andpad_event_receive_loop_exits_totalread as a rate against a stable subscriber count. See the metrics table above for what each label means - TLS: Reverse proxy with valid certificates
- Secure cookies:
PAD_SECURE_COOKIES=true(requires TLS) - Public URL:
PAD_URLset to your public-facing domain - CORS:
PAD_CORS_ORIGINSset if serving from a different domain - Backups: PostgreSQL backup strategy in place (see
docs/backup.md) - Monitoring: Prometheus scraping
/metrics - Admin account: Created via
pad auth setupor web UI on first visit - Email (optional): Maileroo configured for invitation emails
- Resource limits: Set in Docker Compose or K8s manifests
- Log level:
PAD_LOG_LEVEL=info(usedebugonly for troubleshooting)