---
title: Global Observability
description: A unified, searchable log stream from the containers Sencho manages on the hub.
---

The **Logs** tab aggregates output from the containers Sencho manages into a single live feed. Instead of tailing one container at a time, you scan everything in one place, with a live event-rate readout, stream / level / stack filters, and a downloadable replay of the current buffer.

<Note>
  Logs is an administrator view. It is scoped to the hub and is hidden from the nav strip when a remote node is the active selection. See [Multi-Node Management](/features/multi-node#what-top-level-views-show-when-a-remote-node-is-active).
</Note>

<Frame>
  <img src="/images/global-observability/global-observability-overview.png" alt="Global Observability page with the live logs masthead, four-tile signal rail, filter strip, and a feed body showing day-banded rows in info green and error rose." />
</Frame>

## Status masthead

The masthead at the top of the page is the single place to read the stream's current condition.

<Frame>
  <img src="/images/global-observability/masthead.png" alt="Logs masthead with the kicker 'LIVE LOGS · NODE · LOCAL', the state word 'Streaming' in editorial italics next to a pulsing brand-cyan dot, and the meta tiles 'LAST EVENT NOW' and 'SESSION 1M 43S' on the right." />
</Frame>

It carries:

- A **kicker** in uppercase mono tracking that reads `LIVE LOGS · NODE · LOCAL`, a reminder that the feed is the hub's own managed containers.
- A **pulsing dot** that mirrors the stream tone: brand cyan and pulsing while events arrive within the live window, gray and steady when the stream goes quiet, solid rose when the underlying SSE connection has errored.
- A **state word** set in the editorial display face: `Streaming` while the dot is live, `Idle` when the stream has gone quiet, `Offline` when the connection has failed.
- A **LAST EVENT** stat that names the band of the most recent event using the same vocabulary as the feed (`NOW`, `2M AGO`, `1H AGO`, then the calendar date), so you can see how stale the latest line is at a glance.
- A **SESSION** stat counting how long this Logs tab has been open, formatted with uppercase letter suffixes: `1H 43M` once the session crosses an hour, `0M 12S` under it.

The state word flips to `Idle` after ten seconds without an event. That is a feature of the readout, not the stream: the SSE connection stays open in the background and the dot will pulse cyan again on the next event.

## Signal rail

A four-tile rail directly under the masthead summarizes the current buffer.

<Frame>
  <img src="/images/global-observability/signal-rail.png" alt="Four-tile signal rail: 'EVENTS / MIN' showing 20 with a 60-second sparkline, 'ERRORS' showing 418 in destructive red, 'WARNINGS' showing 9 in amber, 'CONTAINERS' showing 8 in stat-value white." />
</Frame>

| Tile | What it shows |
|------|---------------|
| **EVENTS / MIN** | Total events received in the last sixty seconds, with a 60-bucket sparkline (one bucket per second) under the number. The sparkline shifts left every tick so you read it as a live rhythm rather than a static chart. |
| **ERRORS** | Count of `ERROR`-level lines in the buffer. Tints destructive red whenever the count is above zero; otherwise the number sits in subtitle gray. |
| **WARNINGS** | Count of `WARN`-level lines in the buffer. Tints warning amber whenever the count is above zero; otherwise the number sits in subtitle gray. |
| **CONTAINERS** | Count of distinct container names present in the in-memory buffer (after the **Clear** cutoff). Containers whose last entry has aged out of the buffer drop out of the count. |

All four counters are scoped to what is currently in the buffer. **Clear** sweeps them along with the visible rows.

## Filter strip

A row of controls between the signal rail and the feed lets you narrow what you see without touching the underlying stream.

<Frame>
  <img src="/images/global-observability/filter-strip.png" alt="Filter strip with a 'Search logs...' textbox, a 'STACKS · ALL' dropdown trigger, a stream segmented control with 'ALL OUT ERR' (ALL selected), and a level segmented control with 'ALL INFO WARN ERROR' (ALL selected)." />
</Frame>

| Control | What it does |
|---------|--------------|
| **Search** | Case-insensitive substring filter against the message body, container name, and stack name. |
| **Stacks** | Dropdown of every stack discovered on the hub; ticking one or more boxes restricts the feed to those stacks. The trigger reads `Stacks · All` when nothing is selected and `Stacks · <n>` once at least one box is ticked. |
| **Stream** | Segmented control with `All`, `Out`, `Err` for filtering by stream source. `Out` is `STDOUT`, `Err` is `STDERR`. |
| **Level** | Segmented control with `All`, `Info`, `Warn`, `Error` for filtering by detected severity. |

Filters AND together and run in the browser, so there is no per-keystroke round-trip to the backend. Switching them does not lose any buffered data; rows reappear if you broaden the filter again.

## Feed

The body of the page is a chronological list of every log line in the buffer that passes the current filter.

<Frame>
  <img src="/images/global-observability/feed-bands.png" alt="Feed body showing day band headers '3M AGO' and '2M AGO' between groups of rows. Each row shows a green INFO dot, a 24-hour timestamp, a cyan container name (seerr, profilarr), and the raw log message." />
</Frame>

Each row carries:

- A small **severity dot** tinted by detected level: success green for `INFO`, warning amber for `WARN`, destructive rose with a soft glow for `ERROR`.
- A **timestamp** in 24-hour local time (`HH:MM:SS`), set in tabular nums so columns line up across rows.
- A **container name** in brand cyan. Hover to see the full `<stack>/<container>` path in a tooltip; the displayed name is the compose service name with the project prefix and replica suffix stripped.
- A **message** containing the raw log output. `STDERR`-source lines render in destructive rose; `STDOUT`-source lines render in stat-value white.

A row's whole-row tint follows its detected level. Error rows pick up a subtle destructive wash, warning rows pick up an amber wash, info rows have no tint and gain a subtle accent on hover. The detected level is independent of the source, so an `STDOUT` line that contains the word `ERROR:` still gets the rose row tint.

When consecutive rows span more than a minute, a tracked-mono **band header** is inserted between them. Bands roll up over time: `NOW` for the last sixty seconds, then `2M AGO`, `12M AGO`, `1H AGO`, and eventually the calendar date.

The feed shows the most recent three hundred rows that pass the filter. If more rows match, a `Showing last 300 of <n>` notice sits at the top of the feed so you know lines have been clipped, even though the full ring buffer behind the scenes still holds them.

The feed uses two empty states, both rendered as a tracked-mono kicker over an italic caption. Before any event arrives, the body shows `Awaiting events` over `Logs will appear here as containers emit them.` When the current filter excludes every row in the buffer, the body shows `No matches` over `Try a broader filter to see logs again.`

The viewport autoscrolls to follow new rows. If you scroll up to read older entries, autoscroll pauses so the feed doesn't yank you back. Scrolling within fifty pixels of the bottom re-engages it.

## Pause, clear, download

A floating control strip is anchored to the bottom-right of the feed.

<Frame>
  <img src="/images/global-observability/paused-resume-chip.png" alt="Floating control strip with a brand-cyan '22 NEW · RESUME' pill on the left and the Pause/Clear/Download chip group on the right. The Pause button shows a play icon and the label 'RESUME', indicating the stream is currently paused." />
</Frame>

| Action | What it does |
|--------|--------------|
| **Pause / Resume** | Freezes the feed without dropping the SSE connection. New events keep flowing into the in-memory buffer so the page does not lose ground. While paused, a brand-cyan **`<n> NEW · RESUME`** pill appears to the left of the strip and counts the events queued since you paused; clicking either the pill or the **Resume** button flushes them and returns to the live view. The pause toggle's icon and label flip between play / `RESUME` and pause / `PAUSE` in lockstep with the state. |
| **Clear** | Hides every row that arrived before the click; counters reset to match. The underlying Docker logs are untouched. New events that arrive after the click reappear in the feed. |
| **Download** | Exports the rows currently passing the filter as a `.txt` file. The filename is `sencho-logs-<ISO8601>.txt` (with `:` and `.` replaced by `-` so Windows accepts the name) and each line is formatted as `[<ISO timestamp>] [<stack>/<container>] <LEVEL>: <message>`. The button disables when the filter currently matches zero rows. |

## Filtering by level

The **Level** segmented control drops the feed to a single severity. Pairing it with **Stacks** is a fast way to investigate a misbehaving stack.

<Frame>
  <img src="/images/global-observability/error-only-filter.png" alt="Logs page with the Level filter set to 'Error'. The feed shows only error-tinted rows from the sencho and radarr containers, with day bands '6M AGO' through 'NOW' visible." />
</Frame>

The level on each row is detected from the message body, not the stream source. Sencho's parser scans the line for keywords (`info`, `debug`, `warn`, `error`) and structured fields (`level=...`, `[INFO]`, `Exception:`) before falling back to a source-based default of `STDERR → ERROR`, `STDOUT → INFO`. That means an `STDOUT` line containing `ERROR:` classifies as `ERROR` (and gets the rose tint), and an `STDERR` line containing `info` classifies as `INFO` (and gets the green dot).

## How streaming works

The page opens an `EventSource` against `/api/logs/global/stream` as soon as it mounts. Each event is one log line emitted by `docker logs --follow` against each managed container on the hub, demultiplexed and parsed before it leaves the server. When the connection opens, the server replays the last two hundred lines per container before switching to live tail, so you see immediate context instead of an empty feed while you wait for the next event.

A 30-second SSE keep-alive comment is written every tick to defeat reverse-proxy idle timeouts, so the stream stays open behind nginx, Cloudflare, Caddy, and so on without configuration.

If the browser cannot open the `EventSource` connection at all (a corporate proxy that strips SSE, an outbound block on the streaming endpoint), the page silently falls back to polling `/api/logs/global` every five seconds. The polling endpoint returns the last five hundred lines on each call as a single snapshot. The transport choice is automatic; you cannot pick one explicitly.

## Display limits

Three budgets keep the page responsive even on a chatty host:

- **Buffer** of two thousand entries in the browser's memory. As new events arrive past the cap, the oldest entries are dropped. The buffer keeps filling while the feed is paused, so a long pause does not stretch memory unbounded.
- **Rendered rows** capped at three hundred. Beyond that, the `Showing last 300 of <n>` banner appears so you know more matches exist behind the slice.
- **Followed containers** capped per stream. When the hub manages more containers than the cap, the feed follows the first set and posts a `Following <n> of <m> managed containers` notice so you know the feed is truncated; use the per-container log viewer for the rest.

When you need to chase older lines than the buffer holds, jump to the [per-container log viewer](/features/editor#log-viewer) in the Editor tab or run `docker compose logs` from the [Host Console](/features/host-console).

## Scope

The Logs tab is scoped to the hub and streams the containers Sencho manages on the local instance.

- The kicker reads `LIVE LOGS · NODE · LOCAL`, so it is always clear the feed is the hub's own managed containers.
- The feed covers managed containers only. Containers running on the host that Sencho does not manage (started outside its compose directory) are not shown; reach those through their own tooling or the [Host Console](/features/host-console).
- There is no fleet-wide aggregation view. When a remote node is the active selection, the Logs entry is hidden from the nav strip; read a remote node's containers from the Logs tab on that node's own Sencho.

## Refresh cadence

The page is built from several independent loops so the masthead and the sparkline animate without flooding the network:

| Surface | Cadence |
|---------|---------|
| UI tick (uptime, band labels, live-window check) | 1 second |
| Buffer flush from the SSE batch into the visible feed | 500 ms |
| Polling fallback when SSE never opens | 5 seconds |
| SSE keep-alive heartbeat from the server | 30 seconds |
| Sparkline rolling window | 60 buckets × 1 second = 60 seconds |
| `Idle` threshold (no events received) | 10 seconds |

## Troubleshooting

<AccordionGroup>
  <Accordion title="The masthead reads 'Offline' and a 'Failed to fetch logs. Retrying...' banner is visible">
    The polling fallback got an HTTP error when it tried to read `/api/logs/global`. Check that the host's Docker daemon is reachable on the socket Sencho is configured to use. The page retries the request every five seconds; once it succeeds, the banner clears on its own.
  </Accordion>
  <Accordion title="The dot is gray and the state reads 'Idle' even though logs are appearing further down the page">
    `Idle` only checks whether an event has arrived in the last ten seconds. If your fleet is genuinely quiet (a maintenance window, a single low-traffic container), the dot rests gray and the band headers in the feed roll up to `2M AGO`, `5M AGO`, and so on. The next event flips the dot back to brand cyan and the state word back to `Streaming` on the next one-second tick.
  </Accordion>
  <Accordion title="A line that says 'ERROR' renders without the rose row tint">
    Severity is detected once per line by scanning the message body. Some loggers print `ERROR` as part of a stack trace header on a line that does not itself match the parser's keyword set, in which case the line is classified by its source: `STDOUT` falls back to `INFO`. To force the row to your preferred level, change the application's log format to something the parser recognises (`level=error`, `[ERROR]`, or a leading `Error:` / `Exception:`). The behavior is intentional: it keeps a noisy `STDOUT` line that quotes the word `ERROR` from polluting the error count.
  </Accordion>
  <Accordion title="The Resume pill keeps growing while I read a paused feed">
    The buffer caps at two thousand entries even while paused. Once you cross the cap, the oldest queued events drop, and the pill's count reflects the size of the queue the next flush will deliver, not the total number of events that arrived during the pause. Resume to flush, then re-pause if you need to keep reading; the page will not lose its place because autoscroll only re-engages when you scroll back to the bottom.
  </Accordion>
  <Accordion title="Clear hid the rows but a few old lines came back a moment later">
    **Clear** records the click time and hides every row whose timestamp is earlier than that moment, but the SSE stream keeps delivering events whose timestamps were generated by Docker before the click. A line emitted at `16:51:43.987` that arrives at the browser at `16:51:44.020` will still pass the filter even if you clicked **Clear** at `16:51:44.000`. The lag is normally a few hundred milliseconds; click **Clear** again to hide them.
  </Accordion>
  <Accordion title="A container I expect is missing from the feed">
    The feed shows the containers Sencho manages (those started from its compose directory). A container started outside Sencho, or one belonging to a stack on a different node, does not appear. Open it from the [per-container log viewer](/features/editor#log-viewer) in the Editor, or use the [Host Console](/features/host-console) for ad-hoc `docker compose logs`.
  </Accordion>
  <Accordion title="A row says a log stream ended unexpectedly">
    When one container's follow stream drops (the container was removed, or the Docker daemon hiccuped), the feed posts a single warning row for that container and stops following it; the other containers keep streaming. Reopen the Logs tab to re-attach to every managed container.
  </Accordion>
  <Accordion title="I want to see logs from every node at once">
    The Logs tab does not aggregate across the fleet. Use the per-container log viewer in the Editor for a single container's history, or jump to the Host Console for ad-hoc `docker compose logs`. Fleet-wide aggregation is not on the current roadmap; pipe container logs to your existing observability stack (Loki, ELK, Datadog) if you need cross-node search.
  </Accordion>
</AccordionGroup>

<Note>
  Log retention on the backend is controlled by **Settings · Operations · Data Retention** (default thirty days). That setting governs alert and notification history stored in Sencho's database. It does not apply to the live container stream this page renders, which is bound entirely by Docker's own log driver and the in-browser buffers above.
</Note>
